[PATCH] Fixing package-initialize, adding early init file

[PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

Hello all,

Attached is a preliminary patch for fixing "the package-initialize
problem" (see [1] [2] [3]) by adding an early init file. Feedback
welcome. In particular,

- do we want to try to factor out some common logic in loading the
  early init file versus the regular init file?
- have I broken anything in moving the check for an invalid username
  earlier in startup.el? what about moving package-initialize earlier?
- should `early-init-file' be defined in C?
- did I miss any documentation fixes?
- have I broken any style guidelines for the repository?

Additional question: if we moved forward with this patch, would it
make it into Emacs 26.1?

Best,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
[2]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
[3]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 24502 bytes --]

From dc8ff75037f90938f1d3eb9676f674aad868c2da Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Sep 2017 22:17:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* src/lread.c (Vearly_init_file): New variable, for the filename of
  the early init file that was loaded.

* lisp/startup.el: Load the early init file, if it is found. Move the
  check for an invalid username to just before that, and move the
  initialization of the package system to just after.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove. Burn
  with fire. Remove related code and documentation.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
  package-initialize is called, and advise against calling it in the
  init file.

* doc/emacs/package.texi: Document changes to when package-initialize
  is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
  init file.

* etc/NEWS: Document changes to startup and package.el.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
---
 doc/emacs/package.texi     |  28 ++-----
 doc/lispref/os.texi        |  14 ++++
 doc/lispref/package.texi   |  16 +++-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  18 ++++
 lisp/emacs-lisp/package.el |  71 +---------------
 lisp/startup.el            | 204 +++++++++++++++++++++++++++++++++------------
 src/lread.c                |   9 ++
 8 files changed, 212 insertions(+), 152 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 215f50cb40..bd7cccce6c 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -253,30 +253,16 @@ Package Installation
 consult the package's help buffer.
 
   By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+subsequent Emacs sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 441fda5d82..062aa28222 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, here you can customize the process
+  of loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 153ee48741..2e93f29bd2 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the early init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is @code{nil}.  (Of course, the
+setting of this user option must be done in the early init file.)
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index ca57501f3d..c487d43a13 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -891,9 +891,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 371cdf686c..80678551c7 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -28,6 +28,24 @@ When you add a new item, use the appropriate mark if you are sure it applies,
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called early-init.el, in `user-emacs-directory'.  It is
+loaded very early in the startup process: in particular, before
+graphical elements such as the tool bar are initialized, and before
+the package manager is initialized.
+
++++
+** Emacs now initializes package.el before loading the init-file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to (package-initialize) into the
+init-file, which was previously done when Emacs was started.  Users
+who do not configure package.el variables such as `package-load-list'
+and `package-user-dir' need not make any configuration changes.  Users
+who do configure such variables should place the configuration into
+the newly introduced early init file, which is loaded before
+package.el is initialized.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 8b101c1323..305453a4ba 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1436,16 +1436,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1454,13 +1449,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1877,64 +1866,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 7cf6fee425..56f688e334 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1021,6 +1021,156 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial (if (eq init-file-debug t)
+                                    'startup
+                                  init-file-debug))
+        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
+    (let ((debug-on-error debug-on-error-initial)
+          (inner
+           (lambda ()
+             ;; If no username, don't load the init file.
+             (when init-file-user
+               (let ((early-init-file-1
+                      (expand-file-name
+                       "early-init"
+                       (file-name-as-directory
+                        (concat "~" init-file-user "/.emacs.d")))))
+                 ;; Setting `user-init-file' to t tells `load' to
+                 ;; store the name of the file that was loaded, if
+                 ;; possible, into `user-init-file'. We're not using
+                 ;; `user-init-file' yet, so we can re-use it here for
+                 ;; the early init file.
+                 (setq user-init-file t)
+
+                 ;; Attempt to load the early init file. If it doesn't
+                 ;; exist, do nothing.
+                 (load early-init-file-1 t t)
+
+                 ;; If the init file could be loaded, move its
+                 ;; discovered filename from `user-init-file' into
+                 ;; `early-init-file', where it belongs.
+                 (unless (eq user-init-file t)
+                   (setq early-init-file user-init-file)
+                   (when (and early-init-file
+                              (equal (file-name-extension early-init-file)
+                                     "elc"))
+                     (let* ((source (file-name-sans-extension early-init-file))
+                            (alt (concat source ".el")))
+                       (setq source (cond ((file-exists-p alt) alt)
+                                          ((file-exists-p source) source)
+                                          (t nil)))
+                       (when source
+                         (when (file-newer-than-file-p source early-init-file)
+                           (message "Warning: %s is newer than %s"
+                                    source early-init-file)
+                           (sit-for 1))
+                         (setq early-init-file source))))))))))
+      (if init-file-debug
+          (funcall inner)
+        (condition-case error
+            (progn
+              (funcall inner)
+              (setq init-file-had-error nil))
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            ;; Use `user-init-file' here because if
+                            ;; there was an error while loading the
+                            ;; init file, then `early-init-file' may
+                            ;; not have been reassigned; but
+                            ;; `user-init-file' will still be set by
+                            ;; `load'.
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+    (if debug-on-error-should-be-set
+	  (setq debug-on-error debug-on-error-from-init-file))
+      (unless (or (default-value 'enable-multibyte-characters)
+		  (eq orig-enable-multibyte (default-value
+					      'enable-multibyte-characters)))
+	;; Init file changed to unibyte.  Reset existing multibyte
+	;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
+	;; Arguably this should only be done if they're free of
+	;; multibyte characters.
+	(mapc (lambda (buffer)
+		(with-current-buffer buffer
+		  (if enable-multibyte-characters
+		      (set-buffer-multibyte nil))))
+	      (buffer-list))
+	;; Also re-set the language environment in case it was
+	;; originally done before unibyte was set and is sensitive to
+	;; unibyte (display table, terminal coding system &c).
+	(set-language-environment current-language-environment)))
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1127,33 +1277,6 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
     (let (debug-on-error-from-init-file
 	  debug-on-error-should-be-set
@@ -1312,33 +1435,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 6bc93b1481..b4823956e7 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4910,6 +4910,15 @@ While Emacs loads and evaluates the init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
+  DEFVAR_LISP ("early-init-file", Vearly_init_file,
+               doc: /* File name, including directory, of user's early init file.
+If the file loaded had extension `.elc', and the corresponding source file
+exists, this variable contains the name of source file, suitable for use
+by functions like `custom-save-all' which edit the init file.
+While Emacs loads and evaluates the init file, value is the real name
+of the file, regardless of whether or not it has the `.elc' extension.  */);
+  Vearly_init_file = Qnil;
+
   DEFVAR_LISP ("current-load-list", Vcurrent_load_list,
 	       doc: /* Used for internal purposes by `load'.  */);
   Vcurrent_load_list = Qnil;
-- 
2.13.3

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> Attached is a preliminary patch for fixing "the package-initialize
> problem" (see [1] [2] [3]) by adding an early init file. Feedback
> welcome. In particular,

Thanks, comments below.

>  ;;;###autoload
>  (defun package-initialize (&optional no-activate)
[...]
> +  (setq package-enable-at-startup nil)

BTW, in a subsequent patch we could rename this to package-initialized,
so the name reflects what the variable contains rather than what it's
used for.

> +  DEFVAR_LISP ("early-init-file", Vearly_init_file,
> +               doc: /* File name, including directory, of user's early init file.
> +If the file loaded had extension `.elc', and the corresponding source file
> +exists, this variable contains the name of source file, suitable for use
> +by functions like `custom-save-all' which edit the init file.
> +While Emacs loads and evaluates the init file, value is the real name
> +of the file, regardless of whether or not it has the `.elc' extension.  */);
> +  Vearly_init_file = Qnil;

This is not used from C code, so make it a plain old defvar in startup.el.

I haven't yet looked at the startup.el part, sorry.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1.1: Type: text/plain, Size: 534 bytes --]

> BTW, in a subsequent patch we could rename this to
> package-initialized, so the name reflects what the variable contains
> rather than what it's used for.

I don't think that's a good idea, since the most common usage of this
variable will be

    (setq package-initialized nil)

in the early init file, and that doesn't make much sense from a
readability perspective (are we asserting that package.el wasn't
initialized?).

> This is not used from C code, so make it a plain old defvar in
> startup.el.

Done. New patch attached.

[-- Attachment #1.2: Type: text/html, Size: 810 bytes --]

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 24190 bytes --]

From 9d2e25b8c7c0f11fc8e5fb46047701d43e69af6f Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Sep 2017 22:17:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el: New variable `early-init-file', for the filename of
  the early init file that was loaded.  Load the early init file, if
  it is found.  Move the check for an invalid username to just before
  that, and move the initialization of the package system to just
  after.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove. Burn
  with fire. Remove related code and documentation.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
  package-initialize is called, and advise against calling it in the
  init file.

* doc/emacs/package.texi: Document changes to when package-initialize
  is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
  init file.

* etc/NEWS: Document changes to startup and package.el.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
---
 doc/emacs/package.texi     |  28 ++----
 doc/lispref/os.texi        |  14 +++
 doc/lispref/package.texi   |  16 +++-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  18 ++++
 lisp/emacs-lisp/package.el |  71 +--------------
 lisp/startup.el            | 213 +++++++++++++++++++++++++++++++++------------
 7 files changed, 212 insertions(+), 152 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 215f50cb40..bd7cccce6c 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -253,30 +253,16 @@ Package Installation
 consult the package's help buffer.
 
   By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+subsequent Emacs sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 441fda5d82..062aa28222 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, here you can customize the process
+  of loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 153ee48741..2e93f29bd2 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the early init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is @code{nil}.  (Of course, the
+setting of this user option must be done in the early init file.)
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index ca57501f3d..c487d43a13 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -891,9 +891,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 371cdf686c..80678551c7 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -28,6 +28,24 @@ When you add a new item, use the appropriate mark if you are sure it applies,
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called early-init.el, in `user-emacs-directory'.  It is
+loaded very early in the startup process: in particular, before
+graphical elements such as the tool bar are initialized, and before
+the package manager is initialized.
+
++++
+** Emacs now initializes package.el before loading the init-file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to (package-initialize) into the
+init-file, which was previously done when Emacs was started.  Users
+who do not configure package.el variables such as `package-load-list'
+and `package-user-dir' need not make any configuration changes.  Users
+who do configure such variables should place the configuration into
+the newly introduced early init file, which is loaded before
+package.el is initialized.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 8b101c1323..305453a4ba 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1436,16 +1436,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1454,13 +1449,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1877,64 +1866,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 7cf6fee425..52fd1f9cf5 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,15 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+If the file loaded had extension `.elc', and the corresponding
+source file exists, this variable contains the name of source
+file, suitable for use by functions like `custom-save-all' which
+edit the init file.  While Emacs loads and evaluates the init
+file, value is the real name of the file, regardless of whether
+or not it has the `.elc' extension.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -1021,6 +1030,156 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial (if (eq init-file-debug t)
+                                    'startup
+                                  init-file-debug))
+        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
+    (let ((debug-on-error debug-on-error-initial)
+          (inner
+           (lambda ()
+             ;; If no username, don't load the init file.
+             (when init-file-user
+               (let ((early-init-file-1
+                      (expand-file-name
+                       "early-init"
+                       (file-name-as-directory
+                        (concat "~" init-file-user "/.emacs.d")))))
+                 ;; Setting `user-init-file' to t tells `load' to
+                 ;; store the name of the file that was loaded, if
+                 ;; possible, into `user-init-file'. We're not using
+                 ;; `user-init-file' yet, so we can re-use it here for
+                 ;; the early init file.
+                 (setq user-init-file t)
+
+                 ;; Attempt to load the early init file. If it doesn't
+                 ;; exist, do nothing.
+                 (load early-init-file-1 t t)
+
+                 ;; If the init file could be loaded, move its
+                 ;; discovered filename from `user-init-file' into
+                 ;; `early-init-file', where it belongs.
+                 (unless (eq user-init-file t)
+                   (setq early-init-file user-init-file)
+                   (when (and early-init-file
+                              (equal (file-name-extension early-init-file)
+                                     "elc"))
+                     (let* ((source (file-name-sans-extension early-init-file))
+                            (alt (concat source ".el")))
+                       (setq source (cond ((file-exists-p alt) alt)
+                                          ((file-exists-p source) source)
+                                          (t nil)))
+                       (when source
+                         (when (file-newer-than-file-p source early-init-file)
+                           (message "Warning: %s is newer than %s"
+                                    source early-init-file)
+                           (sit-for 1))
+                         (setq early-init-file source))))))))))
+      (if init-file-debug
+          (funcall inner)
+        (condition-case error
+            (progn
+              (funcall inner)
+              (setq init-file-had-error nil))
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            ;; Use `user-init-file' here because if
+                            ;; there was an error while loading the
+                            ;; init file, then `early-init-file' may
+                            ;; not have been reassigned; but
+                            ;; `user-init-file' will still be set by
+                            ;; `load'.
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+    (if debug-on-error-should-be-set
+	  (setq debug-on-error debug-on-error-from-init-file))
+      (unless (or (default-value 'enable-multibyte-characters)
+		  (eq orig-enable-multibyte (default-value
+					      'enable-multibyte-characters)))
+	;; Init file changed to unibyte.  Reset existing multibyte
+	;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
+	;; Arguably this should only be done if they're free of
+	;; multibyte characters.
+	(mapc (lambda (buffer)
+		(with-current-buffer buffer
+		  (if enable-multibyte-characters
+		      (set-buffer-multibyte nil))))
+	      (buffer-list))
+	;; Also re-set the language environment in case it was
+	;; originally done before unibyte was set and is sensitive to
+	;; unibyte (display table, terminal coding system &c).
+	(set-language-environment current-language-environment)))
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1127,33 +1286,6 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
     (let (debug-on-error-from-init-file
 	  debug-on-error-should-be-set
@@ -1312,33 +1444,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
-- 
2.14.1

Re: [PATCH] Fixing package-initialize, adding early init file

[John Wiegley]

>>>>> "RR" == Radon Rosborough <radon.neon@gmail.com> writes:

RR> Done. New patch attached.

This seems like a lot of changes that add new complications to the startup of
Emacs. Since I don't have time just now to review those 150 messages, may I
please ask for an executive summary to motivate this change?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> This seems like a lot of changes that add new complications to the
> startup of Emacs. Since I don't have time just now to review those
> 150 messages, may I please ask for an executive summary to motivate
> this change?

You could read [1] for an explanation of what the problem is, and [2]
for a list of the different ways people have proposed to solve it. But
I will summarize again:

* Currently, Emacs modifies the init-file to add (package-initialize)
  to it when Emacs starts. This has many disadvantages [1] which I'm
  not going to rehash here.
* There are many proposals [2] on how to make package.el work out of
  the box without requiring Emacs to modify the init-file. These all
  have practical disadvantages, except for the proposal to add an
  early init file, which solves the package.el problem with
  essentially no disadvantages, and furthermore solves several other
  problems at the same time (for example, people having no way to
  disable the menu bar before it is initialized the first time).

You can disagree with the above two points, but they were the outcome
of the aforementioned 150 emails, and everybody seems to agree that
this is the best approach. The only difficulty is actually making the
relevant changes to startup.el, which are honestly pretty small in the
grand scheme of things.

> new complications

The only new code is to load an additional init-file. It's definitely
a new complication, but I don't think it's that bad. The other changes
to startup.el are to add a new variable, and move the
(package-initialize) call and the check for an invalid user name to a
bit earlier. Sure, we need to test it thoroughly, but it's certainly
not *complicated*.

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
[2]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Mark Oteiza]

Radon Rosborough <radon.neon@gmail.com> writes:

>> This seems like a lot of changes that add new complications to the
>> startup of Emacs. Since I don't have time just now to review those
>> 150 messages, may I please ask for an executive summary to motivate
>> this change?
>
> You could read [1] for an explanation of what the problem is, and [2]
> for a list of the different ways people have proposed to solve it. But
> I will summarize again:
>
> * Currently, Emacs modifies the init-file to add (package-initialize)
>   to it when Emacs starts. This has many disadvantages [1] which I'm
>   not going to rehash here.
> * There are many proposals [2] on how to make package.el work out of
>   the box without requiring Emacs to modify the init-file. These all
>   have practical disadvantages, except for the proposal to add an
>   early init file, which solves the package.el problem

I haven't seen one place where the problem_s_ involved have
been clearly articulated, and still this patch does not improve any of
the places where user facing documentation was lacking in the first
place, and still requires the user to not only understand the forms that
must be in an init file, but now there is one more init file for the
user to manage.

The whole discussion and push for a "solution" is predicated on users
who can't be bothered to do things correctly _not_ having to do any
configuration.  This patch does not improve things in this regard, and
frankly I don't see that predicate as a good one for devising a
solution. At least it explains how we ended up with
package--ensure-init-file.

> with
>   essentially no disadvantages, and furthermore solves several other
>   problems at the same time (for example, people having no way to
>   disable the menu bar before it is initialized the first time).
>
> You can disagree with the above two points, but they were the outcome
> of the aforementioned 150 emails, and everybody seems to agree that
> this is the best approach. The only difficulty is actually making the
> relevant changes to startup.el, which are honestly pretty small in the
> grand scheme of things.

I suppose it's time I pipe up and indicate I vehemently oppose this
change.  AIUI it still breaks existing config of package-archives,
package-load-list, etc, which is something the incumbent solution also
breaks.  The only proposal I can support at this time is reverting the
incumbent solution, yet none from the maintainership is willing to
entertain the idea.

>> new complications
>
> The only new code is to load an additional init-file. It's definitely
> a new complication, but I don't think it's that bad.

I think adding another init file is unacceptable.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> I haven't seen one place where the problem_s_ involved have been
> clearly articulated

As for the problems with the current situation, check out [1] and also
the section for Proposal A in [2]. But again, I'll re-summarize:

* Emacs modifies the init-file automatically, without asking the user.
* If the user has set any variables related to package.el in their
  init-file, then Emacs will modify the init-file to do the wrong
  thing!

Unless by "problems involved" you meant more generally, for all the
different proposals? That is exactly what [2] was intended to cover;
could you clarify what exactly you would like articulated that hasn't
been already?

> this patch does not improve any of the places where user facing
> documentation was lacking in the first place

Actually, I'd disagree. I think this patch improves the situation, not
by adding more documentation, but by reducing complexity and therefore
reducing the need for additional documentation.

In particular, we no longer need to document the placement of
(package-initialize) in the init-file, as such placement is no longer
necessary.

> requires the user to not only understand the forms that must be in
> an init file

Could you clarify what you mean by this? This patch has the effect
that users can put package configuration right into their init-file,
or use Custom to achieve the same, without having to know anything
about the package system.

If on the other hand "forms" is referring to 'setq' forms that
customize the package system itself, then indeed, users must know to
put them into a different init-file. However, I'd argue that this
isn't much worse than the current situation, where users must know to
put them *before* (package-initialize), and they must also know that
Emacs will put (package-initialize) in the wrong place with regard to
these customizations.

> The whole discussion and push for a "solution" is predicated on
> users who can't be bothered to do things correctly _not_ having to
> do any configuration. This patch does not improve things in this
> regard

Correct. The current behavior is that things work out of the box, and
this patch *maintains* that behavior. Since things already work out of
the box, I don't see how you expect this patch to somehow improve that
particular aspect any further.

What this patch accomplishes is eliminating the problems inherent in
automatic modification of the init-file, *without* losing the goal of
the package system working out of the box.

> The whole discussion and push for a "solution" is predicated on
> users who can't be bothered to do things correctly _not_ having to
> do any configuration [...] and frankly I don't see that predicate as
> a good one for devising a solution.

The goal is that when someone uses M-x package-install to install a
package, and then they paste some Lisp code from the project's README
into their init-file, then it works as expected. I think this is
desirable behavior, and I think most everyone else thinks this is
desirable behavior as well. That's why the current solution was put in
in the first place, because people really wanted that use case to
"just work". This patch achieves that same goal in a better way.

> AIUI it still breaks existing config of package-archives,
> package-load-list, etc, which is something the incumbent solution
> also breaks.

Correct, but here is the important point: it will only do so *once*,
when people upgrade to Emacs 26. As opposed to the current solution,
which breaks existing config every time (package-initialize) is
inserted, which may happen many times for a given user. So this patch
is a definite improvement.

> The only proposal I can support at this time is reverting the
> incumbent solution, yet none from the maintainership is willing to
> entertain the idea.

That's because it would result in the built-in package manager not
working out of the box, which would be embarrassing from a UX
perspective. That's the whole *point* of having a built-in package
manager: that it works right away without additional setup.

> I think adding another init file is unacceptable.

Why? (Especially given that being able to run code before the first
graphical frame is initialized would be super useful for lots of
reasons, entirely separate from the package system.)

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
[2]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Mark Oteiza]

On 25/09/17 at 10:16pm, Radon Rosborough wrote:
> > I haven't seen one place where the problem_s_ involved have been
> > clearly articulated
> 
> Unless by "problems involved" you meant more generally, for all the
> different proposals? That is exactly what [2] was intended to cover;
> could you clarify what exactly you would like articulated that hasn't
> been already?

I meant more explicitly--each of these use cases should be documented
with examples.  While the manual in its current state does explain
things, it can be better.

> > this patch does not improve any of the places where user facing
> > documentation was lacking in the first place
> 
> Actually, I'd disagree. I think this patch improves the situation, not
> by adding more documentation, but by reducing complexity and therefore
> reducing the need for additional documentation.
> 
> In particular, we no longer need to document the placement of
> (package-initialize) in the init-file, as such placement is no longer
> necessary.

The only documentation I see is in NEWS and the manual, which is where
the old documentation was.

> > requires the user to not only understand the forms that must be in
> > an init file
> 
> Could you clarify what you mean by this? This patch has the effect
> that users can put package configuration right into their init-file,
> or use Custom to achieve the same, without having to know anything
> about the package system.

At the cost of users who customize package.el and don't need another
init file.

> If on the other hand "forms" is referring to 'setq' forms that
> customize the package system itself, then indeed, users must know to
> put them into a different init-file. However, I'd argue that this
> isn't much worse than the current situation, where users must know to
> put them *before* (package-initialize) 

The current situation should never have happened.

> > The whole discussion and push for a "solution" is predicated on
> > users who can't be bothered to do things correctly _not_ having to
> > do any configuration. This patch does not improve things in this
> > regard
> 
> Correct. The current behavior is that things work out of the box

Except for users who who don't need garbage dumped into their init file
and for users who change their package archives or the list of activated
packages in their init.el without the need of a package-initialize form
because they read the docs.

Apparently changing those custom variables is an edge case.
https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00209.html

> and
> this patch *maintains* that behavior.

which is why it gets a -1 from me.

> > AIUI it still breaks existing config of package-archives,
> > package-load-list, etc, which is something the incumbent solution
> > also breaks.
> 
> Correct, but here is the important point: it will only do so *once*,
> when people upgrade to Emacs 26. As opposed to the current solution,
> which breaks existing config every time (package-initialize) is
> inserted, which may happen many times for a given user. So this patch
> is a definite improvement.

An improvement from package--ensure-init-file is great, but that's still
more breakage than prior to package--ensure-init-file.

> > The only proposal I can support at this time is reverting the
> > incumbent solution, yet none from the maintainership is willing to
> > entertain the idea.
> 
> That's because it would result in the built-in package manager not
> working out of the box, which would be embarrassing from a UX
> perspective. That's the whole *point* of having a built-in package
> manager: that it works right away without additional setup.

Emacs itself doesn't work OOTB for most people.  That it's customizable
in Elisp is a feature.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> > > I haven't seen one place where the problem_s_ involved have been
> > > clearly articulated
> >
> > Unless by "problems involved" you meant more generally, for all the
> > different proposals? That is exactly what [2] was intended to cover;
> > could you clarify what exactly you would like articulated that hasn't
> > been already?
>
> I meant more explicitly--each of these use cases should be documented
> with examples.  While the manual in its current state does explain
> things, it can be better.

So you're complaining about the documentation? Fine, agreed. The
documentation can/should be improved. That is totally orthogonal to
this patch, though.

> > > this patch does not improve any of the places where user facing
> > > documentation was lacking in the first place
> >
> > Actually, I'd disagree. I think this patch improves the situation, not
> > by adding more documentation, but by reducing complexity and therefore
> > reducing the need for additional documentation.
> >
> > In particular, we no longer need to document the placement of
> > (package-initialize) in the init-file, as such placement is no longer
> > necessary.
>
> The only documentation I see is in NEWS and the manual, which is where
> the old documentation was.

So, again, you're complaining about the documentation? Again, fine,
agreed. But I don't think that's a reason to criticize this patch.
More documentation can be added in future patches; let's not confuse
matters by combining this change with general doc improvements.

> > > requires the user to not only understand the forms that must be in
> > > an init file
> >
> > Could you clarify what you mean by this? This patch has the effect
> > that users can put package configuration right into their init-file,
> > or use Custom to achieve the same, without having to know anything
> > about the package system.
>
> At the cost of users who customize package.el and don't need another
> init file.

Right, so we benefit the 100% (people who customize packages) at the
cost of the 0.1% (people who need to customize package.el in the early
init-file). A win-win, no?

And it's still not clear to me why you think having an early init-file
is such a big problem.

> package archives

If I understand correctly, package-archives need not be set before
package-initialize is called. Thus the number of people who would need
to use the early init-file is quite small indeed.

> > and this patch *maintains* that behavior.
>
> which is why it gets a -1 from me.

The behavior I was referring to was "the built-in package manager
works out of the box". Are you really saying that having the built-in
package manager work out of the box is a bad thing?

> An improvement from package--ensure-init-file is great, but that's still
> more breakage than prior to package--ensure-init-file.

Prior to package--ensure-init-file, when people pasted Lisp code from
their packages' READMEs into their init-file, they would get errors.
This was a very common problem for people new to Emacs, see [1]. That
affects everyone, so it should be considered "major breakage".

> Emacs itself doesn't work OOTB for most people.

I think we all agree that this is a bad thing. Shouldn't we be trying
to remedy this problem?

> That it's customizable in Elisp is a feature.

Yeah, but having sane defaults gets more and more important the more
powerful your tool gets.

[1]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> Date: Mon, 25 Sep 2017 19:15:03 -0400
> From: Mark Oteiza <mvoteiza@udel.edu>
> Cc: jwiegley@gmail.com, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	emacs-devel@gnu.org
> 
> On 25/09/17 at 10:16pm, Radon Rosborough wrote:
> > > I haven't seen one place where the problem_s_ involved have been
> > > clearly articulated
> > 
> > Unless by "problems involved" you meant more generally, for all the
> > different proposals? That is exactly what [2] was intended to cover;
> > could you clarify what exactly you would like articulated that hasn't
> > been already?
> 
> I meant more explicitly--each of these use cases should be documented
> with examples.  While the manual in its current state does explain
> things, it can be better.

It definitely can, but that's a separate issue, I think.  And we
cannot finalize the documentation before we finalize the code.

> > Could you clarify what you mean by this? This patch has the effect
> > that users can put package configuration right into their init-file,
> > or use Custom to achieve the same, without having to know anything
> > about the package system.
> 
> At the cost of users who customize package.el and don't need another
> init file.

Do those users have a "fire escape"?  If they don't, I agree we should
provide one.  Otherwise, I expect such users to be knowledgeable
enough to work around these changes with minimal hassle.

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> BTW, in a subsequent patch we could rename this to
>> package-initialized, so the name reflects what the variable contains
>> rather than what it's used for.
> I don't think that's a good idea, since the most common usage of this
> variable will be
>     (setq package-initialized nil)
> in the early init file, and that doesn't make much sense from a
> readability perspective (are we asserting that package.el wasn't
> initialized?).

Why would there be (setq package-initialized nil) in the early init file?
I could imagine there being

    (setq package-initialized t)

OTOH, to inhibit subsequent initialization of package.el.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> Why would there be (setq package-initialized nil) in the early init file?
> I could imagine there being
>
>     (setq package-initialized t)
>
> OTOH, to inhibit subsequent initialization of package.el.

Whoops, that was indeed what I meant to say. But I still argue that
the readability is hurt, because (setq package-initialized t) is
basically a lie: you're saying that package.el was initialized
already, when in fact the purpose is quite possibly to ensure that
package.el is *never* initialized.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

I noticed that discussion seems to have died off on the
package-initialize patch. Is there anything I can do to
help it along?

> Attached is a preliminary patch for fixing "the package-initialize
> problem" (see [1] [2] [3]) by adding an early init file. Feedback
> welcome. In particular,
>
> - do we want to try to factor out some common logic in loading the
>   early init file versus the regular init file?
> - have I broken anything in moving the check for an invalid username
>   earlier in startup.el? what about moving package-initialize earlier?
> - should `early-init-file' be defined in C?
> - did I miss any documentation fixes?
> - have I broken any style guidelines for the repository?
>
> Additional question: if we moved forward with this patch, would it
> make it into Emacs 26.1?
>
> Best,
> Radon Rosborough
>
> [1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
> [2]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
> [3]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Mon, Sep 18, 2017 at 5:57 PM, Radon Rosborough <radon.neon@gmail.com>
wrote:

> Hello all,
>
> Attached is a preliminary patch for fixing "the package-initialize
> problem" (see [1] [2] [3]) by adding an early init file.
>

​My thoughts are:

1. For most Emacs users who do not program in Elisp, there should be only a
single init file that they would personalize, generally as they find
snippets of code they can cut and paste for their use.​

2. For Elispers, another optional init file for more fine-grained control
would be okay, though not ideal.

But this means whatever problem exists with package-initialize, it should
be solved without requiring the use of a 2nd init file.

Bob

[-- Attachment #2: Type: text/html, Size: 1972 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Robert Weiner <rsw@gnu.org>
> Date: Tue, 10 Oct 2017 12:52:32 -0400
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> 1. For most Emacs users who do not program in Elisp, there should be only a single init file that they would
> personalize, generally as they find snippets of code they can cut and paste for their use.​
> 
> 2. For Elispers, another optional init file for more fine-grained control would be okay, though not ideal.

I think the proposed solution satisfies these requirements.  The 2nd
init file is only needed if you want to tweak package initialization
by putting Lisp code into that 2nd file.

> But this means whatever problem exists with package-initialize, it should be solved without requiring the use
> of a 2nd init file.

I don't understand how you reach this conclusion.  The problems we are
trying to solve exists only for those whom you call "Lispers", so
another init file is only needed in that case.  What am I missing?

Re: [PATCH] Fixing package-initialize, adding early init file

[Mark Oteiza]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Weiner <rsw@gnu.org>
>> Date: Tue, 10 Oct 2017 12:52:32 -0400
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> 
>> 1. For most Emacs users who do not program in Elisp, there should be
>> only a single init file that they would personalize, generally as
>> they find snippets of code they can cut and paste for their use.​
>> 
>> 2. For Elispers, another optional init file for more fine-grained
>> control would be okay, though not ideal.
>
> I think the proposed solution satisfies these requirements.  The 2nd
> init file is only needed if you want to tweak package initialization
> by putting Lisp code into that 2nd file.

Breaking the ability to do such tweaking with the init file we already
have is insane.

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> Breaking the ability to do such tweaking with the init file we already
> have is insane.

FWIW this second init file has another purpose: it lets you run
configuration code before the initial frame is mapped.


        Stefan "who intends to use a single config file, which will be
                called ~/.emacs.d/early-init.el"

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

> Breaking the ability to do such tweaking with the init file we
> already have is insane.

It is equally legitimate for me to say that all of the other proposed
solutions are insane. Please provide some reasoning to substantiate
your claim, since I have already provided ample reasoning to
substantiate my claim to the contrary (see [1]).

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

[-- Attachment #2: Type: text/html, Size: 651 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Mark Oteiza <mvoteiza@udel.edu>
> Cc: rswgnu@gmail.com,  radon.neon@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Oct 2017 17:30:41 -0400
> 
> > I think the proposed solution satisfies these requirements.  The 2nd
> > init file is only needed if you want to tweak package initialization
> > by putting Lisp code into that 2nd file.
> 
> Breaking the ability to do such tweaking with the init file we already
> have is insane.

I hope the "insane" part is a slight exaggeration...

Anyway, AFAIU, that was the best/only solution that came up which
would solve the original problem without causing more trouble to the
uninitiated.

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> > I think the proposed solution satisfies these requirements.  The 2nd
>> > init file is only needed if you want to tweak package initialization
>> > by putting Lisp code into that 2nd file.
>> Breaking the ability to do such tweaking with the init file we already
>> have is insane.

Actually, it doesn't really break it.  All it does really (assuming you
don't use the new early-init.el file) is that packages in
~/.emacs.d/elpa will be automatically activated before ~/.emacs
is loaded.

So if you set package-user-dir in your ~/.emacs presumably you don't
have any packages in ~/.emacs.d/elpa and hence it won't make much of
a difference: you can still set package-user-dir in your ~/.emacs and
then call package-initialize.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

Sorry to be a nag, but discussion seems to have died off again.
Again, is there anything I can do to help out with the process of
reviewing this patch for merge?

> Attached is a preliminary patch for fixing "the package-initialize
> problem" (see [1] [2] [3]) by adding an early init file. Feedback
> welcome. In particular,
>
> - do we want to try to factor out some common logic in loading the
>   early init file versus the regular init file?
> - have I broken anything in moving the check for an invalid username
>   earlier in startup.el? what about moving package-initialize earlier?
> - should `early-init-file' be defined in C?
> - did I miss any documentation fixes?
> - have I broken any style guidelines for the repository?
>
> Additional question: if we moved forward with this patch, would it
> make it into Emacs 26.1?
>
> Best,
> Radon Rosborough
>
> [1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
> [2]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
> [3]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> - do we want to try to factor out some common logic in loading the
>> early init file versus the regular init file?

I think so, yes.

>> - have I broken anything in moving the check for an invalid username
>> earlier in startup.el? what about moving package-initialize earlier?

Time will tell.

>> - should `early-init-file' be defined in C?

If it's not indispensable, then I'd say no.

>> Additional question: if we moved forward with this patch, would it
>> make it into Emacs 26.1?

No, it's too late for that.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

I am sorry for the delay in response. Now that the fall semester is
over, I have much more time. The revised patch is attached. I have
done some basic testing with it applied to the latest master. Please
let me know if you have more feedback.

> > - do we want to try to factor out some common logic in loading the
> > early init file versus the regular init file?
>
> I think so, yes.

This is done; the function `load-user-init-file' is used for loading
both the early and regular init-files, and could easily be used for
loading more.

> > - should `early-init-file' be defined in C?
>
> If it's not indispensable, then I'd say no.

In the attached patch, it is defined as a Lisp variable in startup.el.

Best,
Radon

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 33786 bytes --]

From c74093ab3dde9042a50eb65e7af46625a834b689 Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable, for the filename of
  the early init file after it has been loaded.

* lisp/startup.el (load-user-init-file): New function, used to
  eliminate duplicate code in loading the early and regular init
  files.

* lisp/startup.el (command-line): Load the early init file using
  `load-user-init-file'. Move the check for an invalid username to
  just before that, and move the initialization of the package system
  to just after. Load the regular init file using
  `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
  usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
  definition, usage, and documentation.

* lisp/emacs-lisp/package.el (package--init-file-ensured): Remove
  definition and usage.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
  package-initialize is called, and advise against calling it in the
  init file.

* doc/emacs/package.texi: Document changes to when package-initialize
  is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
  init file.

* etc/NEWS: Document changes to startup and package.el.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/package.texi     |  28 +--
 doc/lispref/os.texi        |  14 ++
 doc/lispref/package.texi   |  16 +-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  18 ++
 lisp/emacs-lisp/package.el |  71 +-------
 lisp/startup.el            | 425 +++++++++++++++++++++++++--------------------
 src/lread.c                |   2 +-
 8 files changed, 289 insertions(+), 289 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 215f50cb40..bd7cccce6c 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -253,30 +253,16 @@ Package Installation
 consult the package's help buffer.
 
   By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+subsequent Emacs sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 501960bdc3..c6990909ab 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, here you can customize the process
+  of loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 153ee48741..2e93f29bd2 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the early init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is @code{nil}.  (Of course, the
+setting of this user option must be done in the early init file.)
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index 1f6e10287d..36b7f87248 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 1382f96a37..4bbf3e237a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -41,6 +41,24 @@ can enable it when configuring, e.g., './configure CFLAGS="-g3 -O2
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called early-init.el, in `user-emacs-directory'.  It is
+loaded very early in the startup process: in particular, before
+graphical elements such as the tool bar are initialized, and before
+the package manager is initialized.
+
++++
+** Emacs now initializes package.el before loading the init-file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to (package-initialize) into the
+init-file, which was previously done when Emacs was started.  Users
+who do not configure package.el variables such as `package-load-list'
+and `package-user-dir' need not make any configuration changes.  Users
+who do configure such variables should place the configuration into
+the newly introduced early init file, which is loaded before
+package.el is initialized.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f8b4cc888d..815b679cae 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 4575f1f94d..de85933983 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,15 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+If the file loaded had extension `.elc', and the corresponding
+source file exists, this variable contains the name of source
+file, suitable for use by functions like `custom-save-all' which
+edit the init file.  While Emacs loads and evaluates the init
+file, value is the real name of the file, regardless of whether
+or not it has the `.elc' extension.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +879,129 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (compute-filename &optional compute-alternate-filename load-defaults)
+  "Load a user init-file.
+COMPUTE-FILENAME is called with no arguments and should return
+the name of the init-file to load. If this file cannot be loaded,
+and COMPUTE-ALTERNATE-FILENAME is non-nil, then it is called with
+no arguments and should return the name of an alternate init-file
+to load. If LOAD-DEFAULTS is non-nil, then load default.el after
+the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug))
+        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall compute-filename)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (eq user-init-file t)
+                   (let ((alt-file-name (funcall compute-alternate-filename)))
+                     (load alt-file-name 'noerror 'nomessage)
+
+                     ;; If we did not find the user's init file, set
+                     ;; user-init-file conclusively.  Don't let it be
+                     ;; set from default.el.
+                     (when (eq user-init-file t)
+                       (setq user-init-file init-file-name)))))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (progn
+              (funcall read-init-file)
+
+              ;; If a previous init-file had an error, don't forget
+              ;; about that.
+              (unless init-file-had-error
+                (setq init-file-had-error nil)))
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))
+
+    (unless (or (default-value 'enable-multibyte-characters)
+                (eq orig-enable-multibyte (default-value
+                                            'enable-multibyte-characters)))
+
+      ;; Init file changed to unibyte.  Reset existing multibyte
+      ;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
+      ;; Arguably this should only be done if they're free of
+      ;; multibyte characters.
+      (mapc (lambda (buffer)
+              (with-current-buffer buffer
+                (if enable-multibyte-characters
+                    (set-buffer-multibyte nil))))
+            (buffer-list))
+
+      ;; Also re-set the language environment in case it was
+      ;; originally done before unibyte was set and is sensitive to
+      ;; unibyte (display table, terminal coding system &c).
+      (set-language-environment current-language-environment))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1153,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1127,170 +1322,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug))
-	  (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file))
-      (unless (or (default-value 'enable-multibyte-characters)
-		  (eq orig-enable-multibyte (default-value
-					      'enable-multibyte-characters)))
-	;; Init file changed to unibyte.  Reset existing multibyte
-	;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
-	;; Arguably this should only be done if they're free of
-	;; multibyte characters.
-	(mapc (lambda (buffer)
-		(with-current-buffer buffer
-		  (if enable-multibyte-characters
-		      (set-buffer-multibyte nil))))
-	      (buffer-list))
-	;; Also re-set the language environment in case it was
-	;; originally done before unibyte was set and is sensitive to
-	;; unibyte (display table, terminal coding system &c).
-	(set-language-environment current-language-environment)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1312,33 +1384,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 52897b4fcc..5ff438041b 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4892,7 +4892,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.14.3

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

Since it has been more than a month with no response, I am re-posting
the patch which fixes the problems previously discussed [1] [2] [3]
with `package-initialize' by adding a second (optional) init-file
`early-init.el'. This patch includes some refactoring of the code in
`startup.el' to provide for loading any number of init-files in an
extensible way. I hope that the change can be included in Emacs 26.2
or Emacs 27.

Feedback much appreciated.

Best,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
[2]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
[3]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 33786 bytes --]

From c74093ab3dde9042a50eb65e7af46625a834b689 Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable, for the filename of
  the early init file after it has been loaded.

* lisp/startup.el (load-user-init-file): New function, used to
  eliminate duplicate code in loading the early and regular init
  files.

* lisp/startup.el (command-line): Load the early init file using
  `load-user-init-file'. Move the check for an invalid username to
  just before that, and move the initialization of the package system
  to just after. Load the regular init file using
  `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
  usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
  definition, usage, and documentation.

* lisp/emacs-lisp/package.el (package--init-file-ensured): Remove
  definition and usage.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
  package-initialize is called, and advise against calling it in the
  init file.

* doc/emacs/package.texi: Document changes to when package-initialize
  is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
  init file.

* etc/NEWS: Document changes to startup and package.el.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/package.texi     |  28 +--
 doc/lispref/os.texi        |  14 ++
 doc/lispref/package.texi   |  16 +-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  18 ++
 lisp/emacs-lisp/package.el |  71 +-------
 lisp/startup.el            | 425 +++++++++++++++++++++++++--------------------
 src/lread.c                |   2 +-
 8 files changed, 289 insertions(+), 289 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 215f50cb40..bd7cccce6c 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -253,30 +253,16 @@ Package Installation
 consult the package's help buffer.
 
   By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+subsequent Emacs sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 501960bdc3..c6990909ab 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, here you can customize the process
+  of loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 153ee48741..2e93f29bd2 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the early init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is @code{nil}.  (Of course, the
+setting of this user option must be done in the early init file.)
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index 1f6e10287d..36b7f87248 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 1382f96a37..4bbf3e237a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -41,6 +41,24 @@ can enable it when configuring, e.g., './configure CFLAGS="-g3 -O2
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called early-init.el, in `user-emacs-directory'.  It is
+loaded very early in the startup process: in particular, before
+graphical elements such as the tool bar are initialized, and before
+the package manager is initialized.
+
++++
+** Emacs now initializes package.el before loading the init-file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to (package-initialize) into the
+init-file, which was previously done when Emacs was started.  Users
+who do not configure package.el variables such as `package-load-list'
+and `package-user-dir' need not make any configuration changes.  Users
+who do configure such variables should place the configuration into
+the newly introduced early init file, which is loaded before
+package.el is initialized.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f8b4cc888d..815b679cae 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 4575f1f94d..de85933983 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,15 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+If the file loaded had extension `.elc', and the corresponding
+source file exists, this variable contains the name of source
+file, suitable for use by functions like `custom-save-all' which
+edit the init file.  While Emacs loads and evaluates the init
+file, value is the real name of the file, regardless of whether
+or not it has the `.elc' extension.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +879,129 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (compute-filename &optional compute-alternate-filename load-defaults)
+  "Load a user init-file.
+COMPUTE-FILENAME is called with no arguments and should return
+the name of the init-file to load. If this file cannot be loaded,
+and COMPUTE-ALTERNATE-FILENAME is non-nil, then it is called with
+no arguments and should return the name of an alternate init-file
+to load. If LOAD-DEFAULTS is non-nil, then load default.el after
+the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug))
+        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall compute-filename)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (eq user-init-file t)
+                   (let ((alt-file-name (funcall compute-alternate-filename)))
+                     (load alt-file-name 'noerror 'nomessage)
+
+                     ;; If we did not find the user's init file, set
+                     ;; user-init-file conclusively.  Don't let it be
+                     ;; set from default.el.
+                     (when (eq user-init-file t)
+                       (setq user-init-file init-file-name)))))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (progn
+              (funcall read-init-file)
+
+              ;; If a previous init-file had an error, don't forget
+              ;; about that.
+              (unless init-file-had-error
+                (setq init-file-had-error nil)))
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))
+
+    (unless (or (default-value 'enable-multibyte-characters)
+                (eq orig-enable-multibyte (default-value
+                                            'enable-multibyte-characters)))
+
+      ;; Init file changed to unibyte.  Reset existing multibyte
+      ;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
+      ;; Arguably this should only be done if they're free of
+      ;; multibyte characters.
+      (mapc (lambda (buffer)
+              (with-current-buffer buffer
+                (if enable-multibyte-characters
+                    (set-buffer-multibyte nil))))
+            (buffer-list))
+
+      ;; Also re-set the language environment in case it was
+      ;; originally done before unibyte was set and is sensitive to
+      ;; unibyte (display table, terminal coding system &c).
+      (set-language-environment current-language-environment))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1153,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1127,170 +1322,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug))
-	  (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file))
-      (unless (or (default-value 'enable-multibyte-characters)
-		  (eq orig-enable-multibyte (default-value
-					      'enable-multibyte-characters)))
-	;; Init file changed to unibyte.  Reset existing multibyte
-	;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
-	;; Arguably this should only be done if they're free of
-	;; multibyte characters.
-	(mapc (lambda (buffer)
-		(with-current-buffer buffer
-		  (if enable-multibyte-characters
-		      (set-buffer-multibyte nil))))
-	      (buffer-list))
-	;; Also re-set the language environment in case it was
-	;; originally done before unibyte was set and is sensitive to
-	;; unibyte (display table, terminal coding system &c).
-	(set-language-environment current-language-environment)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1312,33 +1384,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 52897b4fcc..5ff438041b 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4892,7 +4892,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.14.3

Re: [PATCH] Fixing package-initialize, adding early init file

[Clément Pit-Claudel]

On 2018-01-24 23:35, Radon Rosborough wrote:
> Since it has been more than a month with no response, I am re-posting
> the patch which fixes the problems previously discussed [1] [2] [3]
> with `package-initialize' by adding a second (optional) init-file
> `early-init.el'. This patch includes some refactoring of the code in
> `startup.el' to provide for loading any number of init-files in an
> extensible way. I hope that the change can be included in Emacs 26.2
> or Emacs 27.

Thanks for this patch. I think it'd be great to have it.  I've added some (very few) comments below.

> diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
> index 215f50cb40..bd7cccce6c 100644
> --- a/doc/emacs/package.texi
> +++ b/doc/emacs/package.texi
> @@ -253,30 +253,16 @@ Package Installation
>  consult the package's help buffer.
>  
>    By default, Emacs also automatically loads all installed packages in
> -subsequent Emacs sessions.  This happens at startup, after processing
> -the init file (@pxref{Init File}).  As an exception, Emacs does not
> -load packages at startup if invoked with the @samp{-q} or
> -@samp{--no-init-file} options (@pxref{Initial Options}).
> +subsequent Emacs sessions.  This happens at startup, before processing

Was this supposed to be "previous" rather than "subsequent" (this isn't from your patch, though).

> +the init file but after processing the early init file (@pxref{Early
> +Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
> +exception, Emacs does not load packages at startup if invoked with the
> +@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
>  
>  @vindex package-enable-at-startup
>    To disable automatic package loading, change the variable
> -@code{package-enable-at-startup} to @code{nil}.
> -
> -@findex package-initialize
> -  The reason automatic package loading occurs after loading the init
> -file is that user options only receive their customized values after
> -loading the init file, including user options which affect the
> -packaging system.  In some circumstances, you may want to load
> -packages explicitly in your init file (usually because some other code
> -in your init file depends on a package).  In that case, your init file
> -should call the function @code{package-initialize}.  It is up to you
> -to ensure that relevant user options, such as @code{package-load-list}
> -(see below), are set up prior to the @code{package-initialize} call.
> -This will automatically set @code{package-enable-at-startup} to @code{nil}, to
> -avoid loading the packages again after processing the init file.
> -Alternatively, you may choose to completely inhibit package loading at
> -startup, and invoke the command @kbd{M-x package-initialize} to load
> -your packages manually.
> +@code{package-enable-at-startup} to @code{nil}.  You must do this in
> +the early init file.  Currently it cannot be done via Customize.

I'd add ", as this variable is read before loading the regular init file." after "in the early init file".

>  @vindex package-load-list
>    For finer control over package loading, you can use the variable
> diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
> index 501960bdc3..c6990909ab 100644
> --- a/doc/lispref/os.texi
> +++ b/doc/lispref/os.texi
> @@ -361,6 +361,7 @@ Init File
>  @cindex init file
>  @cindex @file{.emacs}
>  @cindex @file{init.el}
> +@cindex @file{early-init.el}
>  
>    When you start Emacs, it normally attempts to load your @dfn{init
>  file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
> @@ -384,6 +385,19 @@ Init File
>  file.  If those environment variables are absent, though, Emacs uses
>  your user-id to find your home directory.
>  
> +@cindex early init file
> +  Emacs also attempts to load a second init file, called the
> +  @dfn{early init file}, if it exists.  This is a file named
> +  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
> +  home directory.  The difference is that the early init file is
> +  loaded much earlier during the startup process, so you can use it to
> +  customize some things that are initialized before loading the
> +  regular init file.  For example, here you can customize the process

I'd say "For example, you can use that file to customize the process"

> +  of loading installed packages, by setting variables such as
> +  @var{package-load-list} or
> +  @var{package-enable-at-startup}. @xref{Package Installation,,,
> +  emacs,The GNU Emacs Manual}.
> +
>  @cindex default init file
>    An Emacs installation may have a @dfn{default init file}, which is a
>  Lisp library named @file{default.el}.  Emacs finds this file through
> diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
> index 153ee48741..2e93f29bd2 100644
> --- a/doc/lispref/package.texi
> +++ b/doc/lispref/package.texi
> @@ -106,10 +106,12 @@ Packaging Basics
>  
>    Whenever Emacs starts up, it automatically calls the function
>  @code{package-initialize} to load installed packages.  This is done
> -after loading the init file and abbrev file (if any) and before
> -running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
> -package loading is disabled if the user option
> -@code{package-enable-at-startup} is @code{nil}.
> +after loading the early init file, but before loading the early init

This should say "but before loading the *reguar* init file."

> +file and abbrev file (if any) and before running
> +@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
> +loading is disabled if the user option
> +@code{package-enable-at-startup} is @code{nil}.  (Of course, the
> +setting of this user option must be done in the early init file.)

I'd simplify this to "if the option … is *set to* nil in the early init file."

>  @deffn Command package-initialize &optional no-activate
>  This function initializes Emacs' internal record of which packages are
> @@ -123,6 +125,12 @@ Packaging Basics
>  The optional argument @var{no-activate}, if non-@code{nil}, causes
>  Emacs to update its record of installed packages without actually
>  loading them; it is for internal use only.
> +
> +In most cases, you should not need to call @code{package-initialize},
> +as this is done automatically during startup.  Simply make sure to put
> +any code that should run before @code{package-initialize} in the early
> +init file, and any code that should run after in the primary init

after *it*, maybe?

> +file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
>  @end deffn
>  
>  @node Simple Packages
> diff --git a/doc/misc/org.texi b/doc/misc/org.texi
> index 1f6e10287d..36b7f87248 100644
> --- a/doc/misc/org.texi
> +++ b/doc/misc/org.texi
> @@ -890,9 +890,7 @@ Installation
>  been visited, i.e., where no Org built-in function have been loaded.
>  Otherwise autoload Org functions will mess up the installation.
>  
> -Then, to make sure your Org configuration is taken into account, initialize
> -the package system with @code{(package-initialize)} in your Emacs init file
> -before setting any Org option.  If you want to use Org's package repository,
> +If you want to use Org's package repository,
>  check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
>  
>  @subsubheading Downloading Org as an archive
> diff --git a/etc/NEWS b/etc/NEWS
> index 1382f96a37..4bbf3e237a 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -41,6 +41,24 @@ can enable it when configuring, e.g., './configure CFLAGS="-g3 -O2
>  \f
>  * Startup Changes in Emacs 27.1
>  
> ++++
> +** Emacs can now be configured using an early init file.
> +The file is called early-init.el, in `user-emacs-directory'.  It is
> +loaded very early in the startup process: in particular, before

no need for "in particular," here?

> +graphical elements such as the tool bar are initialized, and before
> +the package manager is initialized.
> +
> ++++
> +** Emacs now initializes package.el before loading the init-file.

I think "calls package-initialize" would be clearer.

> +This is part of a change intended to eliminate the behavior of
> +package.el inserting a call to (package-initialize) into the
> +init-file, which was previously done when Emacs was started.  Users
> +who do not configure package.el variables such as `package-load-list'
> +and `package-user-dir' need not make any configuration changes.  Users
> +who do configure such variables should place the configuration into
> +the newly introduced early init file, which is loaded before
> +package.el is initialized.

I'd mention here that variables like package-archives do not need to move to the early init file.

> +
>  \f
>  * Changes in Emacs 27.1
>  
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index f8b4cc888d..815b679cae 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -1431,16 +1431,11 @@ package-read-all-archive-contents
>  ;; available on disk.
>  (defvar package--initialized nil)
>  
> -(defvar package--init-file-ensured nil
> -  "Whether we know the init file has package-initialize.")
> -
>  ;;;###autoload
>  (defun package-initialize (&optional no-activate)
>    "Load Emacs Lisp packages, and activate them.
>  The variable `package-load-list' controls which packages to load.
>  If optional arg NO-ACTIVATE is non-nil, don't activate packages.
> -If `user-init-file' does not mention `(package-initialize)', add
> -it to the file.
>  If called as part of loading `user-init-file', set
>  `package-enable-at-startup' to nil, to prevent accidentally
>  loading packages twice.
> @@ -1449,13 +1444,7 @@ package-initialize
>  taken care of by `package-initialize'."
>    (interactive)
>    (setq package-alist nil)
> -  (if after-init-time
> -      (package--ensure-init-file)
> -    ;; If `package-initialize' is before we finished loading the init
> -    ;; file, it's obvious we don't need to ensure-init.
> -    (setq package--init-file-ensured t
> -          ;; And likely we don't need to run it again after init.
> -          package-enable-at-startup nil))
> +  (setq package-enable-at-startup nil)
>    (package-load-all-descriptors)
>    (package-read-all-archive-contents)
>    (unless no-activate
> @@ -1872,64 +1861,6 @@ package-download-transaction
>  using `package-compute-transaction'."
>    (mapc #'package-install-from-archive packages))
>  
> -(defun package--ensure-init-file ()
> -  "Ensure that the user's init file has `package-initialize'.
> -`package-initialize' doesn't have to be called, as long as it is
> -present somewhere in the file, even as a comment.  If it is not,
> -add a call to it along with some explanatory comments."
> -  ;; Don't mess with the init-file from "emacs -Q".
> -  (when (and (stringp user-init-file)
> -             (not package--init-file-ensured)
> -             (file-readable-p user-init-file)
> -             (file-writable-p user-init-file))
> -    (let* ((buffer (find-buffer-visiting user-init-file))
> -           buffer-name
> -           (contains-init
> -            (if buffer
> -                (with-current-buffer buffer
> -                  (save-excursion
> -                    (save-restriction
> -                      (widen)
> -                      (goto-char (point-min))
> -                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
> -              ;; Don't visit the file if we don't have to.
> -              (with-temp-buffer
> -                (insert-file-contents user-init-file)
> -                (goto-char (point-min))
> -                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
> -      (unless contains-init
> -        (with-current-buffer (or buffer
> -                                 (let ((delay-mode-hooks t)
> -                                       (find-file-visit-truename t))
> -                                   (find-file-noselect user-init-file)))
> -          (when buffer
> -            (setq buffer-name (buffer-file-name))
> -            (set-visited-file-name (file-chase-links user-init-file)))
> -          (save-excursion
> -            (save-restriction
> -              (widen)
> -              (goto-char (point-min))
> -              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
> -                          (not (eobp)))
> -                (forward-line 1))
> -              (insert
> -               "\n"
> -               ";; Added by Package.el.  This must come before configurations of\n"
> -               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
> -               ";; just comment it out by adding a semicolon to the start of the line.\n"
> -               ";; You may delete these explanatory comments.\n"
> -               "(package-initialize)\n")
> -              (unless (looking-at-p "$")
> -                (insert "\n"))
> -              (let ((file-precious-flag t))
> -                (save-buffer))
> -              (if buffer
> -                  (progn
> -                    (set-visited-file-name buffer-name)
> -                    (set-buffer-modified-p nil))
> -                (kill-buffer (current-buffer)))))))))
> -  (setq package--init-file-ensured t))
> -
>  ;;;###autoload
>  (defun package-install (pkg &optional dont-select)
>    "Install the package PKG.
> diff --git a/lisp/startup.el b/lisp/startup.el
> index 4575f1f94d..de85933983 100644
> --- a/lisp/startup.el
> +++ b/lisp/startup.el
> @@ -312,6 +312,15 @@ inhibit-startup-hooks
>  Currently this applies to: `emacs-startup-hook', `term-setup-hook',
>  and `window-setup-hook'.")
>  
> +(defvar early-init-file nil
> +  "File name, including directory, of user's early init file.
> +If the file loaded had extension `.elc', and the corresponding
> +source file exists, this variable contains the name of source
> +file, suitable for use by functions like `custom-save-all' which
> +edit the init file.  While Emacs loads and evaluates the init
> +file, value is the real name of the file, regardless of whether
> +or not it has the `.elc' extension.")
> +
>  (defvar keyboard-type nil
>    "The brand of keyboard you are using.
>  This variable is used to define the proper function and keypad
> @@ -870,6 +879,129 @@ startup--setup-quote-display
>            (when standard-display-table
>              (aset standard-display-table char nil)))))))
>  
> +(defun load-user-init-file
> +    (compute-filename &optional compute-alternate-filename load-defaults)
> +  "Load a user init-file.
> +COMPUTE-FILENAME is called with no arguments and should return
> +the name of the init-file to load. If this file cannot be loaded,
> +and COMPUTE-ALTERNATE-FILENAME is non-nil, then it is called with
> +no arguments and should return the name of an alternate init-file
> +to load. If LOAD-DEFAULTS is non-nil, then load default.el after
> +the init-file.
> +
> +This function sets `user-init-file' to the name of the loaded
> +init-file, or to a default value if loading is not possible."
> +  (let ((debug-on-error-from-init-file nil)
> +        (debug-on-error-should-be-set nil)
> +        (debug-on-error-initial
> +         (if (eq init-file-debug t)
> +             'startup
> +           init-file-debug))
> +        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
> +    (let ((debug-on-error debug-on-error-initial)
> +          ;; We create an anonymous function here so that we can call
> +          ;; it in different contexts depending on the value of
> +          ;; `debug-on-error'.
> +          (read-init-file
> +           (lambda ()
> +             (when init-file-user
> +               (let ((init-file-name (funcall compute-filename)))
> +
> +                 ;; If `user-init-file' is t, then `load' will store
> +                 ;; the name of the file that it loads into
> +                 ;; `user-init-file'.
> +                 (setq user-init-file t)
> +                 (load init-file-name 'noerror 'nomessage)
> +
> +                 (when (eq user-init-file t)
> +                   (let ((alt-file-name (funcall compute-alternate-filename)))
> +                     (load alt-file-name 'noerror 'nomessage)
> +
> +                     ;; If we did not find the user's init file, set
> +                     ;; user-init-file conclusively.  Don't let it be
> +                     ;; set from default.el.
> +                     (when (eq user-init-file t)
> +                       (setq user-init-file init-file-name)))))
> +
> +               ;; If we loaded a compiled file, set `user-init-file' to
> +               ;; the source version if that exists.
> +               (when (equal (file-name-extension user-init-file)
> +                            "elc")
> +                 (let* ((source (file-name-sans-extension user-init-file))
> +                        (alt (concat source ".el")))
> +                   (setq source (cond ((file-exists-p alt) alt)
> +                                      ((file-exists-p source) source)
> +                                      (t nil)))
> +                   (when source
> +                     (when (file-newer-than-file-p source user-init-file)
> +                       (message "Warning: %s is newer than %s"
> +                                source user-init-file)
> +                       (sit-for 1))
> +                     (setq user-init-file source))))
> +
> +               (when load-defaults
> +
> +                 ;; Prevent default.el from changing the value of
> +                 ;; `inhibit-startup-screen'.
> +                 (let ((inhibit-startup-screen nil))
> +                   (load "default" 'noerror 'nomessage)))))))
> +      ;; Now call our anonymous function.
> +      (if init-file-debug
> +          ;; Do this without a `condition-case' if the user wants to
> +          ;; debug.
> +          (funcall read-init-file)
> +        (condition-case error
> +            (progn
> +              (funcall read-init-file)
> +
> +              ;; If a previous init-file had an error, don't forget
> +              ;; about that.
> +              (unless init-file-had-error
> +                (setq init-file-had-error nil)))
> +          (error
> +           (display-warning
> +            'initialization
> +            (format-message "\
> +An error occurred while loading `%s':\n\n%s%s%s\n\n\
> +To ensure normal operation, you should investigate and remove the
> +cause of the error in your initialization file.  Start Emacs with
> +the `--debug-init' option to view a complete error backtrace."
> +                            user-init-file
> +                            (get (car error) 'error-message)
> +                            (if (cdr error) ": " "")
> +                            (mapconcat (lambda (s) (prin1-to-string s t))
> +                                       (cdr error) ", "))
> +            :warning)
> +           (setq init-file-had-error t))))
> +
> +      ;; If we can tell that the init file altered debug-on-error,
> +      ;; arrange to preserve the value that it set up.
> +      (or (eq debug-on-error debug-on-error-initial)
> +          (setq debug-on-error-should-be-set t
> +                debug-on-error-from-init-file debug-on-error)))
> +
> +    (when debug-on-error-should-be-set
> +      (setq debug-on-error debug-on-error-from-init-file))
> +
> +    (unless (or (default-value 'enable-multibyte-characters)
> +                (eq orig-enable-multibyte (default-value
> +                                            'enable-multibyte-characters)))
> +
> +      ;; Init file changed to unibyte.  Reset existing multibyte
> +      ;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
> +      ;; Arguably this should only be done if they're free of
> +      ;; multibyte characters.
> +      (mapc (lambda (buffer)
> +              (with-current-buffer buffer
> +                (if enable-multibyte-characters
> +                    (set-buffer-multibyte nil))))
> +            (buffer-list))
> +
> +      ;; Also re-set the language environment in case it was
> +      ;; originally done before unibyte was set and is sensitive to
> +      ;; unibyte (display table, terminal coding system &c).
> +      (set-language-environment current-language-environment))))
> +
>  (defun command-line ()
>    "A subroutine of `normal-top-level'.
>  Amongst another things, it parses the command-line arguments."
> @@ -1021,6 +1153,69 @@ command-line
>      (and command-line-args
>           (setcdr command-line-args args)))
>  
> +  ;; Warn for invalid user name.
> +  (when init-file-user
> +    (if (string-match "[~/:\n]" init-file-user)
> +        (display-warning 'initialization
> +                         (format "Invalid user name %s"
> +                                 init-file-user)
> +                         :error)
> +      (if (file-directory-p (expand-file-name
> +                             ;; We don't support ~USER on MS-Windows
> +                             ;; and MS-DOS except for the current
> +                             ;; user, and always load .emacs from
> +                             ;; the current user's home directory
> +                             ;; (see below).  So always check "~",
> +                             ;; even if invoked with "-u USER", or
> +                             ;; if $USER or $LOGNAME are set to
> +                             ;; something different.
> +                             (if (memq system-type '(windows-nt ms-dos))
> +                                 "~"
> +                               (concat "~" init-file-user))))
> +          nil
> +        (display-warning 'initialization
> +                         (format "User %s has no home directory"
> +                                 (if (equal init-file-user "")
> +                                     (user-real-login-name)
> +                                   init-file-user))
> +                         :error))))
> +
> +  ;; Load the early init file, if found.
> +  (load-user-init-file
> +   (lambda ()
> +     (expand-file-name
> +      "early-init"
> +      (file-name-as-directory
> +       (concat "~" init-file-user "/.emacs.d")))))
> +  (setq early-init-file user-init-file)
> +
> +  ;; If any package directory exists, initialize the package system.
> +  (and user-init-file
> +       package-enable-at-startup
> +       (catch 'package-dir-found
> +	 (let (dirs)
> +	   (if (boundp 'package-directory-list)
> +	       (setq dirs package-directory-list)
> +	     (dolist (f load-path)
> +	       (and (stringp f)
> +		    (equal (file-name-nondirectory f) "site-lisp")
> +		    (push (expand-file-name "elpa" f) dirs))))
> +	   (push (if (boundp 'package-user-dir)
> +		     package-user-dir
> +		   (locate-user-emacs-file "elpa"))
> +		 dirs)
> +	   (dolist (dir dirs)
> +	     (when (file-directory-p dir)
> +	       (dolist (subdir (directory-files dir))
> +		 (when (let ((subdir (expand-file-name subdir dir)))
> +                         (and (file-directory-p subdir)
> +                              (file-exists-p
> +                               (expand-file-name
> +                                (package--description-file subdir)
> +                                subdir))))
> +		   (throw 'package-dir-found t)))))))
> +       (package-initialize))
> +
>    ;; Make sure window system's init file was loaded in loadup.el if
>    ;; using a window system.
>    ;; Initialize the window-system only after processing the command-line
> @@ -1127,170 +1322,47 @@ command-line
>      ;; the startup screen.
>      (setq inhibit-startup-screen nil)
>  
> -    ;; Warn for invalid user name.
> -    (when init-file-user
> -      (if (string-match "[~/:\n]" init-file-user)
> -	  (display-warning 'initialization
> -			   (format "Invalid user name %s"
> -				   init-file-user)
> -			   :error)
> -	(if (file-directory-p (expand-file-name
> -			       ;; We don't support ~USER on MS-Windows
> -			       ;; and MS-DOS except for the current
> -			       ;; user, and always load .emacs from
> -			       ;; the current user's home directory
> -			       ;; (see below).  So always check "~",
> -			       ;; even if invoked with "-u USER", or
> -			       ;; if $USER or $LOGNAME are set to
> -			       ;; something different.
> -			       (if (memq system-type '(windows-nt ms-dos))
> -				   "~"
> -				 (concat "~" init-file-user))))
> -	    nil
> -	  (display-warning 'initialization
> -			   (format "User %s has no home directory"
> -				   (if (equal init-file-user "")
> -				       (user-real-login-name)
> -				     init-file-user))
> -			   :error))))
> -
>      ;; Load that user's init file, or the default one, or none.
> -    (let (debug-on-error-from-init-file
> -	  debug-on-error-should-be-set
> -	  (debug-on-error-initial
> -	   (if (eq init-file-debug t) 'startup init-file-debug))
> -	  (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
> -      (let ((debug-on-error debug-on-error-initial)
> -	    ;; This function actually reads the init files.
> -	    (inner
> -	     (function
> -	      (lambda ()
> -		(if init-file-user
> -		    (let ((user-init-file-1
> -			   (cond
> -			     ((eq system-type 'ms-dos)
> -			      (concat "~" init-file-user "/_emacs"))
> -			     ((not (eq system-type 'windows-nt))
> -			      (concat "~" init-file-user "/.emacs"))
> -			     ;; Else deal with the Windows situation
> -			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
> -			      ;; Prefer .emacs on Windows.
> -			      "~/.emacs")
> -			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
> -			      ;; Also support _emacs for compatibility, but warn about it.
> -			      (push `(initialization
> -				      ,(format-message
> -					"`_emacs' init file is deprecated, please use `.emacs'"))
> -				    delayed-warnings-list)
> -			      "~/_emacs")
> -			     (t ;; But default to .emacs if _emacs does not exist.
> -			      "~/.emacs"))))
> -		      ;; This tells `load' to store the file name found
> -		      ;; into user-init-file.
> -		      (setq user-init-file t)
> -		      (load user-init-file-1 t t)
> -
> -		      (when (eq user-init-file t)
> -			;; If we did not find ~/.emacs, try
> -			;; ~/.emacs.d/init.el.
> -			(let ((otherfile
> -			       (expand-file-name
> -				"init"
> -				(file-name-as-directory
> -				 (concat "~" init-file-user "/.emacs.d")))))
> -			  (load otherfile t t)
> -
> -			  ;; If we did not find the user's init file,
> -			  ;; set user-init-file conclusively.
> -			  ;; Don't let it be set from default.el.
> -			  (when (eq user-init-file t)
> -			    (setq user-init-file user-init-file-1))))
> -
> -		      ;; If we loaded a compiled file, set
> -		      ;; `user-init-file' to the source version if that
> -		      ;; exists.
> -		      (when (and user-init-file
> -				 (equal (file-name-extension user-init-file)
> -					"elc"))
> -			(let* ((source (file-name-sans-extension user-init-file))
> -			       (alt (concat source ".el")))
> -			  (setq source (cond ((file-exists-p alt) alt)
> -					     ((file-exists-p source) source)
> -					     (t nil)))
> -			  (when source
> -			    (when (file-newer-than-file-p source user-init-file)
> -			      (message "Warning: %s is newer than %s"
> -				       source user-init-file)
> -			      (sit-for 1))
> -			    (setq user-init-file source))))
> -
> -		      (unless inhibit-default-init
> -                        (let ((inhibit-startup-screen nil))
> -                          ;; Users are supposed to be told their rights.
> -                          ;; (Plus how to get help and how to undo.)
> -                          ;; Don't you dare turn this off for anyone
> -                          ;; except yourself.
> -                          (load "default" t t)))))))))
> -	(if init-file-debug
> -	    ;; Do this without a condition-case if the user wants to debug.
> -	    (funcall inner)
> -	  (condition-case error
> -	      (progn
> -		(funcall inner)
> -		(setq init-file-had-error nil))
> -	    (error
> -	     (display-warning
> -	      'initialization
> -	      (format-message "\
> -An error occurred while loading `%s':\n\n%s%s%s\n\n\
> -To ensure normal operation, you should investigate and remove the
> -cause of the error in your initialization file.  Start Emacs with
> -the `--debug-init' option to view a complete error backtrace."
> -		      user-init-file
> -		      (get (car error) 'error-message)
> -		      (if (cdr error) ": " "")
> -		      (mapconcat (lambda (s) (prin1-to-string s t))
> -				 (cdr error) ", "))
> -	      :warning)
> -	     (setq init-file-had-error t))))
> -
> -      (if (and deactivate-mark transient-mark-mode)
> -	    (with-current-buffer (window-buffer)
> -	      (deactivate-mark)))
> -
> -	;; If the user has a file of abbrevs, read it (unless -batch).
> -	(when (and (not noninteractive)
> -		   (file-exists-p abbrev-file-name)
> -		   (file-readable-p abbrev-file-name))
> -	    (quietly-read-abbrev-file abbrev-file-name))
> -
> -	;; If the abbrevs came entirely from the init file or the
> -	;; abbrevs file, they do not need saving.
> -	(setq abbrevs-changed nil)
> -
> -	;; If we can tell that the init file altered debug-on-error,
> -	;; arrange to preserve the value that it set up.
> -	(or (eq debug-on-error debug-on-error-initial)
> -	    (setq debug-on-error-should-be-set t
> -		  debug-on-error-from-init-file debug-on-error)))
> -      (if debug-on-error-should-be-set
> -	  (setq debug-on-error debug-on-error-from-init-file))
> -      (unless (or (default-value 'enable-multibyte-characters)
> -		  (eq orig-enable-multibyte (default-value
> -					      'enable-multibyte-characters)))
> -	;; Init file changed to unibyte.  Reset existing multibyte
> -	;; buffers (probably *scratch*, *Messages*, *Minibuf-0*).
> -	;; Arguably this should only be done if they're free of
> -	;; multibyte characters.
> -	(mapc (lambda (buffer)
> -		(with-current-buffer buffer
> -		  (if enable-multibyte-characters
> -		      (set-buffer-multibyte nil))))
> -	      (buffer-list))
> -	;; Also re-set the language environment in case it was
> -	;; originally done before unibyte was set and is sensitive to
> -	;; unibyte (display table, terminal coding system &c).
> -	(set-language-environment current-language-environment)))
> +    (load-user-init-file
> +     (lambda ()
> +       (cond
> +        ((eq system-type 'ms-dos)
> +         (concat "~" init-file-user "/_emacs"))
> +        ((not (eq system-type 'windows-nt))
> +         (concat "~" init-file-user "/.emacs"))
> +        ;; Else deal with the Windows situation.
> +        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
> +         ;; Prefer .emacs on Windows.
> +         "~/.emacs")
> +        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
> +         ;; Also support _emacs for compatibility, but warn about it.
> +         (push `(initialization
> +                 ,(format-message
> +                   "`_emacs' init file is deprecated, please use `.emacs'"))
> +               delayed-warnings-list)
> +         "~/_emacs")
> +        (t ;; But default to .emacs if _emacs does not exist.
> +         "~/.emacs")))
> +     (lambda ()
> +       (expand-file-name
> +        "init"
> +        (file-name-as-directory
> +         (concat "~" init-file-user "/.emacs.d"))))
> +     (not inhibit-default-init))
> +
> +    (when (and deactivate-mark transient-mark-mode)
> +      (with-current-buffer (window-buffer)
> +        (deactivate-mark)))
> +
> +    ;; If the user has a file of abbrevs, read it (unless -batch).
> +    (when (and (not noninteractive)
> +               (file-exists-p abbrev-file-name)
> +               (file-readable-p abbrev-file-name))
> +      (quietly-read-abbrev-file abbrev-file-name))
> +
> +    ;; If the abbrevs came entirely from the init file or the
> +    ;; abbrevs file, they do not need saving.
> +    (setq abbrevs-changed nil)
>  
>      ;; Do this here in case the init file sets mail-host-address.
>      (and mail-host-address
> @@ -1312,33 +1384,6 @@ command-line
>  		 (eq face-ignored-fonts old-face-ignored-fonts))
>        (clear-face-cache)))
>  
> -  ;; If any package directory exists, initialize the package system.
> -  (and user-init-file
> -       package-enable-at-startup
> -       (catch 'package-dir-found
> -	 (let (dirs)
> -	   (if (boundp 'package-directory-list)
> -	       (setq dirs package-directory-list)
> -	     (dolist (f load-path)
> -	       (and (stringp f)
> -		    (equal (file-name-nondirectory f) "site-lisp")
> -		    (push (expand-file-name "elpa" f) dirs))))
> -	   (push (if (boundp 'package-user-dir)
> -		     package-user-dir
> -		   (locate-user-emacs-file "elpa"))
> -		 dirs)
> -	   (dolist (dir dirs)
> -	     (when (file-directory-p dir)
> -	       (dolist (subdir (directory-files dir))
> -		 (when (let ((subdir (expand-file-name subdir dir)))
> -                         (and (file-directory-p subdir)
> -                              (file-exists-p
> -                               (expand-file-name
> -                                (package--description-file subdir)
> -                                subdir))))
> -		   (throw 'package-dir-found t)))))))
> -       (package-initialize))
> -
>    (setq after-init-time (current-time))
>    ;; Display any accumulated warnings after all functions in
>    ;; `after-init-hook' like `desktop-read' have finalized possible
> diff --git a/src/lread.c b/src/lread.c
> index 52897b4fcc..5ff438041b 100644
> --- a/src/lread.c
> +++ b/src/lread.c
> @@ -4892,7 +4892,7 @@ directory.  These file names are converted to absolute at startup.  */);
>  If the file loaded had extension `.elc', and the corresponding source file
>  exists, this variable contains the name of source file, suitable for use
>  by functions like `custom-save-all' which edit the init file.
> -While Emacs loads and evaluates the init file, value is the real name
> +While Emacs loads and evaluates any init file, value is the real name
>  of the file, regardless of whether or not it has the `.elc' extension.  */);
>    Vuser_init_file = Qnil;
>  
> -- 
> 2.14.3
> 

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>>    By default, Emacs also automatically loads all installed packages in
>> -subsequent Emacs sessions.  This happens at startup, after processing
>> -the init file (@pxref{Init File}).  As an exception, Emacs does not
>> -load packages at startup if invoked with the @samp{-q} or
>> -@samp{--no-init-file} options (@pxref{Initial Options}).
>> +subsequent Emacs sessions.  This happens at startup, before processing
>
> Was this supposed to be "previous" rather than "subsequent" (this
> isn't from your patch, though).

Depends how you parse the sentence:

    Emacs also automatically loads all (installed packages) in
    subsequent Emacs sessions
vs
    Emacs also automatically loads all (installed packages in
    previous Emacs sessions)

but in the second case you'd rather write it

    Emacs also automatically loads all packages installed in
    previous Emacs sessions

so I think "subsequent" was not a typo.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Clément Pit-Claudel]

On 2018-01-25 11:03, Stefan Monnier wrote:
> but in the second case you'd rather write it
> 
>     Emacs also automatically loads all packages installed in
>     previous Emacs sessions

Indeed, good point.  Then let me amend the suggestion: just remove the "in subsequent Emacs sessions" part :)

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> but in the second case you'd rather write it
>> 
>>     Emacs also automatically loads all packages installed in
>>     previous Emacs sessions
>
> Indeed, good point.  Then let me amend the suggestion: just remove the "in
> subsequent Emacs sessions" part :)

Fine by me,


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 25 Jan 2018 11:22:25 -0500
> 
> On 2018-01-25 11:03, Stefan Monnier wrote:
> > but in the second case you'd rather write it
> > 
> >     Emacs also automatically loads all packages installed in
> >     previous Emacs sessions
> 
> Indeed, good point.  Then let me amend the suggestion: just remove the "in subsequent Emacs sessions" part :)

There's no need to remove text that is unclear; instead, it should be
rephrased to make it more clear.

Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Discussion of a patch cited this:

  > >     Emacs also automatically loads all packages installed in
  > >     previous Emacs sessions

If that means what I think it means, I think it is bad design for this
feature.

Suppose the user loads a package to try it out.  Will that cause it to
be loaded again in every session?  It appears that way.

I think that asking for a package to be loaded in every session
should be distinguished from loading it for the current session.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> Suppose the user loads a package to try it out.  Will that cause it to
> be loaded again in every session?  It appears that way.
> I think that asking for a package to be loaded in every session
> should be distinguished from loading it for the current session.

There are 3 different steps:
A- installing a package: this places its files under ~/.emacs.d/elpa
B- activating a package: this loads its <pkg>-autoloads.el file.
C- actually loading the package's files.

When a user does A, by default, it will do B in the current session and
causes subsequent Emacs invocations to automatically do B as well
(i.e. B is done by default on all installed packages).

You can configure `package-load-list` if you want to prevent
a particular package from being activated even though you still want it
to be installed.

But regardless, C should only happen on-demand when something actively
requests it.

Of course, a package is free to set up its autoloads such that
B causes C.  I'd consider it a misfeature of that package, tho.


        Stefan

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There are 3 different steps:
  > A- installing a package: this places its files under ~/.emacs.d/elpa
  > B- activating a package: this loads its <pkg>-autoloads.el file.
  > C- actually loading the package's files.

  > When a user does A, by default, it will do B in the current session and
  > causes subsequent Emacs invocations to automatically do B as well
  > (i.e. B is done by default on all installed packages).

That doesn't make sense to me.

I think A should not, automatically or by default, do B.

I think there should be a way to activate the package right now,
and a way to specify to activate it in future sessions.

Perhaps each of those should automatically do A, since B requires A.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>   > When a user does A, by default, it will do B in the current session and
>   > causes subsequent Emacs invocations to automatically do B as well
>   > (i.e. B is done by default on all installed packages).
> That doesn't make sense to me.

Why not?  All the packages that come bundled with Emacs are always
unconditionally activated and we don't offer any way to prevent this
activation (indeed, it's activated during the dump so we basically
can't prevent it).

Package.el follows the same idea (except that it does offer some way to
prevent activation of some packages, by customizing package-load-list).


        Stefan

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Why not?  All the packages that come bundled with Emacs are always
  > unconditionally activated and we don't offer any way to prevent this
  > activation (indeed, it's activated during the dump so we basically
  > can't prevent it).

They are always "activated", but the downloadable packages are not.
Since the status of activation can change, let's handle this in the
way users expect, for things that can be enabled and disabled.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> They are always "activated", but the downloadable packages are not.
> Since the status of activation can change, let's handle this in the
> way users expect, for things that can be enabled and disabled.

I think things already work the way users expect. Let's look at
Python. When you install a package using 'pip', the package files are
placed on Python's search path. Thereafter, 'import <package>' works
without any further "activation" step. The package is automatically
made available in future sessions, although it is not loaded until
requested.

The analogous behavior is to have installed packages' autoloads be
made available automatically in future sessions of Emacs, although the
packages are not loaded until requested.

-- Radon

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think things already work the way users expect. Let's look at
  > Python. When you install a package using 'pip', the package files are
  > placed on Python's search path. Thereafter, 'import <package>' works
  > without any further "activation" step. The package is automatically
  > made available in future sessions, although it is not loaded until
  > requested.

In Python, once a package is on the search path, the user needs to
give a specific command to make it callable from a given module.

I think Emacs should work the same way.

If I understood correctly, it is currently NOT the same, because in
Emacs the package doesn't require any command to make the package
callable.  Simply getting it from ELPA makes it callable _all the time_.

The change I have asked for would make Emacs do like Python.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

On 2018-02-01 21:14, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > I think things already work the way users expect. Let's look at
>   > Python. When you install a package using 'pip', the package files are
>   > placed on Python's search path. Thereafter, 'import <package>' works
>   > without any further "activation" step. The package is automatically
>   > made available in future sessions, although it is not loaded until
>   > requested.
> 
> In Python, once a package is on the search path, the user needs to
> give a specific command to make it callable from a given module.
> 
> I think Emacs should work the same way.

There may be a misunderstanding.  Emacs already works in (mostly) the same way: Emacs' `require' is very close to Python's `import'.

> If I understood correctly, it is currently NOT the same, because in
> Emacs the package doesn't require any command to make the package
> callable.  Simply getting it from ELPA makes it callable _all the time_.

No, this isn't quite right.  For example, if package foo has a function foo-x, you will need (in most cases) to (require 'foo) before you can call foo-x.

There are some differences, but they are small:

* `require' works globally in Emacs, so once package foo is `require'd in one file, all subsequently executed code can call foo-x, without calling (require 'foo) again.  This is why packages that are in loaded in temacs and dumped don't need to be `require'd.

* Emacs has autoloads, small pieces of code from packages that are run inconditionally.  Autoloads are what makes it possible to call certain functions without requiring the corresponding package first.  In spirit, this is very similar to Python's `pth' files.  Autoload files are a performance issue in some cases, because there are many of them (although each is fairly small). Stefan's patch makes them much faster.

Hope this helps,
Clément.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There may be a misunderstanding.  Emacs already works in (mostly) the same way: Emacs' `require' is very close to Python's `import'.

That's the behavior I'd expect a feature like this to have.

  > * Emacs has autoloads, small pieces of code from packages that are run inconditionally.

I do know about autoloads; I implemented them.
The questions are (1) do a lot of these packages have autoloads, or just
a few, and (2) when do the autoloads get installed into Emacs.

If adding a package to the list for loading has the effect of installing
its autploads in all future sessions, that results in behavior very
different from the 'require' behavior, and behavior that doesn't match
what I'd expect such a feature to have.

Does Python have autoloads?  I would expect not.

  > * `require' works globally in Emacs, so once package foo is
  > * `require'd in one file, all subsequently executed code can call
  > * foo-x, without calling (require 'foo) again.

That is true.  With dynamic scoping that was basically inevitable.

However, when lexical scoping becomes the default, it might be
possible to arrange that a Lisp library is in scope only in files that
require it.  That would clean up a lot of things.  For instance,
it would not matter whether you compile A before B or after B.

Since the effect of calling an autoload function is to call 'require',
it could be that lexical handling of 'require' will automatically
clean up the way these autoloads are handled.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

>   > There may be a misunderstanding.  Emacs already works in (mostly) the same way: Emacs' `require' is very close to Python's `import'.
> 
> That's the behavior I'd expect a feature like this to have.
> 
>   > * Emacs has autoloads, small pieces of code from packages that are run inconditionally.
> 
> I do know about autoloads; I implemented them.

I know that; but I'm writing to the entire list, and there may be other readers who are less familiar with autoloads :)

> The questions are (1) do a lot of these packages have autoloads, or just
> a few, and (2) when do the autoloads get installed into Emacs.

Most packages, AFAICT, make their main interactive entry points autoloads.  I think that is the right behavior.

> If adding a package to the list for loading has the effect of installing
> its autploads in all future sessions, that results in behavior very
> different from the 'require' behavior, and behavior that doesn't match
> what I'd expect such a feature to have.
> 
> Does Python have autoloads?  I would expect not.

Yes, in two senses:

* Installing a package with 'pip' commonly installs small binaries in your path, so you can call the program from the command line.
* Python has .pth files that work essentially like Emacs autoloads.  These aren't used very commonly.

> Since the effect of calling an autoload function is to call 'require',
> it could be that lexical handling of 'require' will automatically
> clean up the way these autoloads are handled.

The thing is, I don't a the problem with the way autoloads are currently handled.  seeWe just need to find a way to make processing autoloads faster, and in fact Stefan has a solution for that, IIUC.

The fact that installing a package installs its autoloads is desirable; just like the fact that running 'apt-get install emacs' puts the emacs binary on your path.

I think lexical vs dynamic 'require' is a different issue.

Clément.

Re: Loading a package applies automatically to future sessions?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 4222 bytes --]

While I think I can see what RMS is concerned about, I am not necessarily
convinced the concern is significant enough to warrant implementation of
additional user action that would be required in order to activate packages
in future sessions. One of the great benefits of the ELPA system has been
how much easier it has made it to add useful extensions and keep those
extensions up-to-date without requiring the user to write any ELISP. While
I personally don't find writing ELISP a challenge, many do - or at least
shy away from it initially.

Bottom line is that when you install most ELPA packages, the package will
often install basic autoloads. This means that should you do something or
execute one of the autoload commands, the package will load and run in
future sessions. So, in effect, the answer to the original question is yes,
packages are available and will be automatically loaded in future sessions
if the user executes one of those autloaded functions. However, I'm not
aware of any packages which setup hooks or add themselves to hooks so that
the package is loaded without specific user action in future sessions (I
have not tried all packages and it would certainly be possible for a
package to do this, though it might be tricky to implement unless you edit
the user's init file, which I don't think is normal for an elpa package).

I feel the basic mechanism to prevent packages from being available in
future sessions is to simply remove them. I actually think this is a good
practice as it prevents people from building up large numbers of
unused/unnecessary packages  that only slow down updates. It also reduces
the likelihood of unused packages conflicting with used packages and works
towards a cleaner environment.  Given that re-installing is also trivial, I
don't think this is too much of a burden should there be a package you only
need very rarely.  Anyone who wants something more complex is of course
free to implement such functionality themselves.

Tim





On 5 February 2018 at 02:21, Clément Pit-Claudel <cpitclaudel@gmail.com>
wrote:

> >   > There may be a misunderstanding.  Emacs already works in (mostly)
> the same way: Emacs' `require' is very close to Python's `import'.
> >
> > That's the behavior I'd expect a feature like this to have.
> >
> >   > * Emacs has autoloads, small pieces of code from packages that are
> run inconditionally.
> >
> > I do know about autoloads; I implemented them.
>
> I know that; but I'm writing to the entire list, and there may be other
> readers who are less familiar with autoloads :)
>
> > The questions are (1) do a lot of these packages have autoloads, or just
> > a few, and (2) when do the autoloads get installed into Emacs.
>
> Most packages, AFAICT, make their main interactive entry points
> autoloads.  I think that is the right behavior.
>
> > If adding a package to the list for loading has the effect of installing
> > its autploads in all future sessions, that results in behavior very
> > different from the 'require' behavior, and behavior that doesn't match
> > what I'd expect such a feature to have.
> >
> > Does Python have autoloads?  I would expect not.
>
> Yes, in two senses:
>
> * Installing a package with 'pip' commonly installs small binaries in your
> path, so you can call the program from the command line.
> * Python has .pth files that work essentially like Emacs autoloads.  These
> aren't used very commonly.
>
> > Since the effect of calling an autoload function is to call 'require',
> > it could be that lexical handling of 'require' will automatically
> > clean up the way these autoloads are handled.
>
> The thing is, I don't a the problem with the way autoloads are currently
> handled.  seeWe just need to find a way to make processing autoloads
> faster, and in fact Stefan has a solution for that, IIUC.
>
> The fact that installing a package installs its autoloads is desirable;
> just like the fact that running 'apt-get install emacs' puts the emacs
> binary on your path.
>
> I think lexical vs dynamic 'require' is a different issue.
>
> Clément.
>
>


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 5186 bytes --]

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> Given that re-installing is also trivial

This is not always true, esp. given a large number of packages.

Re: Loading a package applies automatically to future sessions?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1342 bytes --]

I was referring to re-installation of individual packages i.e. you install
a package, find you don't really need it at this time and so remove it and
then find later you do need it as opposed to removing ALL packages and then
re-installing.

However, having said that, I have done this many times in the past and
never run into any problems. It can take a bit of time if you have a large
number of packages, but I've never found any need to take additional action
with only 1 exception and that is with respect to org mode.

The only package which has ever given me any problems has been org mode and
that was partly my fault (I was loading org functionality and then trying
to install a later version from the org repo).

Of course, if your using unofficial elpa repositories, all bets are off as
there is no curation of the content in these repositories. I have certainly
found crappy packages in some of these repos. However, that is not
something Emacs has control of and if as a user you decide to install
packages from such repos, then you need to be willing to deal with the
issues which arise.

Tim

On 5 February 2018 at 21:07, George Plymale II <georgedp@orbitalimpact.com>
wrote:

> > Given that re-installing is also trivial
>
> This is not always true, esp. given a large number of packages.
>



-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1999 bytes --]

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does Python have autoloads?  I would expect not.

  > Yes, in two senses:

  > * Installing a package with 'pip' commonly installs small binaries in your path, so you can call the program from the command line.
  > * Python has .pth files that work essentially like Emacs autoloads.  These aren't used very commonly.

How do these autoload-like constructs in Python interact with
the usual need to 'import' a package to make its entry points accessible?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

On 2018-02-04 20:08, Richard Stallman wrote:
>   > > Does Python have autoloads?  I would expect not.
> 
>   > Yes, in two senses:
> 
>   > * Installing a package with 'pip' commonly installs small binaries in your path, so you can call the program from the command line.
>   > * Python has .pth files that work essentially like Emacs autoloads.  These aren't used very commonly.
> 
> How do these autoload-like constructs in Python interact with
> the usual need to 'import' a package to make its entry points accessible?

They're the first thing the interpreter runs when it starts.  You can write 'import foo' on a separate line, and then python imports 'foo' before running your actual program.  The code in 'foo' can then introduce new global names by modifying the builtins dictionary (this isn't a very common practice)

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I retract my concerns about this feature.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> Suppose the user loads a package to try it out.  Will that cause it to
> be loaded again in every session?  It appears that way.

I think the confusion here is because the Emacs manual currently uses
the term "load" in two incompatible ways. Loading a package means
evaluating its autoloads, while loading a Lisp file means evaluating
the file's Lisp code. The problem is complicated by the fact that the
word "package" is sometimes used to refer to a package in the package
manager sense, and sometimes used to refer to a Lisp file.

Perhaps some better terminology would be in order? I would suggest
that we never refer to "loading" a package, and instead say:

* "activating a package" means evaluating its autoloads
* "loading a file/feature" means evaluating some particular file of
  Lisp code, which is possibly part of a package

I don't have strong opinions, except that the current terminology is
bad.

-- Radon

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Fri, 26 Jan 2018 11:24:32 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> I think the confusion here is because the Emacs manual currently uses
> the term "load" in two incompatible ways. Loading a package means
> evaluating its autoloads, while loading a Lisp file means evaluating
> the file's Lisp code.

Please point out the places in the manual where we use "load" in the
former sense.  I think those places need to be fixed, because that's
not what "load" means in the Emacs context.

Thanks.

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> Please point out the places in the manual where we use "load" in the
> former sense.  I think those places need to be fixed, because that's
> not what "load" means in the Emacs context.

The crux of the problem is here in emacs/package.texi:

"Once a package is downloaded and installed, it is @dfn{loaded} into
the current Emacs session. Loading a package is not quite the same as
loading a Lisp library (@pxref{Lisp Libraries}); loading a package
adds its directory to @code{load-path} and loads its autoloads."

Frankly the name of `package-load-list' is misleading if we're going
to change this terminology. It implies you are setting which packages
are going to be loaded.

We also have usage of this terminology in emacs/package.texi:

"After a package is installed, it is automatically loaded by Emacs in
all subsequent sessions."

"As an exception, Emacs does not load packages at startup if invoked
with the @samp{-q} or @samp{--no-init-file} options (@xref{Initial
Options})."

"To disable automatic package loading, change the variable
@code{package-enable-at-startup} to @code{nil}. You must do this in
the early init file, as the variable is read before loading the
regular init file."

"For finer control over package loading, you can use the variable
@code{package-load-list}."

"A list element of the form @code{(@var{name} @var{version})} tells
Emacs to load version @var{version} of the package named @var{name}.
Here, @var{version} should be a version string (corresponding to a
specific version of the package), or @code{t} (which means to load any
installed version), or @code{nil} (which means no version; this
disables the package, preventing it from being loaded). A list element
can also be the symbol @code{all}, which means to load the latest
installed version of any package not named by the other list
elements."

"For example, if you set @code{package-load-list} to @code{'((muse
"3.20") all)}, then Emacs only loads version 3.20 of the @samp{muse}
package, plus any installed version of packages other than
@samp{muse}."

In lispref/package.texi:

"Whenever Emacs starts up, it automatically calls the function
@code{package-initialize} to load installed packages."

"Automatic package loading is disabled if the user option
@code{package-enable-at-startup} is set to @code{nil} in the early
init file."

"This function initializes Emacs' internal record of which packages
are installed, and loads them. The user option
@code{package-load-list} specifies which packages to load; by default,
all installed packages are loaded. If called during startup, this
function also sets @code{package-enable-at-startup} to @code{nil}, to
avoid accidentally loading the packages twice."

"The optional argument @var{no-activate}, if non-@code{nil}, causes
Emacs to update its record of installed packages without actually
loading them; it is for internal use only."

In lispref/os.texi:

"For example, you can customize the process of loading installed
packages, by setting variables such as @var{package-load-list} or
@var{package-enable-at-startup}. @xref{Package Installation,,,
emacs,The GNU Emacs Manual}."

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We can change the name of a variable painlessly by defining an alias
for it.  So let's decide which terms are best and clearest.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sun, 28 Jan 2018 10:20:49 -0800
> Cc: rms@gnu.org, emacs-devel <emacs-devel@gnu.org>
> 
> > Please point out the places in the manual where we use "load" in the
> > former sense.  I think those places need to be fixed, because that's
> > not what "load" means in the Emacs context.
> 
> The crux of the problem is here in emacs/package.texi:
> 
> "Once a package is downloaded and installed, it is @dfn{loaded} into
> the current Emacs session. Loading a package is not quite the same as
> loading a Lisp library (@pxref{Lisp Libraries}); loading a package
> adds its directory to @code{load-path} and loads its autoloads."
> 
> Frankly the name of `package-load-list' is misleading if we're going
> to change this terminology. It implies you are setting which packages
> are going to be loaded.
> 
> We also have usage of this terminology in emacs/package.texi:
> 
> "After a package is installed, it is automatically loaded by Emacs in
> all subsequent sessions."
> 
> "As an exception, Emacs does not load packages at startup if invoked
> with the @samp{-q} or @samp{--no-init-file} options (@xref{Initial
> Options})."
> 
> "To disable automatic package loading, change the variable
> @code{package-enable-at-startup} to @code{nil}. You must do this in
> the early init file, as the variable is read before loading the
> regular init file."
> 
> "For finer control over package loading, you can use the variable
> @code{package-load-list}."
> 
> "A list element of the form @code{(@var{name} @var{version})} tells
> Emacs to load version @var{version} of the package named @var{name}.
> Here, @var{version} should be a version string (corresponding to a
> specific version of the package), or @code{t} (which means to load any
> installed version), or @code{nil} (which means no version; this
> disables the package, preventing it from being loaded). A list element
> can also be the symbol @code{all}, which means to load the latest
> installed version of any package not named by the other list
> elements."
> 
> "For example, if you set @code{package-load-list} to @code{'((muse
> "3.20") all)}, then Emacs only loads version 3.20 of the @samp{muse}
> package, plus any installed version of packages other than
> @samp{muse}."
> 
> In lispref/package.texi:
> 
> "Whenever Emacs starts up, it automatically calls the function
> @code{package-initialize} to load installed packages."
> 
> "Automatic package loading is disabled if the user option
> @code{package-enable-at-startup} is set to @code{nil} in the early
> init file."
> 
> "This function initializes Emacs' internal record of which packages
> are installed, and loads them. The user option
> @code{package-load-list} specifies which packages to load; by default,
> all installed packages are loaded. If called during startup, this
> function also sets @code{package-enable-at-startup} to @code{nil}, to
> avoid accidentally loading the packages twice."
> 
> "The optional argument @var{no-activate}, if non-@code{nil}, causes
> Emacs to update its record of installed packages without actually
> loading them; it is for internal use only."
> 
> In lispref/os.texi:
> 
> "For example, you can customize the process of loading installed
> packages, by setting variables such as @var{package-load-list} or
> @var{package-enable-at-startup}. @xref{Package Installation,,,
> emacs,The GNU Emacs Manual}."

Thanks.  It looks like all of those places are related to package.el,
so could you please take care of clarifying the issue as part of your
patch?  Most of those places are touched by your patch anyway.

I think the fix should be not to use "load" in this sense, but instead
tell we just make the package available by loading its autoloads.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]

Hi,

Sorry, I don't mean to hijack the conversation, but speaking of patches,
Stefan gave me an updated version of his patch recently. This one allows
byte-compilation of his ~/.emacs.d/package-fastpath.el file. I asked him
in private if he needs any further feedback several days ago, but I did
not receive a reply from him so I'll assume that it's okay to re-post
his patch here on the list.

I have attached two patches. One is the patch that he originally made
which applies to the Emacs trunk, and the other is for Emacs 25.3. Which
is my current working Emacs version. If anyone could provide further
feedback to Stefan to help him improve this patch, I'm sure he would
appreciate it and I would as well.

So far, I haven't noticed any significant performance improvements from
his patch. Although, with the byte-compilation in place (along with some
other optimizations to my startup code) my init time has gone down to
1.1 seconds. It was 1.4 seconds last time I reported about it on this
thread, I think.

Also, I have noticed (and I forgot to tell Stefan about this in our
private thread) that the bug which I noticed regarding info files from
ELPA is gone. I.e., this bug (which I reported on Jan 31):

"The info files for my packages from ELPA are all missing in the info
Directory node."

is no longer happening, so I guess Stefan has fixed it in this most
recent patch. Yay!

Unfortunately, I had more problems with byte-compilation when I had the
gh.el package installed. package-fastpath.el would refuse to
byte-compile. I have since uninstalled this package since I only had one
other package which was dependent on it and I haven't used it
recently. Stefan's patch probably needs some work in order to properly
deal with strange autoload files. However, the developer of gh.el is
currently trying to make its autoload file a bit lighter so this may not
be a concern much longer.

In any case, I think that Stefan's patch here is still the best solution
to the original problem posed by RMS at the inception of this thread. It
just needs some work. It is also improving startup time quite noticeably
for myself and I am sure that it will for others as well.

Thanks,
- George Plymale II


[-- Attachment #2: Stefan's 2nd patch for package.el (trunk version) --]
[-- Type: text/x-patch, Size: 4048 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..b918eb5dfe 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -676,6 +676,8 @@ package--activate-autoloads-and-load-path
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--fastpath-pkgs t)
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -718,7 +720,10 @@ package-activate-1
             (message "Unable to activate package `%s'.\nRequired package `%s-%s' is unavailable"
                      name (car req) (package-version-join (cadr req)))
             (throw 'exit nil))))
-      (package--load-files-for-activation pkg-desc reload)
+      (if (listp package--fastpath-pkgs)
+          ;; We're only collecting the set of packages to activate!
+          (push pkg-desc package--fastpath-pkgs)
+        (package--load-files-for-activation pkg-desc reload))
       ;; Add info node.
       (when (file-exists-p (expand-file-name "dir" pkg-dir))
         ;; FIXME: not the friendliest, but simple.
@@ -3468,6 +3473,67 @@ package-list-packages-no-fetch
   (interactive)
   (list-packages t))
 
+;;;; Fast-path for activation!
+
+(defcustom package-fastpath-file (locate-user-emacs-file "package-fastpath.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package-fastpath-refresh ()
+  "(Re)Generate the `package-fastpath-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (require 'info)
+  (let ((package--fastpath-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ())
+        (Info-directory-list '("")))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (setq package--fastpath-pkgs (nreverse package--fastpath-pkgs))
+    (with-temp-file package-fastpath-file
+      (emacs-lisp-mode)
+      (insert ";;; FastPath file to speed up package activation at startup  -*- lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated, DO NOT EDIT !!\n\n")
+      (dolist (pkg package--fastpath-pkgs)
+        (let* ((file
+                ;; Prefer uncompiled files (and don't accept .so files).
+                (let ((load-suffixes '(".el" ".elc")))
+                  (locate-library (package--autoloads-file-name pkg))))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          ;; FIXME: We also need to setup the Info-directory-list!
+          (while (search-forward "#$" nil 'move)
+            (unless (nth 8 (syntax-ppss))
+              (replace-match pfile t t)))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name package--fastpath-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (let ((info-dirs (butlast Info-directory-list)))
+        (when info-dirs
+          (pp `(progn (require 'info)
+                      (info-initialize)
+                      (setq Info-directory-list
+                            (append ',info-dirs Info-directory-list)))
+              (current-buffer))))
+      ;; Use `\s' instead of a space character, so this code chunk is not
+      ;; mistaken for an actual file-local section of package.el.
+      (insert "\f
+;; Local\sVariables:
+;; version-control: never
+;; no-byte-compile: nil
+;; no-update-autoloads: t
+;; End:
+"))))
+
 (provide 'package)
 
 ;;; package.el ends here

[-- Attachment #3: Stefan's 2nd patch for package.el (25.3 stable version) --]
[-- Type: text/x-patch, Size: 4097 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 32200227de..cbb326f26e 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -656,6 +656,8 @@ PKG-DESC is a `package-desc' object."
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--fastpath-pkgs t)
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -696,7 +698,10 @@ correspond to previously loaded files (those returned by
         (unless (package-activate (car req))
           (error "Unable to activate package `%s'.\nRequired package `%s-%s' is unavailable"
                  name (car req) (package-version-join (cadr req))))))
-    (package--load-files-for-activation pkg-desc reload)
+    (if (listp package--fastpath-pkgs)
+        ;; We're only collecting the set of packages to activate!
+        (push pkg-desc package--fastpath-pkgs)
+      (package--load-files-for-activation pkg-desc reload))
     ;; Add info node.
     (when (file-exists-p (expand-file-name "dir" pkg-dir))
       ;; FIXME: not the friendliest, but simple.
@@ -3437,6 +3442,67 @@ The list is displayed in a buffer named `*Packages*'."
   (interactive)
   (list-packages t))
 
+;;;; Fast-path for activation!
+
+(defcustom package-fastpath-file (locate-user-emacs-file "package-fastpath.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package-fastpath-refresh ()
+  "(Re)Generate the `package-fastpath-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (require 'info)
+  (let ((package--fastpath-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ())
+        (Info-directory-list '("")))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (setq package--fastpath-pkgs (nreverse package--fastpath-pkgs))
+    (with-temp-file package-fastpath-file
+      (emacs-lisp-mode)
+      (insert ";;; FastPath file to speed up package activation at startup  -*- lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated, DO NOT EDIT !!\n\n")
+      (dolist (pkg package--fastpath-pkgs)
+        (let* ((file
+                ;; Prefer uncompiled files (and don't accept .so files).
+                (let ((load-suffixes '(".el" ".elc")))
+                  (locate-library (package--autoloads-file-name pkg))))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          ;; FIXME: We also need to setup the Info-directory-list!
+          (while (search-forward "#$" nil 'move)
+            (unless (nth 8 (syntax-ppss))
+              (replace-match pfile t t)))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name package--fastpath-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (let ((info-dirs (butlast Info-directory-list)))
+        (when info-dirs
+          (pp `(progn (require 'info)
+                      (info-initialize)
+                      (setq Info-directory-list
+                            (append ',info-dirs Info-directory-list)))
+              (current-buffer))))
+      ;; Use `\s' instead of a space character, so this code chunk is not
+      ;; mistaken for an actual file-local section of package.el.
+      (insert "\f
+;; Local\sVariables:
+;; version-control: never
+;; no-byte-compile: nil
+;; no-update-autoloads: t
+;; End:
+"))))
+
 (provide 'package)
 
 ;;; package.el ends here

Stefan's patch to improve package loading (was Loading a package applies automatically to future sessions?)

[George Plymale II]

Hi again,

Does anyone mind if I start a new thread with a new subject line? I
would like to continue discussion of Stefan's patch and improving the
loading of packages in Emacs on this list, but since the thread entitled
"Loading a package applies automatically to future sessions?" diverged
into copyright debates, I thought perhaps we should make a new thread
that specifically discusses Stefan's patches mentioned previously. It
should make things more organized and more easily searchable. Is anyone
opposed to this?

- George Plymale II

Re: Stefan's patch to improve package loading (was Loading a package applies automatically to future sessions?)

[John Wiegley]

>>>>> "GP" == George Plymale <georgedp@orbitalimpact.com> writes:

GP> Does anyone mind if I start a new thread with a new subject line? I would
GP> like to continue discussion of Stefan's patch and improving the loading of
GP> packages in Emacs on this list, but since the thread entitled "Loading a
GP> package applies automatically to future sessions?" diverged into copyright
GP> debates, I thought perhaps we should make a new thread that specifically
GP> discusses Stefan's patches mentioned previously. It should make things
GP> more organized and more easily searchable. Is anyone opposed to this?

This is actually standard practice nowadays. Just create a new thread exactly
as you've done, and propose the new discussion topic.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Stefan's patch to improve package loading (was Loading a package applies automatically to future sessions?)

[George Plymale II]

"John Wiegley" <johnw@gnu.org> writes:

> This is actually standard practice nowadays. Just create a new thread exactly
> as you've done, and propose the new discussion topic.

Ok, great. Thanks for letting me know.

Re: Stefan's patch to improve package loading

[George Plymale II]

Hi Stefan,

I know we've sent each other messages in private about this patch and
you said that you'd post some updates here on the mailing list, but it's
been a few weeks so I thought I'd give you a thread to send it to.

I have to say that your patch has really cut down on my startup time to
a point where I'm satisfied with it again (although it's not always
under one second, but that is mostly due to some of my configuration
plus the workload of my machine). I'm excited to see what kinds of
updates you have for it, as I hope it makes it into the trunk at some point.

Thanks.

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

[-- Attachment #1.1: Type: text/plain, Size: 328 bytes --]

> Thanks.  It looks like all of those places are related to package.el,
> so could you please take care of clarifying the issue as part of your
> patch?  Most of those places are touched by your patch anyway.

I have done this; the patch and my comments are in the main thread.
For convenience, the patch is also attached here.

[-- Attachment #1.2: Type: text/html, Size: 596 bytes --]

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 38832 bytes --]

From aaf0192d3da753df1f5e0dab1907affc719e191c Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable.
(load-user-init-file): New function.
(command-line): Load the early init file using `load-user-init-file'.
Move the check for an invalid username to just before that, and move
the initialization of the package system to just after.  Load the
regular init file using `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
definition, usage, and documentation.
(package--init-file-ensured): Remove definition and usage.

* doc/emacs/custom.texi: Document early init file.

* doc/lispref/os.texi: Document early init file.  Update startup
summary.

* doc/lispref/package.texi: Document changes to when
package-initialize is called, and advise against calling it in the
init file.  Change terminology for package 'loading'.

* doc/emacs/package.texi: Document changes to when package-initialize
is called.  Change terminology for package 'loading'.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
init file.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/custom.texi      |  17 ++
 doc/emacs/package.texi     |  78 +++++-----
 doc/lispref/os.texi        |  38 +++--
 doc/lispref/package.texi   |  30 ++--
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  19 +++
 lisp/emacs-lisp/package.el |  71 +--------
 lisp/startup.el            | 379 ++++++++++++++++++++++++---------------------
 src/lread.c                |   2 +-
 9 files changed, 331 insertions(+), 307 deletions(-)

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index e27760b379..263dff7cb0 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2575,3 +2575,20 @@ Init Non-ASCII
 coding system, for your init file as well as the files you edit.  For
 example, don't mix the @samp{latin-1} and @samp{latin-9} coding
 systems.
+
+@node Early Init File
+@subsection The Early Init File
+@cindex early init file
+
+  Most customizations for Emacs can be put in the normal init file,
+@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes
+desirable to have customizations that take effect during Emacs startup
+earlier than the normal init file is processed.  Such customizations
+can be put in the early init file, @file{~/.emacs.d/early-init.el}.
+This file is loaded before the package system is initialized, so in it
+you can customize variables that affect the initialization process,
+such as @code{package-enable-at-startup} and @code{package-load-list}.
+@xref{Package Installation}.
+
+  For more information on the early init file, @pxref{Early Init
+File,,, elisp, The Emacs Lisp Reference Manual}.
diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index bc6afb7966..d5339e9124 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -241,57 +241,55 @@ Package Installation
 package is available from a higher-priority archive.  (This is
 controlled by the value of @code{package-menu-hide-low-priority}.)
 
-  Once a package is downloaded and installed, it is @dfn{loaded} into
-the current Emacs session.  Loading a package is not quite the same as
-loading a Lisp library (@pxref{Lisp Libraries}); loading a package
-adds its directory to @code{load-path} and loads its autoloads.  The
-effect of a package's autoloads varies from package to package.  Most
-packages just make some new commands available, while others have more
+  Once a package is downloaded and installed, it is made available to
+the current Emacs session.  Making a package available adds its
+directory to @code{load-path} and loads its autoloads.  The effect of
+a package's autoloads varies from package to package.  Most packages
+just make some new commands available, while others have more
 wide-ranging effects on the Emacs session.  For such information,
 consult the package's help buffer.
 
-  By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+  After a package is installed, it is automatically made available by
+Emacs in all subsequent sessions.  This happens at startup, before
+processing the init file but after processing the early init file
+(@pxref{Early Init File,,, elisp, The Emacs Lisp Reference Manual}).
+As an exception, Emacs does not make packages available at startup if
+invoked with the @samp{-q} or @samp{--no-init-file} options
+(@xref{Initial Options}).
 
 @vindex package-enable-at-startup
-  To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
+  To keep Emacs from automatically making packages available at
+startup, change the variable @code{package-enable-at-startup} to
+@code{nil}.  You must do this in the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}), as the variable
+is read before loading the regular init file.  Currently this variable
+cannot be set via Customize.
 
 @findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+  If you have set @code{package-enable-at-startup} to @code{nil}, you
+can still make packages available either during or after startup.  To
+make installed packages available during startup, call the function
+@code{package-initialize} in your init file.  To make installed
+packages available after startup, invoke the command @kbd{M-x
+package-initialize}.
 
 @vindex package-load-list
-  For finer control over package loading, you can use the variable
-@code{package-load-list}.  Its value should be a list.  A list element
-of the form @code{(@var{name} @var{version})} tells Emacs to load
-version @var{version} of the package named @var{name}.  Here,
-@var{version} should be a version string (corresponding to a specific
-version of the package), or @code{t} (which means to load any
-installed version), or @code{nil} (which means no version; this
-disables the package, preventing it from being loaded).  A list
-element can also be the symbol @code{all}, which means to load the
-latest installed version of any package not named by the other list
-elements.  The default value is just @code{'(all)}.
+  For finer control over which packages are made available at startup,
+you can use the variable @code{package-load-list}.  Its value should
+be a list.  A list element of the form @code{(@var{name}
+@var{version})} tells Emacs to make available version @var{version} of
+the package named @var{name}.  Here, @var{version} should be a version
+string (corresponding to a specific version of the package), or
+@code{t} (which means to make available any installed version), or
+@code{nil} (which means no version; this disables the package,
+preventing it from being made available).  A list element can also be
+the symbol @code{all}, which means to make available the latest
+installed version of any package not named by the other list elements.
+The default value is just @code{'(all)}.
 
   For example, if you set @code{package-load-list} to @code{'((muse
-"3.20") all)}, then Emacs only loads version 3.20 of the @samp{muse}
-package, plus any installed version of packages other than
+"3.20") all)}, then Emacs only makes available version 3.20 of the
+@samp{muse} package, plus any installed version of packages other than
 @samp{muse}.  Any other version of @samp{muse} that happens to be
 installed will be ignored.  The @samp{muse} package will be listed in
 the package menu with the @samp{held} status.
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 42be60449d..cac5bfa067 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -95,6 +95,21 @@ Startup Summary
 @item
 It does some basic parsing of the command-line arguments.
 
+@item
+It loads your early init file (@pxref{Early Init File}).  This is not
+done if the options @samp{-q}, @samp{-Q}, or @samp{--batch} were
+specified.  If the @samp{-u} option was specified, Emacs looks for the
+init file in that user's home directory instead.
+
+@item
+It calls the function @code{package-initialize} to activate any
+optional Emacs Lisp package that has been installed.  @xref{Packaging
+Basics}.  However, Emacs doesn't initialize packages when
+@code{package-enable-at-startup} is @code{nil} or when it's started
+with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
+initialize packages in the latter case, @code{package-initialize}
+should be called explicitly (e.g., via the @samp{--funcall} option).
+
 @vindex initial-window-system@r{, and startup}
 @vindex window-system-initialization-alist
 @item
@@ -154,15 +169,6 @@ Startup Summary
 (@pxref{Abbrev Files, abbrev-file-name}).  This is not done if the
 option @samp{--batch} was specified.
 
-@item
-It calls the function @code{package-initialize} to activate any
-optional Emacs Lisp package that has been installed.  @xref{Packaging
-Basics}.  However, Emacs doesn't initialize packages when
-@code{package-enable-at-startup} is @code{nil} or when it's started
-with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
-initialize packages in the latter case, @code{package-initialize}
-should be called explicitly (e.g., via the @samp{--funcall} option).
-
 @vindex after-init-time
 @item
 It sets the variable @code{after-init-time} to the value of
@@ -361,6 +367,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +391,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+@dfn{early init file}, if it exists.  This is a file named
+@file{early-init.el} in your @file{~/.emacs.d} directory.  The
+difference between the early init file and the regular init file is
+that the early init file is loaded much earlier during the startup
+process, so you can use it to customize some things that are
+initialized before loading the regular init file.  For example, you
+can customize the process of initializing the package system, by
+setting variables such as @var{package-load-list} or
+@var{package-enable-at-startup}. @xref{Package Installation,,,
+emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 21dfe1c271..877aaf89a3 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -105,24 +105,32 @@ Packaging Basics
 evaluates the autoload definitions in @file{@var{name}-autoloads.el}.
 
   Whenever Emacs starts up, it automatically calls the function
-@code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+@code{package-initialize} to make installed packages available to the
+current session.  This is done after loading the early init file, but
+before loading the regular init file (@pxref{Startup Summary}).
+Packages are not automatically made available if the user option
+@code{package-enable-at-startup} is set to @code{nil} in the early
+init file.
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
-installed, and loads them.  The user option @code{package-load-list}
-specifies which packages to load; by default, all installed packages
-are loaded.  If called during startup, this function also sets
+installed, and makes the packages available to the current session.
+The user option @code{package-load-list} specifies which packages to
+make available; by default, all installed packages are made available.
+If called during startup, this function also sets
 @code{package-enable-at-startup} to @code{nil}, to avoid accidentally
-loading the packages twice.  @xref{Package Installation,,, emacs, The
-GNU Emacs Manual}.
+evaluating package autoloads more than once.  @xref{Package
+Installation,,, emacs, The GNU Emacs Manual}.
 
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
-loading them; it is for internal use only.
+making them available; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after it in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index aa3b029ab7..68aa01ca18 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 71569c95ad..b6b884b816 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -49,6 +49,25 @@ to reduce differences between developer and production builds.
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called 'early-init.el', in 'user-emacs-directory'.  It is
+loaded very early in the startup process: before graphical elements
+such as the tool bar are initialized, and before the package manager
+is initialized.  The primary purpose is to allow customizing how the
+package system is initialized given that initialization now happens
+before loading the regular init file (see below).
+
++++
+** Emacs now calls 'package-initialize' before loading the init file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to 'package-initialize' into the init
+file, which was previously done when Emacs was started.  As a result
+of this change, it is no longer necessary to call 'package-initialize'
+in your init file.  However, if your init file changes the values of
+'package-load-list' or 'package-user-dir', then that code needs to be
+moved to the early init file (see above).
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..ab02d4255b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 8c36c19e82..69bc8fa781 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,12 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+See `user-init-file'.  The only difference is that
+`early-init-file' is not set during the course of evaluating the
+early init file.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +876,103 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (filename-function &optional alternate-filename-function load-defaults)
+  "Load a user init-file.
+FILENAME-FUNCTION is called with no arguments and should return
+the name of the init-file to load.  If this file cannot be
+loaded, and ALTERNATE-FILENAME-FUNCTION is non-nil, then it is
+called with no arguments and should return the name of an
+alternate init-file to load.  If LOAD-DEFAULTS is non-nil, then
+load default.el after the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall filename-function)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (and (eq user-init-file t) alternate-filename-function)
+                   (load (funcall alternate-filename-function)
+                         'noerror 'nomessage))
+
+                 ;; If we did not find the user's init file, set
+                 ;; user-init-file conclusively.  Don't let it be
+                 ;; set from default.el.
+                 (when (eq user-init-file t)
+                   (setq user-init-file init-file-name)))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (funcall read-init-file)
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1124,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1128,153 +1294,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1296,33 +1356,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 7cacd47d51..d009bd0cd2 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4922,7 +4922,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.16.1

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Perhaps some better terminology would be in order? I would suggest
  > that we never refer to "loading" a package, and instead say:

  > * "activating a package" means evaluating its autoloads
  > * "loading a file/feature" means evaluating some particular file of
  >   Lisp code, which is possibly part of a package

I think it is ok to talk about "loading a package"
if that means loading its files.

However, if you evaluate its autoloads, you won't need to explicitly
load its files ever, since the autoloads will do that.  Thus, the
manual may only need to talk about activating it.

I think there should be a way to activate a package explicitly for the
current session _without_ automatically activating it for future
sessions.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> I think there should be a way to activate a package explicitly for the
> current session _without_ automatically activating it for future
> sessions.

The failure of package.el to support this use case is (one reason) why
I wrote straight.el [1]. Another tool which is designed to extend
package.el to allow for temporary activation of packages is Try [2].
If functionality in this area were added directly to package.el, I
think it would be a great improvement.

[1]: https://github.com/raxod502/straight.el
[2]: https://github.com/larstvei/Try

Re: Loading a package applies automatically to future sessions?

[Charles A. Roelli]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sat, 27 Jan 2018 22:30:14 -0800
> 
> > I think there should be a way to activate a package explicitly for the
> > current session _without_ automatically activating it for future
> > sessions.
> 
> The failure of package.el to support this use case is (one reason) why
> I wrote straight.el [1]. Another tool which is designed to extend
> package.el to allow for temporary activation of packages is Try [2].
> If functionality in this area were added directly to package.el, I
> think it would be a great improvement.
> 
> [1]: https://github.com/raxod502/straight.el
> [2]: https://github.com/larstvei/Try

One of the proposals listed in an earlier message was to not insert
the call to package-initialize anymore:

> URL: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
>
> ==> Proposal H: Make the user do it manually

> Summary:
>   - make it so that package.el doesn't modify the init-file, but don't
>     change anything else
>   - this is how it worked before the first "solution" to this problem
>     was implemented

> Advantages:
>   - Emacs does not modify the user's init-file

> Disadvantages:
>   - package.el does not work out of the box, since package
>     customizations will fail when put into the init-file
>   - consequently we get lots of superfluous bug reports

FWIW, taking this route would allow people to install packages in a
session without affecting their initialization file, so it would be
the user's burden to arrange activating the package in future
sessions.  It could at least lead to a better understanding of
package.el for the average user, but as stated, it would probably be
the cause of at least a few "superfluous bug reports".

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>> I think there should be a way to activate a package explicitly for the
>> current session _without_ automatically activating it for future
>> sessions.
> The failure of package.el to support this use case

FWIW, package.el does support this case:
- one way is to prevent package-initialize from activating packages,
  and then activating the ones you want by explicit calls to package-active.
- another is to set package-load-list.  E.g.

    (setq package-load-list '((auctex nil) all))

This said, if a package's activation gets in the way of the user, I'd
generally consider it to be a bug in that package (similar to the fact
that if (require <foo>) has undesirable effects it's usually a bug in
<foo>).

E.g. packages providing some kind of global minor mode should never have
that minor mode enabled just by activating the package: instead the
activation should do nothing more than autoload the minor mode function.


        Stefan

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> This said, if a package's activation gets in the way of the user, I'd
> generally consider it to be a bug in that package (similar to the fact
> that if (require <foo>) has undesirable effects it's usually a bug in
> <foo>).

Activation of packages is a significant performance hit. It's
therefore not a good idea to be activating superfluous packages. JW
has told me that one of the reasons his Emacs configuration does not
use package.el is that activating packages was too slow.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>> This said, if a package's activation gets in the way of the user, I'd
>> generally consider it to be a bug in that package (similar to the fact
>> that if (require <foo>) has undesirable effects it's usually a bug in
>> <foo>).
> Activation of packages is a significant performance hit.
> It's therefore not a good idea to be activating superfluous packages.
> JW has told me that one of the reasons his Emacs configuration does
> not use package.el is that activating packages was too slow.

The performance impact is indeed something that probably deserves a bit
more work.  But we need to clarify what are the expected use cases and
sources of problems.  E.g.:

- In the case of JW, why does he have so many packages installed to slow
  down his startup, yet he doesn't want them activated?  Are these
  packages that he does use, but just rarely?  Or does he just have lots of
  packages installed that he doesn't use (e.g. that's my case: I always
  have all GNU ELPA packages installed, even though I use very few of
  them)?  If so, why does he have so many packages installed even tho he
  doesn't use them?

- In my case I have around 200 packages installed and I measured the
  startup cost of package-initialize to be 0.9s (with a warm cache).
  That's plenty fast for my use case (where I rarely start a new Emacs
  session), but I understand it may not be for people who like to use
  Emacs for quick one-off editing sessions.  Still, what kind of
  slowdown is JW talking about: is it also in the order of 1s?  more?
  If the slowdown is significant, could it be due to packages that do
  "too much work" in the <pkg>-autoloads.el or is it really just the
  sheer number of <pkg>-autoloads.el and <pkg>-pkg.el to process?

The issue came up recently on stackexchange and I whipped up
a quick&dirty patch to experiment (see below) which shows that my 0.9s
can be brought down to 0.3s by pre-computing a "big autoloads file",
which can then be brought down to 0.2s by loading it with
load-source-file-function bound to nil and can be reduced further to
0.1s by byte-compiling it.

Another thing we could do is add support for activating a package
without calling package-initialize beforehand (since just calling
(package-initialize t) can already take a non-negligible amount of time,
reading all the <pkg>-pkg.el files).  This way, a user could simply skip
package-initialize completely and just "manually" activate those few
packages he wants to activate.  It should be fairly easy to implement.


        Stefan


diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..63ec6f40f7 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -676,6 +676,8 @@ package--activate-autoloads-and-load-path
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--fastpath-pkgs t)
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -718,16 +720,19 @@ package-activate-1
             (message "Unable to activate package `%s'.\nRequired package `%s-%s' is unavailable"
                      name (car req) (package-version-join (cadr req)))
             (throw 'exit nil))))
-      (package--load-files-for-activation pkg-desc reload)
-      ;; Add info node.
-      (when (file-exists-p (expand-file-name "dir" pkg-dir))
-        ;; FIXME: not the friendliest, but simple.
-        (require 'info)
-        (info-initialize)
-        (push pkg-dir Info-directory-list))
-      (push name package-activated-list)
-      ;; Don't return nil.
-      t)))
+      (if (listp package--fastpath-pkgs)
+          ;; We're only collecting the set of packages to activate!
+          (push pkg-desc package--fastpath-pkgs)
+        (package--load-files-for-activation pkg-desc reload)
+        ;; Add info node.
+        (when (file-exists-p (expand-file-name "dir" pkg-dir))
+          ;; FIXME: not the friendliest, but simple.
+          (require 'info)
+          (info-initialize)
+          (push pkg-dir Info-directory-list))
+        (push name package-activated-list)
+        ;; Don't return nil.
+        t))))
 
 (declare-function find-library-name "find-func" (library))
 
@@ -3468,6 +3473,49 @@ package-list-packages-no-fetch
   (interactive)
   (list-packages t))
 
+;;;; Fast-path for activation!
+
+(defcustom package-fastpath-file (locate-user-emacs-file "package-fastpath.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package-fastpath-refresh ()
+  "(Re)Generate the `package-fastpath-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (let ((package--fastpath-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ()))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (with-temp-file package-fastpath-file
+      (insert ";;; FastPath file to speed up package activation at startup  -*- lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated, DO NOT EDIT !!\n\n")
+      (dolist (pkg package--fastpath-pkgs)
+        (let* ((file (locate-library (package--autoloads-file-name pkg)))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          (while (search-forward "#$" nil 'move)
+            (replace-match pfile t t))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name package--fastpath-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (insert "\f
+;; Local Variables:
+;; version-control: never
+;; no-byte-compile: nil
+;; no-update-autoloads: t
+;; End:
+"))))
+
 (provide 'package)
 
 ;;; package.el ends here

Re: Loading a package applies automatically to future sessions?

[T.V Raman]

The trick re load-source-file-function is one I did not know.

A few months ago I discovered that 
let-binding 
file-name-handler-alist to nil during startup significantly sped things up.
-- 

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

> - In the case of JW, why does he have so many packages installed to slow
>   down his startup, yet he doesn't want them activated?  Are these
>   packages that he does use, but just rarely?  Or does he just have lots of
>   packages installed that he doesn't use (e.g. that's my case: I always
>   have all GNU ELPA packages installed, even though I use very few of
>   them)?  If so, why does he have so many packages installed even tho he
>   doesn't use them?

JW does not use package.el. Instead he uses git-subtree to manage the
packages and use-package to autoload them. The point I was trying to
demonstrate by bringing this up was that package activation is not
cheap.

> Still, what kind of slowdown is JW talking about: is it also in the
> order of 1s?

I quote:

  Especially when working on modes with complex state, I frequently
  restart Emacs, sometimes hundreds of times a day. Before use-package,
  my load times were upwards of 10-20 seconds. Today, even with >300
  declarations, it's down to 0.44 seconds, without any loss of
  functionality.

-- Radon

Re: Loading a package applies automatically to future sessions?

[John Wiegley]

>>>>> "RR" == Radon Rosborough <radon.neon@gmail.com> writes:

RR> JW does not use package.el. Instead he uses git-subtree to manage the
RR> packages and use-package to autoload them. The point I was trying to
RR> demonstrate by bringing this up was that package activation is not cheap.

As of tow weeks ago, I now use Nix to manage my environment, instead of
subtrees. :) This makes it easier to vary what is installed based on the
combination of Emacs and OS version, and also I can pin individual packages at
specific versions, install custom patches, or add installation details for
packages not in ELPA or MELPA.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>> Still, what kind of slowdown is JW talking about: is it also in the
>> order of 1s?
> I quote:
>   Especially when working on modes with complex state, I frequently
>   restart Emacs, sometimes hundreds of times a day. Before use-package,
>   my load times were upwards of 10-20 seconds. Today, even with >300
>   declarations, it's down to 0.44 seconds, without any loss of
>   functionality.

Hopefully John will chime in to correct me, but my vague understanding
is that his 10-20s were not due to package-initialize, but rather to
"manual installation" methods and configuration.


        Stefan

Re: Loading a package applies automatically to future sessions?

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> Hopefully John will chime in to correct me, but my vague understanding is
SM> that his 10-20s were not due to package-initialize, but rather to "manual
SM> installation" methods and configuration.

It's not entirely due to package-initialize, but that accounted for about 5
seconds of it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> SM> Hopefully John will chime in to correct me, but my vague understanding is
> SM> that his 10-20s were not due to package-initialize, but rather to "manual
> SM> installation" methods and configuration.
> It's not entirely due to package-initialize, but that accounted for about 5
> seconds of it.

Hmm... since my 200 packages take 0.9s on my Thinkpad T61, there's
a good chance that those 5s with your <400 packages are linked to some
poorly packaged packages (more specifically, packages which do too much
work in their <pkg>-autoloads.el, most likely leading not only to a slow
startup but also to undesirable side-effects for those users who
have the package installed yet don't want to use it).

Maybe we should make more efforts at documenting/promoting good
practices in this regard (tho, we should likely first try and figure
out where those 5s are spent, to make sure we don't bark up the wrong
tree).

I'd also be interested to know how much time it takes your machine to
load the "package-fastpath" file that my patch builds (both in source
form and in compiled form).  To distinguish the time taken by the
packages's activation itself from the time taken by package-initialize to
find and fetch that package activation code.


        Stefan

Re: Loading a package applies automatically to future sessions?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> - In the case of JW, why does he have so many packages installed to slow
SM> down his startup, yet he doesn't want them activated? Are these packages
SM> that he does use, but just rarely? Or does he just have lots of packages
SM> installed that he doesn't use (e.g. that's my case: I always have all GNU
SM> ELPA packages installed, even though I use very few of them)? If so, why
SM> does he have so many packages installed even tho he doesn't use them?

I do use them, all of them. At present there are just over 385 in my local
configuration.

However, they don't all need to be active and available immediately after
startup. They fall into several categories, all of which can be deferred:

  1. A set of keybinding(s) that should "activate" the package and make its
     functionality available. Until I press one of those keys, the package
     can stay dormant, and no time should be spent on it.

  2. An entry in auto-mode-alist, interpreter-mode-list, etc.

  3. A command I must invoke via M-x to trigger autoload.

  4. Some other condition, like the loading of one of the aforementioned
     packages.

  5. Waiting until some amount of idle time has passed, since the mode in
     question is just a "bell & whistle", and I shouldn't need to wait for
     it to start using Emacs.

In fact, the packages I require to be immediately available, the instant Emacs
becomes ready, are incredibly few (I'm not even sure there are any!). By
deferring absolutely everything, and autoloading only precisely what is needed
to trigger activation, I can have Emacs available within 0.4 seconds after
startup, with all the other functionality coming online as I perform the
actions that cause them to lazy load.

And before you ask: Yes, I do restart my Emacs. A lot. This is the whole
reason I evolved this technique. Sometimes I restart Emacs over a 100 times a
day, simply because it's the only way I can be completely sure that my state
has been reset (which includes Emacs state, Nix state, shell state, OS state).

Thanks to use-package, I don't need to think very hard to make such a setup
work. And since there's almost no cost to adding in more functionality to my
Emacs, I add tons and tons: all within easy reach should I need it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> Thanks to use-package, I don't need to think very hard to make such a setup
> work.  And since there's almost no cost to adding in more functionality to my
> Emacs, I add tons and tons: all within easy reach should I need it.

We were talking about package activation rather than configuration, so
use-package is not an apples-to-apples comparison.

`package-initialize` is supposed to do things like:

> 1. A set of keybinding(s) that should "activate" the package and make its
>    functionality available. Until I press one of those keys, the package
>    can stay dormant, and no time should be spent on it.
>
> 2. An entry in auto-mode-alist, interpreter-mode-list, etc.
>
> 3. A command I must invoke via M-x to trigger autoload.

and pretty much nothing else, i.e. a subset of what use-package is often
used for.  AFAIK for packages installed with package.el, this part of
use-package only makes sense when the packages is not properly packaged
(i.e. doesn't do 1/2/3 correctly in their <pkg>-autoloads.el).

Regarding the other two points:

> 4. Some other condition, like the loading of one of the aforementioned
>    packages.
>
> 5. Waiting until some amount of idle time has passed, since the mode in
>    question is just a "bell & whistle", and I shouldn't need to wait for
>    it to start using Emacs.

These have to do with configuration, not with activation, so
package-initialize should never do any of that nor prevent doing any of
that.  This part of use-package is just as useful for ELPA-installed
packages as for any others.

> I do use them, all of them. At present there are just over 385 in my local
> configuration.
[...]
> to trigger activation, I can have Emacs available within 0.4 seconds after
> startup, with all the other functionality coming online as I perform the

I don't think I can easily bring down the speed of package-initialize
much below 0.1s for my 200 packages on my Thinkpad T61.  Whether that
would still let you startup in 0.4s with your config, I can't tell.

There's no doubt that with use-package you have a finer control about
what happens when, so you can delay some of the things done during
package-initialize, but without looking more deeply into it, it's hard
to tell whether that would really be needed.


        Stefan

Re: Loading a package applies automatically to future sessions?

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> I don't think I can easily bring down the speed of package-initialize much
SM> below 0.1s for my 200 packages on my Thinkpad T61. Whether that would
SM> still let you startup in 0.4s with your config, I can't tell.

SM> There's no doubt that with use-package you have a finer control about what
SM> happens when, so you can delay some of the things done during
SM> package-initialize, but without looking more deeply into it, it's hard to
SM> tell whether that would really be needed.

True. Note that I did shave an entire second from my startup time just by
adding:

    (setq package-enable-at-startup nil)

I determined this was slowing things down considerably (at least, in my
environment) by running profiler-start as the very first thing in my init
file.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> True. Note that I did shave an entire second from my startup time just by
> adding:
>     (setq package-enable-at-startup nil)

My experiments indicate that we might be able to divide this time by
almost a factor 10 if we pre-compute and byte-compile one big autoloads
file, so it would only shave 0.1s of your startup time.  Given that it
would also let you simplify the rest of your startup, there's a chance
the result would be pretty competitive with your current setup.


        Stefan

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

On 2018-01-29 16:50, Stefan Monnier wrote:
> My experiments indicate that we might be able to divide this time by
> almost a factor 10 if we pre-compute and byte-compile one big autoloads
> file

Yes please :)

Re: Loading a package applies automatically to future sessions?

[T.V Raman]

How would something like 
(make-thread #'package-initialize) 
do in comparison? Effectively that should run on a separate thread and
contribute little to startup time as measured by when emacs is ready to use?
-- 

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

On 2018-01-29 14:08, Stefan Monnier wrote:
> I don't think I can easily bring down the speed of package-initialize
> much below 0.1s for my 200 packages on my Thinkpad T61.

FWIW, activating about 100 packages in my config takes about .4s:

              - package-initialize                                232  89%
               + package-activate                                 136  52%
               + package-read-all-archive-contents                 64  24%
               + package-load-all-descriptors                      24   9%
               + package--build-compatibility-table                  8   3%

More precisely:

              - package-initialize                                232  89%
               - package-activate                                 136  52%
                - package-activate-1                              136  52%
                 - package--load-files-for-activation              96  36%
                  - package--activate-autoloads-and-load-path      68  26%
                   + load                                          68  26%
                  + file-truename                                  28  10%
                 - package-activate                                32  12%
                  + package-activate-1                             32  12%
                 - require                                          8   3%
                  + defvar                                          4   1%
               - package-read-all-archive-contents                 64  24%
                - package-read-archive-contents                    64  24%
                 - package--add-to-archive-contents                44  16%
                  + package--append-to-alist                        4   1%
                   package--read-archive-file                      20   7%
               - package-load-all-descriptors                      24   9%
                - package-load-descriptor                          24   9%
                 + package-process-define-package                   8   3%
                   #<compiled 0xd255c5>                             4   1%
               + package--build-compatibility-table                 8   3%

The reverse tree is also interesting:

+ file-truename                                                    60  23%
+ package--add-to-archive-contents                                 40  15%
  Automatic GC                                                     24   9%
+ package--read-archive-file                                       20   7%
+ load-with-code-conversion                                        16   6%
+ eval-buffer                                                      16   6%
+ package-load-descriptor                                          12   4%
+ package--mapc                                                     8   3%
+ let                                                               4   1%
+ #<compiled 0xd255c5>                                              4   1%
+ package-desc-from-define                                          4   1%
+ version-to-list                                                   4   1%
+ package-desc-priority-version                                     4   1%

It looks likes package--add-to-archive-contents is linear in the number of packages, and called once per package?
I'm not sure why file-truename is called that much.

Clément.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>> I don't think I can easily bring down the speed of package-initialize
>> much below 0.1s for my 200 packages on my Thinkpad T61.
> FWIW, activating about 100 packages in my config takes about .4s:

That's consistent with my 0.9s for 200 packages (tho not with John's 1s
for almost 400 packages), but it of course depends a lot on the details
of your machine (e.g. mine is almost 10 years old, John's is only 3).

> It looks likes package--add-to-archive-contents is linear in the
> number of packages, and called once per package?  I'm not sure why
> file-truename is called that much.

Note that my attempt to bring it down to 0.1s circumvents all that code
(or rather, pre-runs it offline to generate a single "mega-autoloads"
file, so that activation of all packages reduces to loading that one
file).


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Hi all,

I'm the one who posted the question on Stack Exchange, which Stefan
mentioned the other day. My specific question is detailed here:
https://emacs.stackexchange.com/q/38368/10761

So, I've tried out Stefan's patch and it's definitely an improvement on
startup time, but the improvement is not huge for me. Specifically, I've
done the following in my startup files:

(setq package-enable-at-startup nil)
(load "~/.emacs.d/package-fastpath.el")
(setq package--initialized t)

I put that code in my files after running Stefan's
`package-fastpath-refresh'. I also set `load-source-file-function' to
nil as per Stefan's suggestion. Unfortunately, I couldn't byte-compile
my package-fastpath.el file due to some packages which already
byte-compile their autoload files, such as SLIME.

Now, I do see some improvement in my startup time, but to reiterate, it
is not huge. When I run Emacs with my startup commented out,
`emacs-init-time' yields 1.2 seconds. When I run it with my startup
intact, it yields 1.3 or 1.4 seconds. This contrasts with my previous
`emacs-init-time' which was 1.6 or 1.7 seconds, but as you can see, it's
not a really big difference.

It is possible that this sub-par time is due to something on my own
system, but I'm not sure nor convinced of that. Moreover, Stefan's patch
certainly can use some improvements and some notes on how to use his
changes, as it is a bit vexing to figure it out by reading the diff ;)

I did notice a few bugs with Stefan's changes. One bug was that
`package-installed-p' no longer yields correct results on installed
packages. I believe that this is because `package-desc-p' is also broken
by these changes. This broke some code in my startup which I used to
check whether I need to install new packages.

Another bug was that some packages were placed in bad order in
package-fastpath.el. In other words, if a dependency's autoloads are
written to this file after its dependent package, the dependent package
will err, saying that it couldn't require one of its dependencies. There
is obviously some work to do on making sure that dependencies are placed
in the correct order or to change the code so that package-fastpath.el
is order-agnostic.

As a side note and a bit of an opinion, Radon Rosborough made an
interesting remark in one of his messages. He mentioned pip and how
things are done in Python, which really struck me. You never really
think about using a package in a language like Python or Ruby. You just
`require' or `import' it and that's that. It's really simple and the
amount of packages that you have never hurts the startup of the main
program. I know that Emacs is a bit more complicated since it's more of
a text editor than a language and we have somewhat more intricacies to
worry about. But I think that this kind of a model is the kind we need
to be headed for.

A user should be able to have a million packages and not have to worry
about the subsequent startup time. Heck, in my own Ruby installation I
have 436 gems and I've never even thought about startup time till now.

In any case, I think Stefan's ideas and proposed changes are a good idea
and I am in much the same boat as John Wiegley in that I restart Emacs
often enough that startup time gets on my nerves. So I hope that we'll
have some real progress on this issue and make package.el more robust.

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

One more thing in addition to my last message,

I noticed that Stefan's patch does not work on Emacs 25.3, as the second
hunk fails. It looks like the code is different there so I can only
assume that Stefan is on a newer version of Emacs. I have attached the
patch that should work on Emacs 25.3 in case anyone wants to try it on 25.3.


[-- Attachment #2: Stefan's patch adapted for Emacs 25.3 --]
[-- Type: text/x-patch, Size: 3777 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 32200227de..1fa6ce4758 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -656,6 +656,8 @@ PKG-DESC is a `package-desc' object."
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--fastpath-pkgs t)
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -696,16 +698,19 @@ correspond to previously loaded files (those returned by
         (unless (package-activate (car req))
           (error "Unable to activate package `%s'.\nRequired package `%s-%s' is unavailable"
                  name (car req) (package-version-join (cadr req))))))
-    (package--load-files-for-activation pkg-desc reload)
-    ;; Add info node.
-    (when (file-exists-p (expand-file-name "dir" pkg-dir))
-      ;; FIXME: not the friendliest, but simple.
-      (require 'info)
-      (info-initialize)
-      (push pkg-dir Info-directory-list))
-    (push name package-activated-list)
-    ;; Don't return nil.
-    t))
+    (if (listp package--fastpath-pkgs)
+        ;; We're only collecting the set of packages to activate!
+        (push pkg-desc package--fastpath-pkgs)
+      (package--load-files-for-activation pkg-desc reload)
+      ;; Add info node.
+      (when (file-exists-p (expand-file-name "dir" pkg-dir))
+        ;; FIXME: not the friendliest, but simple.
+        (require 'info)
+        (info-initialize)
+        (push pkg-dir Info-directory-list))
+      (push name package-activated-list)
+      ;; Don't return nil.
+      t)))
 
 (declare-function find-library-name "find-func" (library))
 
@@ -3437,6 +3442,51 @@ The list is displayed in a buffer named `*Packages*'."
   (interactive)
   (list-packages t))
 
+;;;; Fast-path for activation!
+
+(defcustom package-fastpath-file (locate-user-emacs-file "package-fastpath.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package-fastpath-refresh ()
+  "(Re)Generate the `package-fastpath-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (let ((package--fastpath-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ()))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (with-temp-file package-fastpath-file
+      (insert ";;; FastPath file to speed up package activation at startup  -*- lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated, DO NOT EDIT !!\n\n")
+      (dolist (pkg package--fastpath-pkgs)
+        (let* ((file (locate-library (package--autoloads-file-name pkg)))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          (while (search-forward "#$" nil 'move)
+            (replace-match pfile t t))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name package--fastpath-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (insert "\f
+;; Local Variables:
+;; version-control: never
+;; no-byte-compile: nil
+;; no-update-autoloads: t
+;; End:
+"))))
+
+
+
 (provide 'package)
 
 ;;; package.el ends here

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> (setq package-enable-at-startup nil)
> (load "~/.emacs.d/package-fastpath.el")
> (setq package--initialized t)

I think setting package--initialized to t is wrong, here (IIRC
package--initialized means that we've filled the tables keeping track
of which packages are available and such).  Not a problem when you're
just trying to test the speed of my proof-of-concept patch, of course.

> Unfortunately, I couldn't byte-compile my package-fastpath.el file due
> to some packages which already byte-compile their autoload files, such
> as SLIME.

Hmm... are you saying that my code ends up fetching code for
slime-autoloads.elc instead of slime-autoloads.el?
Or that slime-autoloads.el contains (pre)compiled code?

Also, I'm not 100% surprised that byte-compiling byte-compiled code would
fail, but I can't see any immediate reason why it should fail, so it's
quite possible to it'd be easy to make it work.

> Now, I do see some improvement in my startup time, but to reiterate, it
> is not huge.  When I run Emacs with my startup commented out,
> `emacs-init-time' yields 1.2 seconds.

Not quite sure what "my startup commented out" means.  I'll assume it
means a more or less empty ~/.emacs, or maybe the use of -Q ?
In any case I guess it means that 1.2s is the "speed of light", i.e. the
holy grail of package.el on your machine.

> When I run it with my startup intact, it yields 1.3 or 1.4
> seconds. This contrasts with my previous `emacs-init-time' which was
> 1.6 or 1.7 seconds, but as you can see, it's not a really
> big difference.

If you subtract the 1.2s taken by "other things", it means that your
config used to take 0.4-0.5s of startup time, and that the use of a big
precomputed autoloads file reduced that to 0.1-0.2s, so sped it up by
factor between 2.5 and 4, which I find a bit disappointing but still
respectable (and it depends which proportion was taken by
package-initialize vs which proportion was taken by your particular
customizations).

> It is possible that this sub-par time is due to something on my own
> system, but I'm not sure nor convinced of that. Moreover, Stefan's patch
> certainly can use some improvements and some notes on how to use his
> changes, as it is a bit vexing to figure it out by reading the diff ;)

Yes, it needs work.  It's just a quick experiment to see the
potential gains.

> I did notice a few bugs with Stefan's changes.  One bug was that
> `package-installed-p' no longer yields correct results on installed
> packages.

As it stands, my patch only pre-sets package-activated-list, whereas
package-installed-p requires other things set up by package-initialize.
If you don't set package--initialized to t, you'll see that
package-installed-p will rightfully complain that you haven't called
package-initialize.

I guess my patch could also add a setting for `package-alist` to the
package-fastpath.el, but I'm not sure I like the idea (it increases the
consequences of having an out-of-date package-fastpath.el).

> I believe that this is because `package-desc-p' is also broken
> by these changes. This broke some code in my startup which I used to
> check whether I need to install new packages.

Wouldn't it be better for your code to assume that if there's
a package-fastpath.el, then those checks have already been performed
(and to re-execute those checks in package-fastpath-refresh instead)?

Or maybe package-installed-p should be changed such that when it's
called with a symbol argument, it should first check if it's in
package-activated-list, and if it's not, then it should call
package-initialize before checking package-alist.

> Another bug was that some packages were placed in bad order in
> package-fastpath.el. In other words, if a dependency's autoloads are
> written to this file after its dependent package, the dependent package
> will err, saying that it couldn't require one of its dependencies.

Hmm... I thought this can't happen because the files are concatenated in
the same order that they are normally loaded by `package-initialize`.
I guess something doesn't work the way I thought it does.
Can you investigate to see when or even why it happens?

> As a side note and a bit of an opinion, Radon Rosborough made an
> interesting remark in one of his messages. He mentioned pip and how
> things are done in Python, which really struck me. You never really
> think about using a package in a language like Python or Ruby. You just
> `require' or `import' it and that's that. It's really simple and the
> amount of packages that you have never hurts the startup of the main
> program. I know that Emacs is a bit more complicated since it's more of
> a text editor than a language and we have somewhat more intricacies to
> worry about. But I think that this kind of a model is the kind we need
> to be headed for.

The fact that it's a text editor shouldn't make any difference in this
respect.  I'm not sure exactly what you mean by "the amount of packages
that you have never hurts the startup of the main program", I guess you
mean that program A is not slowed down when you install extra packages.

In Emacs the corresponding behavior happens if you install a package by
hand and don't activate/use it.  `package.el` by default tries to
auto-activate all your packages, so indeed it's slowed down by the mere
presence of extra packages.

I think a key element that lets Python and Ruby work that way is the
namespacing and its tight binding to the filesystem.  Currently, while
conventions are "similar" (e.g. function toto-titi is likely found in
file toto*.el in package toto), they're just vague conventions.

Making them more strict would indeed make it possible to change
`require` so that a (require 'toto-tata) will automatically look for
a file toto-tata.el in a package toto without having to "activate" that
package beforehand and it would let us have a "missing function handler"
which would automatically try to find the function toto-titi in one of
the files of the titi package (e.g. by activating this package).

This might make it possible to activate some packages lazily, but
There'd still be aspects of auto-activation that would go missing, tho:
e.g. registering jgraph-mode as a handler for *.jgr files.
So some packages would still need some form of auto-activation.

> A user should be able to have a million packages and not have to worry
> about the subsequent startup time. Heck, in my own Ruby installation I
> have 436 gems and I've never even thought about startup time till now.

The downside is that you have to explicitly `import` those gems when you
need them.  In Emacs you could do the same: call (package-initialize t)
in your ~/.emacs and then call `package-activate` when you need
a particular package.

> In any case, I think Stefan's ideas and proposed changes are a good idea
> and I am in much the same boat as John Wiegley in that I restart Emacs
> often enough that startup time gets on my nerves.

IIUC there are 1.2s of your startup time which are spent without even
running a single line of package.el code, so speeding up package.el
might not be good enough (at least it seems to me that if 1.6s is too
slow, then 1.2s is likely to also be too slow).

The usual answer in Emacs for those problem is "don't do that"
(i.e. use emacsclient instead).  I understand it's not always an option.


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> I think setting package--initialized to t is wrong, here (IIRC
> package--initialized means that we've filled the tables keeping track
> of which packages are available and such).  Not a problem when you're
> just trying to test the speed of my proof-of-concept patch, of course.

Ok, thank you for letting me know the implications of doing that. I had
to do it because I got some complaints from Emacs about my packages not
being initialized. I can post the exact errors if you'd like.

> Hmm... are you saying that my code ends up fetching code for
> slime-autoloads.elc instead of slime-autoloads.el?
> Or that slime-autoloads.el contains (pre)compiled code?

It looks like your code is fetching slime-autoloads.elc:

(let ((load-file-name "/Users/my_username/.emacs.d/elpa/slime-20180126.1033/slime-autoloads.elc"))
;; ... bunches of bytecode
)

> Also, I'm not 100% surprised that byte-compiling byte-compiled code would
> fail, but I can't see any immediate reason why it should fail, so it's
> quite possible to it'd be easy to make it work.

Not sure why it would fail either, but this is the error that I see for
that specific chunk of bytecode:

package-fastpath.el:899:1:Error: Wrong type argument: sequencep, 1004

I could attach the bytecode in another message if you'd like to see it,
but I'm not sure whether you do.

> Not quite sure what "my startup commented out" means.  I'll assume it
> means a more or less empty ~/.emacs, or maybe the use of -Q ?
> In any case I guess it means that 1.2s is the "speed of light", i.e. the
> holy grail of package.el on your machine.

I mean that I commented out my ~/.emacs file, which disables all
initialization code that I run, including in other files. It does _not_
mean using -Q, which brings my startup time to 0.5 seconds (FYI, I have
recently been using the Emacs Mac Port by Yamamoto Mitsuharu so my
startup is a bit slower than vanilla GNU Emacs). That therefore means
that 1.2s is how fast Emacs is with no customizations from myself in
effect. -Q and -q have other implications, which are detailed in the
Elisp manual under "38.1.1 Summary: Sequence of Actions at Startup"

I actually just realized those 1.2 seconds include running
`package-initialize', as per usual. In other words, it was not taking
advantage of package-fastpath.el. I just ran a modified version of my
~/.emacs file which comments out almost everything except:

(load-library "package") ;; I need this else Emacs says that
                         ;; `package-activated-list' is void
(setq package-enable-at-startup nil)
(load "~/.emacs.d/package-fastpath.el")
(setq package--initialized t)

And lo and behold, `emacs-init-time' yields 0.7 seconds. Which is just
0.2 seconds higher than what I get with -Q, which I believe is Emacs'
peak startup time. I also verified that I have access to all my packages
so it is using package-fastpath.el.

So that means your patch has actually increased startup time a lot more
than I initially thought and it's actually my own initialization code
which is bogging things down.

> If you subtract the 1.2s taken by "other things", it means that your
> config used to take 0.4-0.5s of startup time, and that the use of a big
> precomputed autoloads file reduced that to 0.1-0.2s, so sped it up by
> factor between 2.5 and 4, which I find a bit disappointing but still
> respectable (and it depends which proportion was taken by
> package-initialize vs which proportion was taken by your particular
> customizations).

Well, actually those figures are better than what you're saying as per
my information above. I guess that makes your patch even more
respectable for a first stab! :)

I guess since my startup time with my initialization code is 1.3
seconds, and running Emacs with package-fastpath.el is 0.7 seconds, my
own code is taking up about 0.5 to 0.6 seconds.

> As it stands, my patch only pre-sets package-activated-list, whereas
> package-installed-p requires other things set up by package-initialize.
> If you don't set package--initialized to t, you'll see that
> package-installed-p will rightfully complain that you haven't called
> package-initialize.

Ok, thank you for clarifying that issue. Although, as I mentioned above,
I had to set package--initialized to t, otherwise Emacs would halt
startup and complain about me not calling `package-initialize'.

> I guess my patch could also add a setting for `package-alist` to the
> package-fastpath.el, but I'm not sure I like the idea (it increases the
> consequences of having an out-of-date package-fastpath.el).

I'm not sure on that either. Although, we could make it so that
`package-fastpath-refresh' is run more often or in more places so as to
lessen that risk.

> Wouldn't it be better for your code to assume that if there's
> a package-fastpath.el, then those checks have already been performed
> (and to re-execute those checks in package-fastpath-refresh instead)?

Yes, it would likely be better for my code to do that instead.

> Or maybe package-installed-p should be changed such that when it's
> called with a symbol argument, it should first check if it's in
> package-activated-list, and if it's not, then it should call
> package-initialize before checking package-alist.

I can't give a strong opinion on that. Perhaps someone else who is
better-versed on package.el could shed some light here?

> Hmm... I thought this can't happen because the files are concatenated in
> the same order that they are normally loaded by `package-initialize`.
> I guess something doesn't work the way I thought it does.
> Can you investigate to see when or even why it happens?

It specifically happened with gh.el (https://github.com/sigma/gh.el),
which is dependent on marshal.el (https://github.com/sigma/marshal.el)

Apparently gh-autoloads.el uses a `gh-defclass' macro, which in turn
uses a `marshal-defclass' macro. There is definitely some strangeness
that could be going on here.

> The fact that it's a text editor shouldn't make any difference in this
> respect.  I'm not sure exactly what you mean by "the amount of packages
> that you have never hurts the startup of the main program", I guess you
> mean that program A is not slowed down when you install extra packages.

Yes, I mean that program A is not slowed down by extra packages.

> In Emacs the corresponding behavior happens if you install a package by
> hand and don't activate/use it.  `package.el` by default tries to
> auto-activate all your packages, so indeed it's slowed down by the mere
> presence of extra packages.

Right.

> I think a key element that lets Python and Ruby work that way is the
> namespacing and its tight binding to the filesystem.  Currently, while
> conventions are "similar" (e.g. function toto-titi is likely found in
> file toto*.el in package toto), they're just vague conventions.

Yes, Ruby and Python have a simpler "path" system for finding packages
than Emacs does currently.

> Making them more strict would indeed make it possible to change
> `require` so that a (require 'toto-tata) will automatically look for
> a file toto-tata.el in a package toto without having to "activate" that
> package beforehand and it would let us have a "missing function handler"
> which would automatically try to find the function toto-titi in one of
> the files of the titi package (e.g. by activating this package).

> This might make it possible to activate some packages lazily, but
> There'd still be aspects of auto-activation that would go missing, tho:
> e.g. registering jgraph-mode as a handler for *.jgr files.
> So some packages would still need some form of auto-activation.

Right, your last point is what I meant by "we have somewhat more
intricacies to worry about."

> The downside is that you have to explicitly `import` those gems when you
> need them.  In Emacs you could do the same: call (package-initialize t)
> in your ~/.emacs and then call `package-activate` when you need
> a particular package.

Yes, this is true. Still, I think we need some more simplicity in this
system of autoloads and so forth.

> IIUC there are 1.2s of your startup time which are spent without even
> running a single line of package.el code, so speeding up package.el
> might not be good enough (at least it seems to me that if 1.6s is too
> slow, then 1.2s is likely to also be too slow).

Well, those 1.2 seconds which I mentioned are just starting up Emacs
without any _initialization_ code. And that also means that
`package-initialize' is running and not taking advantage of
package-fastpath.el. Again, as I mentioned above, the startup time which
takes advantage of package-fastpath.el is ~0.7 seconds so we have
already sped up package.el by a lot and with noticeable effect.

> The usual answer in Emacs for those problem is "don't do that"
> (i.e. use emacsclient instead).  I understand it's not always an
> option.

Yes, I actually am an avid user of the Emacs client and server. I think
it's a great feature which helps me do a lot of useful
things. Unfortunately, it is not the panacea for startup time woes that
it is often touted as. I know it is that way for a lot of people, but I
unfortunately push Emacs too hard most of the time to be able to leave
it running for weeks or months. If that were the case, I wouldn't have
this itch to scratch.

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Stefan,

As an addition to my last message, I noticed another bug with your patch
that I did not notice yesterday. The info files for my packages from
ELPA are all missing in the info Directory node. When I use
`(package-initialize)' rather than package-fastpath.el, those info nodes
are all there. `(package-initialize t)' also did not bring those info
nodes back. Evidently it is the package activation which is significant
here. There is clearly a lot of work done regarding package activation
from `package-initialize' which we need to incorporate.

I would like to contribute some fixes to your code, but I am a bit
hesitant because of copyright. If I send improvements of your patch,
will it be able to get back into package.el? Because I will not sign any
papers which give my rights to the FSF nor will I license any code that
I write under the GPL. I am fine with licensing any code that I write
under a permissive or free license such as New BSD, MIT, or public
domain, but I will not put it under the GPL. I hate to bring up such a
sticky topic, but I would rather get it out of the way now, before I
write a bunch of code which could ultimately be useless because of
copyright disagreements.

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> I would like to contribute some fixes to your code, but I am a bit
> hesitant because of copyright. If I send improvements of your patch,
> will it be able to get back into package.el? Because I will not sign any
> papers which give my rights to the FSF nor will I license any code that
> I write under the GPL.

Unless your code is sufficiently trivial, we require a copyright
assignment to incorporate your code into Emacs.


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> Unless your code is sufficiently trivial, we require a copyright
> assignment to incorporate your code into Emacs.

Right, that's what I was afraid of. Unfortunately, my improvements to
your code would quite likely be greater than 15 lines so I doubt I'll be
able to help you aside from giving ideas and feedback.

It's a real shame that the FSF has put this policy in place, but I'll
stop myself there before this becomes a rant.

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Also, I am legally under-age to sign any such copyright waivers. So I
couldn't do it even if I wanted to. Another reason that this is a
fruitless policy.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> Also, I am legally under-age to sign any such copyright waivers.

Since it's the second time this issue shows up, I finally asked the
copyright clerk at the FSF and he says that it works as follows:

    If the person is under 18 years old their parent needs to also sign
    the contract.  The biggest sticking point is that the assignment is
    only valid up until that person's 18th birthday.

> So I couldn't do it even if I wanted to.

Apparently you could, but it's kind of a pain, indeed.

> Another reason that this is a fruitless policy.

I also find it fruitless/useless/annoying, but for having tried to fight
it, I know that it's just as fruitless trying to change Richard's mind
about it.

And I find it very hard to imagine how signing such paperwork for one's
Emacs code could harm one, so I sometimes get stuck this way between
Richard who stubbornly wants copyright paperwork and the contributor who
stubbornly refuses to sign it.

Both positions are only "a matter of principle" and are fruitless.

Anyway, happy hacking to you,


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> > Also, I am legally under-age to sign any such copyright waivers.

> Since it's the second time this issue shows up, I finally asked the
> copyright clerk at the FSF and he says that it works as follows:

>     If the person is under 18 years old their parent needs to also sign
>     the contract.  The biggest sticking point is that the assignment is
>     only valid up until that person's 18th birthday.

> > So I couldn't do it even if I wanted to.

> Apparently you could, but it's kind of a pain, indeed.

Yes, that is certainly a pain to say the least. In some cases it makes
contribution an impossibility.

> I also find it fruitless/useless/annoying, but for having tried to fight
> it, I know that it's just as fruitless trying to change Richard's mind
> about it.

> And I find it very hard to imagine how signing such paperwork for one's
> Emacs code could harm one, so I sometimes get stuck this way between
> Richard who stubbornly wants copyright paperwork and the contributor who
> stubbornly refuses to sign it.

> Both positions are only "a matter of principle" and are fruitless.

I sympathize with your difficulties in this regard. Honestly, I wouldn't
have much of a problem contributing code which would be licensed under
the GPL if the paperwork were not involved. I still think the GPL is
too problematic and restrictive, but to each his own.

This paperwork is really just over-the-top and it's not right to force
someone to give you the rights to their code while in the same breath
talking about freedom. This kind of stuff doesn't encourage any openness
or "hacker culture." It just makes people shy away from the project. I
guess all of that is a matter of principle on my part, but aside from
that I also do not want to encourage the FSF's continuation of this
policy by signing the papers (which I still cannot anyway).

Anyhow, happy hacking to you as well.

I'd rather get back on the topic of your patch now that we've gotten
this issue out of the way. Do you need any more feedback or assistance
in solving the bugs that I mentioned?

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> I'd rather get back on the topic of your patch now that we've gotten
> this issue out of the way. Do you need any more feedback or assistance
> in solving the bugs that I mentioned?

Yes (but I haven't gotten to it yet, sorry),


        Stefan

Re: Loading a package applies automatically to future sessions?

[Stephen Berman]

On Thu, 01 Feb 2018 12:47:23 -0500 George Plymale II <georgedp@orbitalimpact.com> wrote:

> This paperwork is really just over-the-top and it's not right to force
> someone to give you the rights to their code while in the same breath
> talking about freedom. 

Assigning copyright to the FSF does not prevent the assignee from doing
anything with their code (except claiming copyright ownership of it), see
<https://www.fsf.org/bulletin/2014/spring/copyright-assignment-at-the-fsf>. in
particular:

   Sometimes contributors are concerned about giving up rights to their
   work. As the assignment is a gift to the free software community, they
   don't want it to come at the expense of having flexibility in the use of
   their own code. Thus, we grant back to contributors a license to use
   their work as they see fit. This means they are free to modify, share,
   and sublicense their own work under terms of their choice. This enables
   contributors to redistribute their work under another free software
   license. While this technically also permits distributing their work
   under a proprietary license, we hope they won't.

>                        This kind of stuff doesn't encourage any openness
> or "hacker culture." It just makes people shy away from the project. I
> guess all of that is a matter of principle on my part, but aside from
> that I also do not want to encourage the FSF's continuation of this
> policy by signing the papers (which I still cannot anyway).

Are you sure you understand the FSF policy?

Steve Berman

Re: Loading a package applies automatically to future sessions?

[Stephen Berman]

On Thu, 01 Feb 2018 20:19:01 +0100 Stephen Berman <stephen.berman@gmx.net> wrote:

> On Thu, 01 Feb 2018 12:47:23 -0500 George Plymale II
> <georgedp@orbitalimpact.com> wrote:
>
>> This paperwork is really just over-the-top and it's not right to force
>> someone to give you the rights to their code while in the same breath
>> talking about freedom. 
>
> Assigning copyright to the FSF does not prevent the assignee from doing
                                                oops: assigner
> anything with their code (except claiming copyright ownership of it), see

Steve Berman

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

>    [...] Thus, we grant back to contributors a license to use
>    their work as they see fit. This means they are free to modify, share,
>    and sublicense their own work under terms of their choice. This enables
>    contributors to redistribute their work under another free software
>    license. While this technically also permits distributing their work
>    under a proprietary license, we hope they won't.
   
That sounds interesting, but I would like to know more of the specifics
about that. Is there a place where I can find very detailed information
about this specific "license" which is "granted back to contributors?"

> Are you sure you understand the FSF policy?

I'm not sure what you're implying.

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Stephen Berman]

On Thu, 01 Feb 2018 16:44:04 -0500 George Plymale II <georgedp@orbitalimpact.com> wrote:

>>    [...] Thus, we grant back to contributors a license to use
>>    their work as they see fit. This means they are free to modify, share,
>>    and sublicense their own work under terms of their choice. This enables
>>    contributors to redistribute their work under another free software
>>    license. While this technically also permits distributing their work
>>    under a proprietary license, we hope they won't.
>    
> That sounds interesting, but I would like to know more of the specifics
> about that. Is there a place where I can find very detailed information
> about this specific "license" which is "granted back to contributors?"

I don't know of an online resource with such details, but maybe it's
somewhere in the FSF website.  There is a statement to that effect in
the assignment agreement itself, at least in the paper version I
received when I made my assignment many years ago:

   FSF agrees to grant back to Developer, and does hereby grant,
   non-exclusive, royalty-free and non-cancellable rights to use the Works
   (i.e., Developer's changes and/or enhancements, not the Program that
   they enhance), as Developer sees fit [...]

>> Are you sure you understand the FSF policy?
>
> I'm not sure what you're implying.

Your wording ("it's not right to force someone to give you the rights to
their code while in the same breath talking about freedom", "This kind
of stuff doesn't encourage any openness or 'hacker culture.'") suggests
a different understanding of the FSF policy than what the FSF itself
expounds, as exemplified by the passage I quoted above.

Steve Berman

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> I don't know of an online resource with such details, but maybe it's
> somewhere in the FSF website.  There is a statement to that effect in
> the assignment agreement itself, at least in the paper version I
> received when I made my assignment many years ago:

>    FSF agrees to grant back to Developer, and does hereby grant,
>    non-exclusive, royalty-free and non-cancellable rights to use the Works
>    (i.e., Developer's changes and/or enhancements, not the Program that
>    they enhance), as Developer sees fit [...]

Huh, well I wasn't aware of that and it sounds fine to me. Are you sure
that there's no strings attached?

> Your wording ("it's not right to force someone to give you the rights to
> their code while in the same breath talking about freedom", "This kind
> of stuff doesn't encourage any openness or 'hacker culture.'") suggests
> a different understanding of the FSF policy than what the FSF itself
> expounds, as exemplified by the passage I quoted above.

Well, my understanding of the policy was that one's _contributions_ are
owned by the FSF. I.e., that you have to sign waivers which tell the
FSF, "Hey, I give up all rights to own any code that I give you guys in
these certain projects." To me, that seems hypocritical and it seemed
that indeed the actual FSF policy versus what the FSF itself expounds
were in disagreement. But, maybe I'm incorrect about that if that above
passage is really what it sounds like.

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: George Plymale II <georgedp@orbitalimpact.com>
> Date: Thu, 01 Feb 2018 18:02:12 -0500
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > I don't know of an online resource with such details, but maybe it's
> > somewhere in the FSF website.  There is a statement to that effect in
> > the assignment agreement itself, at least in the paper version I
> > received when I made my assignment many years ago:
> 
> >    FSF agrees to grant back to Developer, and does hereby grant,
> >    non-exclusive, royalty-free and non-cancellable rights to use the Works
> >    (i.e., Developer's changes and/or enhancements, not the Program that
> >    they enhance), as Developer sees fit [...]
> 
> Huh, well I wasn't aware of that and it sounds fine to me. Are you sure
> that there's no strings attached?

There's no other wording related to that, and the text is pretty
clear.  So I see no other strings that could be attached.  You, as the
author, retain full rights to do anything you like with the code.

> Well, my understanding of the policy was that one's _contributions_ are
> owned by the FSF. I.e., that you have to sign waivers which tell the
> FSF, "Hey, I give up all rights to own any code that I give you guys in
> these certain projects."

Given the above, I believe you now realize that this was a mistaken
interpretation.  You retain all the rights, you just give the FSF the
ability to enforce the copyright if needed.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> There's no other wording related to that, and the text is pretty
> clear.  So I see no other strings that could be attached.  You, as the
> author, retain full rights to do anything you like with the code.

Ok, that sounds good. I'd still like to take a look at this myself so
that I can verify for myself that there are no strings attached.

> Given the above, I believe you now realize that this was a mistaken
> interpretation.  You retain all the rights, you just give the FSF the
> ability to enforce the copyright if needed.

Which copyright does one give the FSF the ability to enforce? And what
does that entail?

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

>> Given the above, I believe you now realize that this was a mistaken
>> interpretation.  You retain all the rights, you just give the FSF the
>> ability to enforce the copyright if needed.
> Which copyright does one give the FSF the ability to enforce?

Only the copyright holder can sue someone for infringement.
AFAIK the things you lose when transferring your copyright to the FSF are:
- the ability to transfer your copyright to someone else (because only
  the copyright holder can transfer his copyright to someone else (note
  that the copyright holder cannot not "copy" his copyright he can only
  transfer it)).  This is not something the FSF really needs/wants which
  is why there is that clause that says that the FSF gives you the right
  to do anything you like with your work (i.e. to gives you back as
  much of the copyright holder's rights as it legally can).
- the ability to sue someone for infringement (because only the
  copyright holder can do that) which is one of the main reasons the FSF
  wants to have the copyright.
- the ability to prevent the FSF from licensing your work under some
  licence with which you happen to disagree (tho, IIRC, the paperwork
  you sign does include some clause promising that your work will stay
  under some kind of Free licence, tho, to try and make sure that
  there's not much point for <yourfavoriteevilcompany> to buy/infiltrate the
  FSF, since changing the licenses to be non-Free would break all those
  copyright assignments).


        Stefan

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: George Plymale II <georgedp@orbitalimpact.com>
> Cc: stephen.berman@gmx.net, monnier@IRO.UMontreal.CA,
> 	emacs-devel@gnu.org
> Date: Fri, 02 Feb 2018 12:19:25 -0500
> 
> > Given the above, I believe you now realize that this was a mistaken
> > interpretation.  You retain all the rights, you just give the FSF the
> > ability to enforce the copyright if needed.
> 
> Which copyright does one give the FSF the ability to enforce? And what
> does that entail?

By assigning the copyright to your code to the FSF, you give them the
ability to enforce adherence to the GPL for the package to which you
contributed.  That's because, since the FSF holds the copyright to
every line of code in the package, it can prosecute anyone who
violates the GPL for that package.

Re: Loading a package applies automatically to future sessions?

[Phillip Lord]

George Plymale II <georgedp@orbitalimpact.com> writes:

>>    FSF agrees to grant back to Developer, and does hereby grant,
>>    non-exclusive, royalty-free and non-cancellable rights to use the Works
>>    (i.e., Developer's changes and/or enhancements, not the Program that
>>    they enhance), as Developer sees fit [...]
>
> Huh, well I wasn't aware of that and it sounds fine to me. Are you sure
> that there's no strings attached?

There are some strings -- for instance, you are guaranteeing that you
actually own the copyright in the first place. I didn't know about the
grant back, incidentally. For Emacs, it makes not practical
difference. As far as I can see, it's not possible to release Emacs code
that is not GPL or public domain anyway, since it automatically links
with Emacs.

>
>> Your wording ("it's not right to force someone to give you the rights to
>> their code while in the same breath talking about freedom", "This kind
>> of stuff doesn't encourage any openness or 'hacker culture.'") suggests
>> a different understanding of the FSF policy than what the FSF itself
>> expounds, as exemplified by the passage I quoted above.
>
> Well, my understanding of the policy was that one's _contributions_ are
> owned by the FSF. I.e., that you have to sign waivers which tell the
> FSF, "Hey, I give up all rights to own any code that I give you guys in
> these certain projects." To me, that seems hypocritical and it seemed
> that indeed the actual FSF policy versus what the FSF itself expounds
> were in disagreement. But, maybe I'm incorrect about that if that above
> passage is really what it sounds like.

It does require copyright assignment, yes. Whether that is hypocritical
or not is up to you. For Emacs, copyright assignment makes little or not
difference to what you can do with the code. So I don't think that's an
issue.

It is a significant practical issue for sure and involves a lot of
book-keeping for package maintainers.

Phil

Re: Loading a package applies automatically to future sessions?

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

> As far as I can see, it's not possible to release Emacs code
> that is not GPL or public domain anyway, since it automatically links
> with Emacs.

That's a matter of opinion. My Emacs packages are released under the
MIT License.

[-- Attachment #2: Type: text/html, Size: 336 bytes --]

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Well, my understanding of the policy was that one's _contributions_ are
  > owned by the FSF. I.e., that you have to sign waivers which tell the
  > FSF, "Hey, I give up all rights to own any code that I give you guys in
  > these certain projects." To me, that seems hypocritical and it seemed
  > that indeed the actual FSF policy versus what the FSF itself expounds
  > were in disagreement.

If you accuse us of hypocrisy, you should be prepared to present
particulars.  What is the principle that the FSF stands for, that
conflicts with our practices?

My hunch is that you've been misinformed about our principles.
If you say what you believe our principles to be, we can compare
them with our real principles.

Our main principle is: users should have control of the software
they use.  Therefore, a nonfree program is an injustice.

To eliminate that injustice, we release free software and encourage
others to do so.  We use copyleft to prevent our free software from
being perverted by middlemen into nonfree software and thus used to
subjugate others.

As for copyright, we do not consider that an issue of principle, not
directly.  We consider it a sort of weapon that can be used for good
or for bad.  When it is used to subjugate people, that is bad.  When
it is used, through copyleft, to protect people from subjugation, that
is good.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

A response to RMS (was Loading a package applies automatically to future sessions?)

[George Plymale II]

>   > Well, my understanding of the policy was that one's _contributions_ are
>   > owned by the FSF. I.e., that you have to sign waivers which tell the
>   > FSF, "Hey, I give up all rights to own any code that I give you guys in
>   > these certain projects." To me, that seems hypocritical and it seemed
>   > that indeed the actual FSF policy versus what the FSF itself expounds
>   > were in disagreement.

> If you accuse us of hypocrisy, you should be prepared to present
> particulars.  What is the principle that the FSF stands for, that
> conflicts with our practices?

Well, one of the principles which I believed you to be in contradiction
of was this:

"You should also have the freedom to make modifications and use them
privately in your own work or play, without even mentioning that they
exist. If you do publish your changes, you should not be required to
notify anyone in particular, or in any particular way."

( from https://www.gnu.org/philosophy/free-sw.html )

According to my understanding of the FSF's copyright policy, I have to
notify the FSF when I want to distribute my changes because they're no
longer mine; they're theirs.

Moreover, I believed you to be in contradiction of this principle as
well:

"I cannot in good conscience sign a nondisclosure agreement or a
software license agreement."

( from https://www.gnu.org/gnu/manifesto.en.html )

To me, the signing of a waiver feels like it's in much the same vein as
either of those. If not a bit worse in some ways. In light of the new
information that I received from others on this mailing list, it may not
be as bad as I thought.

Regardless, that is what I meant by "hypocrisy" and it is not a word
that I use lightly nor without cause.

> My hunch is that you've been misinformed about our principles.

Your hunch is incorrect. I ascertained my opinions on the GNU project
and the FSF mainly via objective analysis of their own documents and
manifestos. It was also coupled with some examination of how their
values worked out in reality, along with comparisons to other
philosophies and facts. My conclusion of these analyses was mostly what
I have already related in previous messages.

> Our main principle is: users should have control of the software
> they use.  Therefore, a nonfree program is an injustice.

> To eliminate that injustice, we release free software and encourage
> others to do so.  We use copyleft to prevent our free software from
> being perverted by middlemen into nonfree software and thus used to
> subjugate others.

> As for copyright, we do not consider that an issue of principle, not
> directly.  We consider it a sort of weapon that can be used for good
> or for bad.  When it is used to subjugate people, that is bad.  When
> it is used, through copyleft, to protect people from subjugation, that
> is good.

All users have control over the software that they use. Oftentimes, even
hardware cannot stop one from modifying the software which runs atop
it. Enough ingenuity and craftiness trumps even the likes of
Tivo. Whether or not this is legal is an entirely different question and
forms the basis of the GNU campaign.

I know that many will read my statement and scoff. Ask yourself, though:
what stops a user from modifying the bits that run on their computer?
Not the law. It is knowledge. Knowledge of how a computer works and how
well its machine code can be understood. Indeed, the GNU project's
efforts and funds would be far better spent creating tools that would
allow users to universally understand machine code that would truly
allow them to control any software that they have, regardless of its
underlying machinery. This would enable freedom of software in a much
more tangible and truer sense than anything that anyone has done. Ever.

But instead the whole affair has become an argument of copyright or
copyleft. The ability to own capital and sell it or to have that capital
controlled and dolled out by one big entity (or maybe a few) who claim
to be in the interests of the people. Or we may call those people
"users."

Sound familiar? If not, I would highly suggest reading John Wiegley's
poignant and cogent letter to the FSF:
http://newartisans.com/2011/04/letter-to-the-fsf/

I didn't even want to write all of this until you stipulated that my
opinions are based on misinformation and smear campaigns done by others
to the GNU project. Such a presumption insults the intellect of very
intelligent people who I know that share my opinion and who formed the
same opinion based on rational, moral, and objective analysis.

In any case, I no longer desire to discuss these things. My opinion and
your opinion have both been stated. I doubt that either of us will
change them based on the words of the other. I am no longer as
dissatisfied or upset with the FSF's policy as I was, although I will
still need to investigate the agreement for myself to determine whether
it is indeed a fair agreement. That does not, however, convince me of
the merits of copyleft nor the philosophies which I have already
expressed disagreement with.

Thanks,
- George Plymale II

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[Clément Pit-Claudel]

On 2018-02-05 04:17, George Plymale II wrote:
> According to my understanding of the FSF's copyright policy, I have
> to notify the FSF when I want to distribute my changes because
> they're no longer mine; they're theirs.

I think you misunderstood.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[George Plymale II]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> On 2018-02-05 04:17, George Plymale II wrote:
>> According to my understanding of the FSF's copyright policy, I have
>> to notify the FSF when I want to distribute my changes because
>> they're no longer mine; they're theirs.
>
> I think you misunderstood.

Sorry, I meant to write that sentence in the past tense. I incorrectly
wrote it in the present tense, as that statement _was_ my understanding,
but it is no longer. It was clarified to me by other messages on this
thread that this is not part of the policy. Hence, I wrote the word
"hypocritical" at the time when I did not have that clarified to me.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > "You should also have the freedom to make modifications and use them
  > privately in your own work or play, without even mentioning that they
  > exist. If you do publish your changes, you should not be required to
  > notify anyone in particular, or in any particular way."

  > ( from https://www.gnu.org/philosophy/free-sw.html )

All our software releaes give you this; they give this
to the public at large.

  > According to my understanding of the FSF's copyright policy, I have to
  > notify the FSF when I want to distribute my changes because they're no
  > longer mine; they're theirs.

You don't have to notify the FSF to redistribute our software releases
under the GPL.

The unlimited nonexclusive license in the assignment contracts
concerns using that code in other ways, not necessarily in accord with
the GPL.

Some of our assignment contracts say that the author has to explicitly
activate the nonexclusive license.  Others say that the nonexclusive
license starts right away.  If the author prefers the latter form, we
always use it.

You seem to be bending over backwards to put us in the wrong.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[George Plymale II]

> All our software releaes give you this; they give this
> to the public at large.

>   > According to my understanding of the FSF's copyright policy, I have to
>   > notify the FSF when I want to distribute my changes because they're no
>   > longer mine; they're theirs.

Sorry, as I said to Clement Pit-Claudel, I should have (and intended to)
write this sentence in the past tense. I mistakenly wrote it in the
present tense. It was clarified to me by other messages on this thread
that this is not part of the FSF's policy. When I wrote the word
"hypocritical," that was not clarified to me.

> You don't have to notify the FSF to redistribute our software releases
> under the GPL.

> The unlimited nonexclusive license in the assignment contracts
> concerns using that code in other ways, not necessarily in accord with
> the GPL.

> Some of our assignment contracts say that the author has to explicitly
> activate the nonexclusive license.  Others say that the nonexclusive
> license starts right away.  If the author prefers the latter form, we
> always use it.

Right, this is what I was not aware of in my original message where I
used the word "hypocritical."

> You seem to be bending over backwards to put us in the wrong.

I am not trying to put anyone "in the wrong." If I were, this would be a
rather different (uglier) debate.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It was clarified to me by other messages on this thread
  > that this is not part of the FSF's policy. When I wrote the word
  > "hypocritical," that was not clarified to me.

I'm glad that is cleared up.  I won't hold criticism against you,
now that you no longer believe that.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Ask yourself, though:
  > what stops a user from modifying the bits that run on their computer?
  > Not the law. It is knowledge. Knowledge of how a computer works and how
  > well its machine code can be understood.

With all due respect, you're mistaken about the reasons for this.

Common obstacles include

  the user has no access to the source code.
  the system or the hardware is designed to refuse to run modified versions.

But the most common reason is,

  the user is not a programmer, and can't ask the development
  community for help because the proprietary nature of a program prevents
  the existence of one.

Changing a large program at the machine code level, without the
comments that help programmes understand the code, is too hard to be
feasible -- except for quite small changes, which are merely a great
pain in the neck.

  > Indeed, the GNU project's
  > efforts and funds would be far better spent creating tools that would
  > allow users to universally understand machine code that would truly
  > allow them to control any software that they have, regardless of its
  > underlying machinery.

That would be a superhuman AI.  If you succeed at this, I agree it
would be a great advance.  I'd rather spend our funds on areas where I
expect that effort can result in progress.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: A response to RMS (was Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I didn't even want to write all of this until you stipulated that my
  > opinions are based on misinformation and smear campaigns done by others
  > to the GNU project. Such a presumption insults the intellect of very
  > intelligent people who I know that share my opinion and who formed the
  > same opinion based on rational, moral, and objective analysis.

If they share your opinion and your reasoning,
they are mistaken for the same reasons.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Sam Steingold]

> * George Plymale II <trbetrqc@beovgnyvzcnpg.pbz> [2018-02-01 18:02:12 -0500]:
>
> Well, my understanding of the policy was that one's _contributions_
> are owned by the FSF. I.e., that you have to sign waivers which tell
> the FSF, "Hey, I give up all rights to own any code that I give you
> guys in these certain projects." To me, that seems hypocritical and it
> seemed that indeed the actual FSF policy versus what the FSF itself
> expounds were in disagreement. But, maybe I'm incorrect about that if
> that above passage is really what it sounds like.

This is a very common misunderstanding.

You _can_ distribute GNU software with your own changes under the GNU
GPL - this is called "forking", and has been done many times. The most
notable example is Lucid/X Emacs.

However, if you want the FSF to accept your changes into their source
code repository, you will have to assign them the copyright so that you
(or your heirs) will not be able to sabotage the FSF software
development by revoking your license to use your changes. This also
happened before (Gosling Emacs), and the FSF is now understandably
paranoid.

So, to repeat: you _can_ distribute your changes yourself.
If you want _the FSF_ to do that, you have to sign the papers.

HTH.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1561
http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il
http://mideasttruth.com http://honestreporting.com https://jihadwatch.org
Complete tolerance is impossible: it is insulting to bigots.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't know of an online resource with such details, but maybe it's
  > somewhere in the FSF website.

I think we should post an explanation of this.

It appears that we don't have one on gnu.org.  Would you like to check
fsf.org?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Stephen Berman]

On Fri, 02 Feb 2018 17:53:53 -0500 Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I don't know of an online resource with such details, but maybe it's
>   > somewhere in the FSF website.
>
> I think we should post an explanation of this.
>
> It appears that we don't have one on gnu.org.  Would you like to check
> fsf.org?

I probably cannot make a systematic check any time soon, but if I do
find the time, or happen to notice a relevant page by chance, I'll be
sure to point it out.

Steve Berman

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > I don't know of an online resource with such details, but maybe it's
  > >   > somewhere in the FSF website.
  > >
  > > I think we should post an explanation of this.
  > >
  > > It appears that we don't have one on gnu.org.  Would you like to check
  > > fsf.org?

Can someone look for one now on fsf.org?  Because the first step
to posting one is finding out if we already have one.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Stephen Berman]

On Sat, 03 Feb 2018 00:12:12 +0100 Stephen Berman <stephen.berman@gmx.net> wrote:

> On Fri, 02 Feb 2018 17:53:53 -0500 Richard Stallman <rms@gnu.org> wrote:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>>   > I don't know of an online resource with such details, but maybe it's
>>   > somewhere in the FSF website.
>>
>> I think we should post an explanation of this.
>>
>> It appears that we don't have one on gnu.org.  Would you like to check
>> fsf.org?
>
> I probably cannot make a systematic check any time soon, but if I do
> find the time, or happen to notice a relevant page by chance, I'll be
> sure to point it out.

I don't have the time to manually look through every page of the FSF
website but I did enter "assign" in the site's search form; it returned
55 hits, most of which are lists of new copyright assigners.  But the
following pages have some information about what copyright assignment
means:

https://www.fsf.org/licensing/index_html
 FSF Licensing & Compliance Team
 by Joshua Gay — published Mar 13, 2013 — last modified Aug 11, 2017 03:13 PM
 [...]
 Copyright & Compliance
 The Free Software Foundation holds the copyright to many GNU packages,
 such as GCC and EMACS. When hackers contribute to these projects, we
 ask that they assign their copyright to enable us to enforce the
 license. For questions about assigning to the FSF, please contact us at
 assign@gnu.org.

https://www.fsf.org/blogs/rms/assigning-copyright
 When a company asks for your copyright
 by Richard Stallman — published Sep 29, 2010 — last modified Nov 05, 2010 09:46 AM 
 [...]
 The company will probably invite you to assign or license your
 copyright to the company. That in itself is not inherently bad; for
 instance, many GNU software developers have assigned copyrights to the
 FSF. However, the FSF never sells exceptions, and its assignment
 contracts include a commitment to distribute the contributor's code
 only with source and only permitting redistribution.

https://www.fsf.org/licensing/assignunisrule.html
 Assignment University
 by jacobson — published Sep 19, 2006 — last modified Jul 29, 2013 02:55 PM 
 [...]
 Although the FSF does not require the assignment of copyright on new
 GNU project programs, on existing programs where it already holds
 copyright, the FSF does collect assignments and register copyrights.
 [...]
 some reasons why a university might like to assign copyright to the FSF:
 -> University assignment of code allows the university freedom from the
 burden of protecting the work created by the developers at the
 university. The FSF accepts copyright so that it can do its enforcement
 work.
 [...]

https://www.fsf.org/licensing/20050325novalis.html
 The Basics
 by novalis — published Mar 25, 2005 — last modified Mar 12, 2010 10:06 AM 
 [...]
 5. You can also assign your copyright. By way of analogy, licensing
 software to someone is like inviting them to visit your house;
 assigning copyright to someone is like selling them your house.
 a. You might want to assign your copyright to someone else if they
 promise to do the work of enforcing the license, or they otherwise
 won't take your patches.

Steve Berman

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for the search results.  It looks like we don't have a FAQ
about the topic of copyright assignments, so we will work on one,
over time.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Oops, I didn't see this message before sending my other message to you
which read:

> Yes, I would appreciate knowing about this as well.

Apologies!

- George Plymale II

Finding an online resource for the agreement (was Loading a package applies automatically to future sessions?)

[George Plymale II]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Fri, 02 Feb 2018 17:53:53 -0500 Richard Stallman <rms@gnu.org> wrote:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>>   > I don't know of an online resource with such details, but maybe it's
>>   > somewhere in the FSF website.
>>
>> I think we should post an explanation of this.
>>
>> It appears that we don't have one on gnu.org.  Would you like to check
>> fsf.org?
>
> I probably cannot make a systematic check any time soon, but if I do
> find the time, or happen to notice a relevant page by chance, I'll be
> sure to point it out.
>
> Steve Berman

Yes, I would appreciate knowing about this as well.

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That sounds interesting, but I would like to know more of the specifics
  > about that. Is there a place where I can find very detailed information
  > about this specific "license" which is "granted back to contributors?"

It is an unlimited nonexclusive license; it says you can use your own
code/text "as you see fit".  It may say you need to inform us the
first time you want to do this, and then you get the unlimited
nonexclusive license forever after.



-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > That sounds interesting, but I would like to know more of the specifics
>   > about that. Is there a place where I can find very detailed information
>   > about this specific "license" which is "granted back to contributors?"
>
> It is an unlimited nonexclusive license; it says you can use your own
> code/text "as you see fit".  It may say you need to inform us the
> first time you want to do this, and then you get the unlimited
> nonexclusive license forever after.

Well that sounds much better than I originally thought. Thank you for
clarifying that.

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 01 Feb 2018 21:14:53 -0500
> Cc: stephen.berman@gmx.net, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> It is an unlimited nonexclusive license; it says you can use your own
> code/text "as you see fit".  It may say you need to inform us the
> first time you want to do this, and then you get the unlimited
> nonexclusive license forever after.

In my assignments, both those from 20 years ago, and those from the
recent years, there's no such caveat as described in the last sentence
above.  I'm just given "nonexclusive, royalty-free, fully paid up and
non-cancellable worldwide rights to use [the code I developed], as
Developer sees fit."

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Date: Thu, 01 Feb 2018 21:14:53 -0500
>> Cc: stephen.berman@gmx.net, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>> 
>> It is an unlimited nonexclusive license; it says you can use your own
>> code/text "as you see fit".  It may say you need to inform us the
>> first time you want to do this, and then you get the unlimited
>> nonexclusive license forever after.
>
> In my assignments, both those from 20 years ago, and those from the
> recent years, there's no such caveat as described in the last sentence
> above.  I'm just given "nonexclusive, royalty-free, fully paid up and
> non-cancellable worldwide rights to use [the code I developed], as
> Developer sees fit."


Thanks for expounding on Richard's last point with your
observations. I'll probably have to look at one of these assignments
myself so that I can know exactly what it stipulates.

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> In my assignments, both those from 20 years ago, and those from the
> recent years, there's no such caveat as described in the last sentence
> above.

Can't remember what mine l;ooked like, but I've just found one (called
assign.ncurses) in fencepost which says:

       Upon thirty days' prior written notice(s), the Foundation agrees to
    grant to any one or more, or all, of us non-exclusive rights to use the
    Package as we see fit; (and the Foundation's rights shall otherwise
    continue unchanged).


-- Stefan

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It is an unlimited nonexclusive license; it says you can use your own
  > > code/text "as you see fit".  It may say you need to inform us the
  > > first time you want to do this, and then you get the unlimited
  > > nonexclusive license forever after.

  > In my assignments, both those from 20 years ago, and those from the
  > recent years, there's no such caveat as described in the last sentence
  > above.  I'm just given "nonexclusive, royalty-free, fully paid up and
  > non-cancellable worldwide rights to use [the code I developed], as
  > Developer sees fit."

I recall that SOME of our assignments used to say the developer
had to ask for the nonexclusive license, and would then automatically
receive it.  But it has been many years since I signed the assignments
myself, and I don't know whether we ever use that wording nowadays.

It doesn't make a difference in practice.  With the other wording that
make sit an option to request the nonexclusive license, the
contributor can immediately exercise the option.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The FSF staff checked.  We have not included that 30-day notice
provision in our copyright assignments since some time before 2000.
(They didn't check further back than that.)  The assignments since
then give the contributor unlimited nonexclusive rights starting
immediately.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> The FSF staff checked.  We have not included that 30-day notice
> provision in our copyright assignments since some time before 2000.
> (They didn't check further back than that.)  The assignments since
> then give the contributor unlimited nonexclusive rights starting
> immediately.

Ok, thank you very much for providing that information.

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: George Plymale II <georgedp@orbitalimpact.com>
> Date: Thu, 01 Feb 2018 16:44:04 -0500
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> >    [...] Thus, we grant back to contributors a license to use
> >    their work as they see fit. This means they are free to modify, share,
> >    and sublicense their own work under terms of their choice. This enables
> >    contributors to redistribute their work under another free software
> >    license. While this technically also permits distributing their work
> >    under a proprietary license, we hope they won't.
>    
> That sounds interesting, but I would like to know more of the specifics
> about that. Is there a place where I can find very detailed information
> about this specific "license" which is "granted back to contributors?"

When you sign the copyright assignment, the text of that assignment
includes the above-mentioned permissions.  If you want, I can cite the
text as it appears in my assignment agreements, I don't believe that
text is any kind of secret.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: George Plymale II <georgedp@orbitalimpact.com>
>> Date: Thu, 01 Feb 2018 16:44:04 -0500
>> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>> 
>> >    [...] Thus, we grant back to contributors a license to use
>> >    their work as they see fit. This means they are free to modify, share,
>> >    and sublicense their own work under terms of their choice. This enables
>> >    contributors to redistribute their work under another free software
>> >    license. While this technically also permits distributing their work
>> >    under a proprietary license, we hope they won't.
>>    
>> That sounds interesting, but I would like to know more of the specifics
>> about that. Is there a place where I can find very detailed information
>> about this specific "license" which is "granted back to contributors?"
>
> When you sign the copyright assignment, the text of that assignment
> includes the above-mentioned permissions.  If you want, I can cite the
> text as it appears in my assignment agreements, I don't believe that
> text is any kind of secret.

I would appreciate such a citation if you don't mind. You can also send
it to me privately, if you'd like.

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: George Plymale II <georgedp@orbitalimpact.com>
> Cc: stephen.berman@gmx.net, monnier@IRO.UMontreal.CA,
> 	emacs-devel@gnu.org
> Date: Fri, 02 Feb 2018 12:21:33 -0500
> 
> >> That sounds interesting, but I would like to know more of the specifics
> >> about that. Is there a place where I can find very detailed information
> >> about this specific "license" which is "granted back to contributors?"
> >
> > When you sign the copyright assignment, the text of that assignment
> > includes the above-mentioned permissions.  If you want, I can cite the
> > text as it appears in my assignment agreements, I don't believe that
> > text is any kind of secret.
> 
> I would appreciate such a citation if you don't mind. You can also send
> it to me privately, if you'd like.

I meanwhile already cited that.  Here it is again:

> [...]  I'm just given "nonexclusive, royalty-free, fully paid up and
> non-cancellable worldwide rights to use [the code I developed], as
> Developer sees fit."

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>    This kind of stuff doesn't encourage any openness

Our goal is freedom for all users of software -- much more than
"openness".

We release programs under a copyleft license such as the GNU GPL in
order to defend freedom for all users.  We ask for copyright
assignments so we can enforce the GPL better in court.  It's all part
of a careful plan for achieving freedom.

See https://gnu.org/licenses/copyleft.html for more explanation,
and https://gnu.org/licenses/why-assign.html.

Our goal is not "openness" -- we don't use that word.  We campaign for
users' freedom, specifically to control the software they use.

See https://gnu.org/philosophy/open-source-misses-the-point.html
for more explanation of the difference between free software and open
source.  See also https://thebaffler.com/salvos/the-meme-hustler for
Evgeny Morozov's article on the same point.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> Our goal is freedom for all users of software -- much more than
> "openness".
>
> We release programs under a copyleft license such as the GNU GPL in
> order to defend freedom for all users.  We ask for copyright
> assignments so we can enforce the GPL better in court.  It's all part
> of a careful plan for achieving freedom.
>
> See https://gnu.org/licenses/copyleft.html for more explanation,
> and https://gnu.org/licenses/why-assign.html.
>
> Our goal is not "openness" -- we don't use that word.  We campaign for
> users' freedom, specifically to control the software they use.
>
> See https://gnu.org/philosophy/open-source-misses-the-point.html
> for more explanation of the difference between free software and open
> source.  See also https://thebaffler.com/salvos/the-meme-hustler for
> Evgeny Morozov's article on the same point.

By using the word "openness," I wasn't specifically referring to the
open-source movement. I just meant that this policy decreases general
openness in the community since there is a certain gate, if you will,
which one must pass through in order to contribute to Emacs.

The other thing which I meant regarding "openness" is succinctly
expressed by Nikos Mavrogiannopoulos, one of the developers of GnuTLS:

"(b) The feeling of participation in the GNU project is very low, as even
expressing a different opinion in the internal mailing lists is hard if
not impossible.
(c) There is no process for decision making or transparency in GNU.
The only existing process I saw is "Stallman said so"
(this may not be bad, unless some threshold of disagreement has been
reached - but then it is fair to be able to disassociate myself from the
project)."

( from https://lwn.net/Articles/529565/ )

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > By using the word "openness," I wasn't specifically referring to the
  > open-source movement. I just meant that this policy decreases general
  > openness in the community since there is a certain gate, if you will,
  > which one must pass through in order to contribute to Emacs.

Freedom sometimes requires a sacrifice, and this is one example.
"Openness" is not a priority for us -- freedom is our goal.

  > (c) There is no process for decision making or transparency in GNU.

Do you mean, that the contributors can't change the goals and the
philosophy of the project.  That's true, and it has to be that way.

Most free software projects don't have a commitment to any particular
political goal or philosophy.  Those questions are simply up to the
main developers of the project.  They can change goals as thery see
fit.

GNU is different.  It has a commitment to a particular political
philosophy, and that never changes.

People who expect the other way may be surprised when they see GNU is
different.  They expect to be able to argue about the basic goal
and get it changed.  However, the goal of GNU came before GNU,
so they cannot change it.

You have a right to criticize various aspects of GNU, but please don't
use this list for that; that is off-topic here.  This list is for
developing Emacs.  Explaining our assignment policies was just barely
on-topic, but this is not.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Another response to RMS (was Loading a package applies automatically to future sessions?)

[George Plymale II]

> Freedom sometimes requires a sacrifice, and this is one example.
> "Openness" is not a priority for us -- freedom is our goal.

Ok. I am of the opinion that these virtues go hand in hand.

> Most free software projects don't have a commitment to any particular
> political goal or philosophy.  Those questions are simply up to the
> main developers of the project.  They can change goals as thery see
> fit.

> GNU is different.  It has a commitment to a particular political
> philosophy, and that never changes.

> People who expect the other way may be surprised when they see GNU is
> different.  They expect to be able to argue about the basic goal
> and get it changed.  However, the goal of GNU came before GNU,
> so they cannot change it.

Ok, I understand and respect that.

> You have a right to criticize various aspects of GNU, but please don't
> use this list for that; that is off-topic here.  This list is for
> developing Emacs.  Explaining our assignment policies was just barely
> on-topic, but this is not.

I apologize, I only brought this up as a genuine concern for improving
some code which was relevant. My subsequent messages about this were
responses to others who in turn messaged me about this issue (including
you). If you want, I will stop discussing all such issues regarding this
on this list from henceforth.

If that is the case, where may relevantly I take such concerns should
they arise again?

Re: Another response to RMS (was Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If that is the case, where may relevantly I take such concerns should
  > they arise again?

We have a mailing list for the purpose, gnu-misc-discuss@gnu.org.
Nowadays it is not used much.  I gather that people mostly discuss
their agreement or disagreement with the free software movement in
various large forum sites which I don't use at all.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Another response to RMS (was Loading a package applies automatically to future sessions?)

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > If that is the case, where may relevantly I take such concerns should
>   > they arise again?
>
> We have a mailing list for the purpose, gnu-misc-discuss@gnu.org.
> Nowadays it is not used much.  I gather that people mostly discuss
> their agreement or disagreement with the free software movement in
> various large forum sites which I don't use at all.

Thanks for the info. I apologize again for cluttering up this mailing
list with this sort of discussion.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I also find it fruitless/useless/annoying,

When you say it is "annoying", that describes your feelings.  I can't
argue with your feelings, but to get annoyed by a little work that
one needs to do at most twice in a lifetime seems to me excessive.

On the other hand, saying it is "fruitless" and "useless" is simply
incorrect.  It helps us enforce the GNU GPL.

  > Both positions are only "a matter of principle"

Our policy of requiring copyright assignments is entirely a practical
matter -- to help us enforce the GPL.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Also, I am legally under-age to sign any such copyright waivers. So I
  > couldn't do it even if I wanted to.  Another reason that this is a
  > fruitless policy.

This policy gives us (1) confidence we have the right to put the code
into Emacs, and (2) a firm basis for enforcing the GPL.
It is sometimes inconvenient, but it is hardly fruitless.
On the contrary, it is very important.

If you are too young to sign, you could ask your parents to sign on
your behalf.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> This policy gives us (1) confidence we have the right to put the code
> into Emacs,

Is not a statement of my permission in an email enough? If an issue came
up in a court of law, couldn't you just subpoena the email message where
I (or whoever else) granted you permission to have my code? At least,
that would take away the hassle factor of paperwork.

Heck, in your FAQ about the GPL, you say:

"We [keep the copyright status of the program as simple as possible] by
asking each contributor to either assign the copyright on contributions
to the FSF, or disclaim copyright on contributions."

https://www.gnu.org/licenses/gpl-faq.en.html#AssignCopyright

As I told Stefan in my initial message about this, I would be fine with
releasing my contributions into the public domain. I think that makes a
lot of sense; I'm giving the code away so anyone can use it however they
like.

Of course, all of this would probably not be necessary were it not for
the controversial nature of the GPL which leads to lawsuits a lot more
than other open-source licenses.

> (2) a firm basis for enforcing the GPL.
> It is sometimes inconvenient, but it is hardly fruitless.
> On the contrary, it is very important.

All that you can do to enforce the GPL is sue people. Isn't there enough
litigiousness in the world today? Moreover, there is no other fruit
borne by these restrictions than that won in a court battle. You
certainly aren't recruiting more developers in any projects which are
subject to this policy. That is worrisome when you have core developers,
such as Eli, saying that the Emacs core is falling by the wayside
because there are too few newcomers who will delve into the guts of the
project. And there are few newcomers at all.

> If you are too young to sign, you could ask your parents to sign on
> your behalf.

What if they can't? What if I don't have parents? What if I live in
foster homes and my legal guardian is frequently changed? I don't see
how a policy which prevents such people from contributing to the project
is very helpful.

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Tim Landscheidt]

George Plymale II <georgedp@orbitalimpact.com> wrote:

> […]

>> If you are too young to sign, you could ask your parents to sign on
>> your behalf.

> What if they can't? What if I don't have parents? What if I live in
> foster homes and my legal guardian is frequently changed? I don't see
> how a policy which prevents such people from contributing to the project
> is very helpful.

I doubt that under those conditions someone could legally
release one's code under any licence or into the public do-
main; i. e. "a policy" means "every policy" in that context.

Tim

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > What if they can't? What if I don't have parents? What if I live in
  > > foster homes and my legal guardian is frequently changed? I don't see
  > > how a policy which prevents such people from contributing to the project
  > > is very helpful.

  > I doubt that under those conditions someone could legally
  > release one's code under any licence or into the public do-
  > main; i. e. "a policy" means "every policy" in that context.

I think you are right.  (Though I hesitate to say it is _impossible_;
there may be a legal avenue, but we'd need to ask a lawyer what it may
be.)

Our policy does not create the problem, but rather helps us avoid
being oblivious to it.)

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As I told Stefan in my initial message about this, I would be fine with
  > releasing my contributions into the public domain.

That would be sufficient for us to use the code.

We want to have something we can point at, to show a court that you
have did put it in the public domain.  You can do that by signing a
copyright disclaimer.  Please write to assign@gnu,org saying you'd
like to disclaim copyright on a contribution to Emacs, and they will
say how.

I think you'd still need your parent or guardian to sign it,
but it wouldn't assign the copyright.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > As I told Stefan in my initial message about this, I would be fine with
>   > releasing my contributions into the public domain.
>
> That would be sufficient for us to use the code.
>
> We want to have something we can point at, to show a court that you
> have did put it in the public domain.  You can do that by signing a
> copyright disclaimer.  Please write to assign@gnu,org saying you'd
> like to disclaim copyright on a contribution to Emacs, and they will
> say how.
>
> I think you'd still need your parent or guardian to sign it,
> but it wouldn't assign the copyright.

That sounds like it is sufficient (or at least an improvement on what I
originally thought).

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 01 Feb 2018 21:17:15 -0500
> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>   > As I told Stefan in my initial message about this, I would be fine with
>   > releasing my contributions into the public domain.
> 
> That would be sufficient for us to use the code.

However, AFAIU and AFAIR, this is impossible to do once and for all
the future contributions, as we do with copyright assignments.  Which
means that every single contribution will need a separate disclaimer.
People who choose that way should be aware of that caveat (assuming
I'm not confused about that, in which case apologies in advance).

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Date: Thu, 01 Feb 2018 21:17:15 -0500
>> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>> 
>>   > As I told Stefan in my initial message about this, I would be fine with
>>   > releasing my contributions into the public domain.
>> 
>> That would be sufficient for us to use the code.
>
> However, AFAIU and AFAIR, this is impossible to do once and for all
> the future contributions, as we do with copyright assignments.  Which
> means that every single contribution will need a separate disclaimer.
> People who choose that way should be aware of that caveat (assuming
> I'm not confused about that, in which case apologies in advance).


Ok. Thanks for clarifying that.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We don't have a way to do a disclaimer for future changes.  We have
not had many contributors who wanted to disclaim instead of assigning,
so we haven't tried.  But it might be possible.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> We don't have a way to do a disclaimer for future changes.  We have
> not had many contributors who wanted to disclaim instead of assigning,
> so we haven't tried.  But it might be possible.

That sounds like a great idea. It's definitely something to be looked into.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We don't have a way to do a disclaimer for future changes.  We have
  > > not had many contributors who wanted to disclaim instead of assigning,
  > > so we haven't tried.  But it might be possible.

  > That sounds like a great idea. It's definitely something to be looked into.

I will ask a lawyer whether in principle this should be possible.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > We don't have a way to do a disclaimer for future changes.  We have
>   > > not had many contributors who wanted to disclaim instead of assigning,
>   > > so we haven't tried.  But it might be possible.
>
>   > That sounds like a great idea. It's definitely something to be looked into.
>
> I will ask a lawyer whether in principle this should be possible.

Thanks. Please let me know if it is.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What if they can't? What if I don't have parents? What if I live in
  > foster homes and my legal guardian is frequently changed?

If that happens, I will worry about what to do.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > What if they can't? What if I don't have parents? What if I live in
>   > foster homes and my legal guardian is frequently changed?
>
> If that happens, I will worry about what to do.

Well, I don't think I could convince any of my parents to sign copyright
waivers. And I don't want to subject them to such concerns. Moreover, it
is possible that there is further ambiguity because of things that I
cannot divulge publicly.

In any case, this policy is certainly a pain for minors.

RE: Loading a package applies automatically to future sessions?

[Drew Adams]

> Well, I don't think I could convince any of my parents to sign copyright
> waivers. And I don't want to subject them to such concerns. Moreover, it
> is possible that there is further ambiguity because of things that I
> cannot divulge publicly.
> 
> In any case, this policy is certainly a pain for minors.

It's parents et al who are a pain for minors. ;-)
Only one (partial) cure for that.

Generations (was: Loading a package applies automatically to future sessions?)

[Stefan Monnier]

> It's parents et al who are a pain for minors. ;-)

I think you have this backward!


        Stefan

RE: Generations (was: Loading a package applies automatically to future sessions?)

[Drew Adams]

> > It's parents et al who are a pain for minors. ;-)
> 
> I think you have this backward!

Both are true.  Mine was a reply to a minor complaint. ;-)
Yours is a major complaint.

Re: Generations (was: Loading a package applies automatically to future sessions?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It's parents et al who are a pain for minors. ;-)

  > I think you have this backward!

However, parents do have the ability to avoid having children.
Children don't have a way to avoid having parents.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

RE: Generations (was: Loading a package applies automatically to future sessions?)

[Drew Adams]

>   > > It's parents et al who are a pain for minors. ;-)
> 
>   > I think you have this backward!
> 
> However, parents do have the ability to avoid having
> children.  Children don't have a way to avoid having
> parents.

Interesting research project, no doubt.

Re: Loading a package applies automatically to future sessions?

[Phillip Lord]

George Plymale II <georgedp@orbitalimpact.com> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>>   > What if they can't? What if I don't have parents? What if I live in
>>   > foster homes and my legal guardian is frequently changed?
>>
>> If that happens, I will worry about what to do.
>
> Well, I don't think I could convince any of my parents to sign copyright
> waivers. And I don't want to subject them to such concerns. Moreover, it
> is possible that there is further ambiguity because of things that I
> cannot divulge publicly.
>
> In any case, this policy is certainly a pain for minors.

Indeed. But, without the copyright policy, you should still be asking
your parents about your ability to licence your software under
GPL. Because who ever uses your software is, effectively, entering into
a contract with you, that gives them the right to use your softawre

Now the contractual terms of the GPL state that is irrevocable. I don't
know how it is in the US, but in the UK, my understanding anyway, is
that while as a minor you can make a contract, you can revoke that
contract before you reach 18 or shortly after. Even if that
disadvantages the other party.

So, I don't think the assignment policy is the issue here. The law is
with respect to minors is difficult; it's not meant to be a pain, it's
meant to be a protection. Either way, the complexities of being a minor
do apply to an assignment policy, but they also apply to licencing.

Usual "I am not a lawyer" restrictions apply to anything I say here.

Phil

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I'm responding to the points you raised so as to educate you and
others about what we do and where we stand.  You don't have to agree,
but I want incorrect statements about us to be corrected.

  > All that you can do to enforce the GPL is sue people.

On the contrary, most of our enforcement actions don't go as far as a
court case.  But we need to have that option as a last resort
in order to win most of the time without suing.

							  Isn't there enough
  > litigiousness in the world today?

Yes, but that doesn't imply that any given lawsuit is wrong.

For instance, look at the ACLU's lawsuits.  The ACLU is constantly
suing to overturn unjust laws and protect human rights.  I support the
ACLU because I support those lawsuits.

The FSF's GPL enforcement is also a way of protecting human rights:
specifically, every user's right to change and redistribute certain
code.

				      Moreover, there is no other fruit
  > borne by these restrictions

No, it's the other way around.  We use the GNU GPL to STOP those who
redistribute our code from placing restrictions on subsequent users of
it.

				than that won in a court battle.

Usually we succeed without a court battle.  However, if do go to court
and win, that's still good -- it protects the users' freedom.

  > You
  > certainly aren't recruiting more developers in any projects which are
  > subject to this policy.

Yes we are.  There are several Emacs contributors that weren't
involved a few years ago.

However, Emacs would fail to give people freedom if we gave up on
defending it.

  > Of course, all of this would probably not be necessary were it not for
  > the controversial nature of the GPL which leads to lawsuits a lot more
  > than 

Our defense of freedom is controversial, but we don't mind.  The GPL
sometimes leads to lawsuits because it defends users' freedom.
If we gave up without a fight, we'd never have a fight, but that would
not be better.

	 other open-source licenses.

This is the free software movement.  We believe that nonfree software
is an injustice because it denies users freedom.  We don't just wish
everyone had software freedom; we work and campaign and when necessary
fight for it.

Open source is a different idea; the term was coined specifically to
reject our views.  They disagree with us, and we disagree with them.

See https://gnu.org/philosophy/open-source-misses-the-point.html
for more explanation of the difference between free software and open
source.  See also https://thebaffler.com/salvos/the-meme-hustler for
Evgeny Morozov's article on the same point.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> I'm responding to the points you raised so as to educate you and
> others about what we do and where we stand.  You don't have to agree,
> but I want incorrect statements about us to be corrected.

I don't agree on some of your points and I would like to respond to some
of those points to clarify to you where I stand and others who share
similar opinions. I don't want to respond to all of them since I do not
desire a flame war. But I will respond to some which I feel will not
incite ill feelings.

> Yes, but that doesn't imply that any given lawsuit is wrong.

You are correct. Yet I find that most lawsuits regarding software are
frivolous and ultimately a waste of time for everyone involved
(including society at large). There are much more serious crimes and
problems in the world today and wasting a court's time on stuff
involving electrical pulses in metal boxes just seems totally silly, at
best. No matter if it's about copyright or copyleft.

> No, it's the other way around.  We use the GNU GPL to STOP those who
> redistribute our code from placing restrictions on subsequent users of
> it.

I understand the enforcing nature of the GPL. What I meant is that there
is no fruit for the project in terms of new manpower or technical
improvement.

Honestly, though, the GPL is just a different set of restrictions. And,
contrary to popular belief, it restricts users as well as
developers. Users are restricted from sharing or modifying this program,
except under certain conditions, by the barrier of copyright
issues. These restrictions are supposed to prevent other restrictions,
but honestly you could say the same thing about some proprietary
software licenses.

> Yes we are.  There are several Emacs contributors that weren't
> involved a few years ago.

I would like to see Emacs get more recruitment which is more appreciable
in terms of numbers, though. The excitement around Emacs is surprisingly
growing. I believe it is in large part due to Spacemacs (
http://spacemacs.org/ ) and similar projects which appeal to those who
are used to modern text editors. I also believe that Emacs would see
significantly more contributors improving the project if they weren't
scared off by murmurs of copyright issues and philosophical
disagreements.

> However, Emacs would fail to give people freedom if we gave up on
> defending it.

Emacs has suffered from this defense, though. lldb still hasn't gotten
proper support in Emacs because of Apple's mere association with its
parent project. Eli went on their mailing list some months ago and tried
to resolve some technical disagreements with the project, but lldb
probably would have been supported in Emacs a long time ago were it not
for the political brouhaha surrounding it.

People have tried to make improvements to GCC to allow the creation of
development modes in Emacs which are now supported by Clang
instead. E.g. https://github.com/Sarcasm/irony-mode

Those improvements to GCC were denied also in the name of defending freedom.

Sometimes it seems that in the free software world, you have to fly the
GNU banner or else you're an enemy.

> Open source is a different idea; the term was coined specifically to
> reject our views.  They disagree with us, and we disagree with them.

This division is unproductive at best.

> See https://gnu.org/philosophy/open-source-misses-the-point.html
> for more explanation of the difference between free software and open
> source.  See also https://thebaffler.com/salvos/the-meme-hustler for
> Evgeny Morozov's article on the same point.

Thank you, I'm aware of the philosophy of the GNU project.

In any case, the copyright issues which I was worried about do not seem
as bad as they did to me initially. It seems that there is more
flexibility than I thought so that is good. I just hope that perhaps we
can get to a point someday where Emacs (and other GNU programs) will be
able to exist without the burdens of copyright worries. I know that
those worries seem helpful to some, but it's hard to believe that when
looking at things as a whole.

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Paul Eggert]

On 02/01/2018 11:24 PM, George Plymale II wrote:
> most lawsuits regarding software are
> frivolous and ultimately a waste of time for everyone involved

Having been the target of a frivolous software lawsuit myself 
<https://www.eff.org/press/releases/eff-wins-protection-time-zone-database> 
I am sympathetic to this argument. However, such lawsuits don't mean we 
should give up on the legal system entirely; that would be throwing out 
the baby with the bathwater. Instead, we should use the legal system as 
best we can, and work to improve it where it has significant flaws. And 
this is what the Free Software Foundation is trying to do.

> In any case, the copyright issues which I was worried about do not seem
> as bad as they did to me initially.

Yes, that's often the case. A lot of FUD has been spread about the GPL 
and the GNU project.

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> Having been the target of a frivolous software lawsuit myself 
> <https://www.eff.org/press/releases/eff-wins-protection-time-zone-database> 
> I am sympathetic to this argument. However, such lawsuits don't mean we 
> should give up on the legal system entirely; that would be throwing out 
> the baby with the bathwater. Instead, we should use the legal system as 
> best we can, and work to improve it where it has significant flaws.

Totally in agreement with you. I just dislike myself (or others) having
to worry about these kinds of things at all. But I guess that's just the
world we live in today.

> Yes, that's often the case. A lot of FUD has been spread about the GPL 
> and the GNU project.

Well, I actually distilled this opinion from information that I got from
the FSF's website and gnu.org. So I doubt I was reading any GPL FUD from
them ;)

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Well, I actually distilled this opinion from information that I got from
  > the FSF's website and gnu.org. So I doubt I was reading any GPL FUD from
  > them ;)

I would guess that you reacted to what you read there based on ideas
you have heard others express.  "Don't you dare restrict me from
putting restrictions on others" is an idea that I've seen for 20
years, from those opposed to copyleft.  I wrote this haiku back then
to show why it is misguided.

   Using GPL
   Is encroaching on our rights
   To encroach on yours


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Honestly, though, the GPL is just a different set of restrictions. And,
  > contrary to popular belief, it restricts users as well as
  > developers. Users are restricted from sharing or modifying this program,
  > except under certain conditions, by the barrier of copyright
  > issues.

It's just the opposite: the GNU GPL _prevents_ restrictions.
It assures that all users can freely redistribute the program,
by stopping middlemen from putting on restrictions.

You look at freedom as a word game, so you say, "Hay, if I can't put
on restrictions, that's a restriction on me."  That paradox doesn't
bother us because we understand freedom as a matter of substance, not
a matter of form.  We judge by the substantive question: are users
free to do the things that give them control over the program?

You don't have to agree with us, but arguing against our philosophy is
outside the purpose of this list.  This list is for Emacs development.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Loading a package applies automatically to future sessions?

[Clément Pit-Claudel]

On 2018-02-01 16:36, George Plymale II wrote:
> What if they can't? What if I don't have parents? What if I live in
> foster homes and my legal guardian is frequently changed?

Note that (depending on where you live) you most likely need a parent's or guardian's approval to release your code into the public domain (or under any license that grants rights to others).  The GPL and Emacs' copyright policy are not different in that regard.

Please remember to change the title of the thread when you change the topic, too: it makes it easier for others to filter discussions by topic :)

Clément.

RE: Loading a package applies automatically to future sessions?

[Drew Adams]

> Please remember to change the title of the thread when you change the
> topic, too: it makes it easier for others to filter discussions by topic
> :)

+1

Re: Loading a package applies automatically to future sessions?

[Herring, Davis]

> Of course, all of this would probably not be necessary were it not for
> the controversial nature of the GPL which leads to lawsuits a lot more
> than other open-source licenses.

While we're about clarifying these issues: the reason that fewer lawsuits are pursued over software subject to permissive licenses is of course their lack of provisions which anyone would expect to profit by violating.  (Innocent mistakes are made, of course, but they are not litigated.)  The controversy over the GPL of course also results from its provisions, but that makes it a sister effect to the lawsuits, not a cause of them.

Davis

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> It looks like your code is fetching slime-autoloads.elc:
> (let ((load-file-name "/Users/my_username/.emacs.d/elpa/slime-20180126.1033/slime-autoloads.elc"))
> ;; ... bunches of bytecode
> )

Hmm... good catch.  I just added a corresponding FIXME in the code
(hopefully M-x dwim will figure out how to fix it).

>> Also, I'm not 100% surprised that byte-compiling byte-compiled code would
>> fail, but I can't see any immediate reason why it should fail, so it's
>> quite possible to it'd be easy to make it work.
> Not sure why it would fail either, but this is the error that I see for
> that specific chunk of bytecode:
> package-fastpath.el:899:1:Error: Wrong type argument: sequencep, 1004

I'll see if I can reproduce it (it's probably better to try and use the
.el file instead, tho).

> And lo and behold, `emacs-init-time' yields 0.7 seconds. Which is just
> 0.2 seconds higher than what I get with -Q, which I believe is Emacs'
> peak startup time. I also verified that I have access to all my packages
> so it is using package-fastpath.el.

Good, thanks.  Now the numbers correspond better to my expectations.

> So that means your patch has actually increased startup time a lot more
                                        ^^^^^^^^^
                                        improved

> Well, actually those figures are better than what you're saying as per
> my information above. I guess that makes your patch even more
> respectable for a first stab! :)

FWIW "a first stab" often ends up being faster than the end product,
because the first stab was too optimistic.

>> Hmm... I thought this can't happen because the files are concatenated in
>> the same order that they are normally loaded by `package-initialize`.
>> I guess something doesn't work the way I thought it does.
>> Can you investigate to see when or even why it happens?
> It specifically happened with gh.el (https://github.com/sigma/gh.el),
> which is dependent on marshal.el (https://github.com/sigma/marshal.el)
>
> Apparently gh-autoloads.el uses a `gh-defclass' macro, which in turn
> uses a `marshal-defclass' macro. There is definitely some strangeness
> that could be going on here.

Could you try and give some more details about this problem?
E.g. check the fastpath file to see in which order the two packages's
autoloads are placed.  If gh-autoloads.el comes before
marshal-autoloads.el, could you try and figure out why (e.g. maybe
starting with `M-x trace-function RET package-activate RET`
and `M-x trace-function RET package-activate-1 RET`)?


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> Hmm... good catch.  I just added a corresponding FIXME in the code
> (hopefully M-x dwim will figure out how to fix it).

Sounds good.

> I'll see if I can reproduce it (it's probably better to try and use the
> .el file instead, tho).

Ok, let me know if you need help from my end.

> FWIW "a first stab" often ends up being faster than the end product,
> because the first stab was too optimistic.

Hopefully it won't come to that ;)

> Could you try and give some more details about this problem?
> E.g. check the fastpath file to see in which order the two packages's
> autoloads are placed.  If gh-autoloads.el comes before
> marshal-autoloads.el, could you try and figure out why (e.g. maybe
> starting with `M-x trace-function RET package-activate RET`
> and `M-x trace-function RET package-activate-1 RET`)?

gh-autoloads.el does indeed come before marshal-autoloads.el (although I
currently have them switched around to prevent init errors).

I did those `trace-function' invocations that you suggested, but I can't
quite make heads or tails out of them so I will just send you the
*trace-output* dumps to look at it yourself. I will send said dumps in a
private email since I don't want to post all of my installed packages
publicly.

Thanks,

- George Plymale II

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

One more thing regarding my bug about info nodes from ELPA,

I noticed that all my info nodes were back after upgrading my packages
in `package-list-packages'. Do you have any idea about why that is,
Stefan?

Thanks,
- George Plymale II

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> One more thing regarding my bug about info nodes from ELPA,

Oh, I didn't respond to that part, but this is a "known issue" (the
patch I sent doesn't bother to keep track and reproduce changes to
Info-directory-list).

> I noticed that all my info nodes were back after upgrading my packages
> in `package-list-packages'.  Do you have any idea about why that is,
> Stefan?

Hmm... no I can't explain how/why the problem disappeared when you did that.


        Stefan

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> One more thing regarding my bug about info nodes from ELPA,
>
> Oh, I didn't respond to that part, but this is a "known issue" (the
> patch I sent doesn't bother to keep track and reproduce changes to
> Info-directory-list).
>
>> I noticed that all my info nodes were back after upgrading my packages
>> in `package-list-packages'.  Do you have any idea about why that is,
>> Stefan?
>
> Hmm... no I can't explain how/why the problem disappeared when you did that.
>
>
>         Stefan

Ok, please consider fixing this even though it's a known issue.

Re: Loading a package applies automatically to future sessions?

[T.V Raman]

George Plymale II <georgedp@orbitalimpact.com> writes:

try let-binding file-name-handler-alist to nil -- in my case, that gave
a  significant speed-up  a few months ago > Hi all,
>
> I'm the one who posted the question on Stack Exchange, which Stefan
> mentioned the other day. My specific question is detailed here:
> https://emacs.stackexchange.com/q/38368/10761
>
> So, I've tried out Stefan's patch and it's definitely an improvement on
> startup time, but the improvement is not huge for me. Specifically, I've
> done the following in my startup files:
>
> (setq package-enable-at-startup nil)
> (load "~/.emacs.d/package-fastpath.el")
> (setq package--initialized t)
>
> I put that code in my files after running Stefan's
> `package-fastpath-refresh'. I also set `load-source-file-function' to
> nil as per Stefan's suggestion. Unfortunately, I couldn't byte-compile
> my package-fastpath.el file due to some packages which already
> byte-compile their autoload files, such as SLIME.
>
> Now, I do see some improvement in my startup time, but to reiterate, it
> is not huge. When I run Emacs with my startup commented out,
> `emacs-init-time' yields 1.2 seconds. When I run it with my startup
> intact, it yields 1.3 or 1.4 seconds. This contrasts with my previous
> `emacs-init-time' which was 1.6 or 1.7 seconds, but as you can see, it's
> not a really big difference.
>
> It is possible that this sub-par time is due to something on my own
> system, but I'm not sure nor convinced of that. Moreover, Stefan's patch
> certainly can use some improvements and some notes on how to use his
> changes, as it is a bit vexing to figure it out by reading the diff ;)
>
> I did notice a few bugs with Stefan's changes. One bug was that
> `package-installed-p' no longer yields correct results on installed
> packages. I believe that this is because `package-desc-p' is also broken
> by these changes. This broke some code in my startup which I used to
> check whether I need to install new packages.
>
> Another bug was that some packages were placed in bad order in
> package-fastpath.el. In other words, if a dependency's autoloads are
> written to this file after its dependent package, the dependent package
> will err, saying that it couldn't require one of its dependencies. There
> is obviously some work to do on making sure that dependencies are placed
> in the correct order or to change the code so that package-fastpath.el
> is order-agnostic.
>
> As a side note and a bit of an opinion, Radon Rosborough made an
> interesting remark in one of his messages. He mentioned pip and how
> things are done in Python, which really struck me. You never really
> think about using a package in a language like Python or Ruby. You just
> `require' or `import' it and that's that. It's really simple and the
> amount of packages that you have never hurts the startup of the main
> program. I know that Emacs is a bit more complicated since it's more of
> a text editor than a language and we have somewhat more intricacies to
> worry about. But I think that this kind of a model is the kind we need
> to be headed for.
>
> A user should be able to have a million packages and not have to worry
> about the subsequent startup time. Heck, in my own Ruby installation I
> have 436 gems and I've never even thought about startup time till now.
>
> In any case, I think Stefan's ideas and proposed changes are a good idea
> and I am in much the same boat as John Wiegley in that I restart Emacs
> often enough that startup time gets on my nerves. So I hope that we'll
> have some real progress on this issue and make package.el more robust.
>
> Thanks,
> - George Plymale II
>

-- 

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> try let-binding file-name-handler-alist to nil -- in my case, that gave
> a  significant speed-up  a few months ago

I have done that. It did help my startup time as well, but I still have
the issues which I've described.

Re: Loading a package applies automatically to future sessions?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

Can I ask what would be considered an acceptable startup time? Times under
2 seconds seem pretty good to me. I'm just curious to know what the general
expectation is?

Tim



On 31 January 2018 at 16:18, George Plymale II <georgedp@orbitalimpact.com>
wrote:

> > try let-binding file-name-handler-alist to nil -- in my case, that gave
> > a  significant speed-up  a few months ago
>
> I have done that. It did help my startup time as well, but I still have
> the issues which I've described.
>
>


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1140 bytes --]

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> Can I ask what would be considered an acceptable startup time? Times
> under 2 seconds seem pretty good to me. I'm just curious to know what
> the general expectation is?

I'm hoping to get my startup time at least under 1 second, say 0.7 or
0.8 seconds. I used to have it under half a second, but that doesn't
seem possible on the Emacs Mac Port given that `emacs -Q' is already 0.5
seconds. Of course, I would be most happy if I could somehow make that
happen.

Re: Loading a package applies automatically to future sessions?

[John Wiegley]

>>>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:

TC> Can I ask what would be considered an acceptable startup time? Times under
TC> 2 seconds seem pretty good to me. I'm just curious to know what the
TC> general expectation is?

My ideal is .25 (I've gotten it as low as 0.14 on GNU/Linux).  I don't
complain if it's below 2.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Loading a package applies automatically to future sessions?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

I dunno, perhaps I'm just slow, but I know my emacs startup is certainly
more than 2 seconds and the time has never bothered me. As I rarely hack on
elisp these days, I don't find it necessary to restart very often and I've
got everything integrated using emacsclient, so once I'm running, I forget
about startup time. However, I will put some timing code in just to see
(out of interest) what my startup is like.

For me, I'm more frustrated by modes like helm, which seem to offer
benefits, but I find such add-ons tend to slow down my use of emacs, which
I find more frustrating than startup times.

The slowest component in my setup is definitely me though!

Tim


On 31 January 2018 at 19:05, John Wiegley <johnw@gnu.org> wrote:

> >>>>> "TC" == Tim Cross <theophilusx@gmail.com> writes:
>
> TC> Can I ask what would be considered an acceptable startup time? Times
> under
> TC> 2 seconds seem pretty good to me. I'm just curious to know what the
> TC> general expectation is?
>
> My ideal is .25 (I've gotten it as low as 0.14 on GNU/Linux).  I don't
> complain if it's below 2.
>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
>



-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 2150 bytes --]

Re: Loading a package applies automatically to future sessions?

[Stefan Monnier]

> I dunno, perhaps I'm just slow, but I know my emacs startup is certainly
> more than 2 seconds and the time has never bothered me.

It very much depends on your usage pattern.  My Emacs's startup time
doesn't bother me either, but I understand that it can be annoying for
other users.


        Stefan

Re: Loading a package applies automatically to future sessions?

[T.V Raman]

M-x emacs-init-time will show your startup time.

You can find some code for more fine-grained instrumentation in
tvr/emacs-startup.el in the emacspeak Github repo.
-- 

Re: Loading a package applies automatically to future sessions?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

Thanks Raman, I was actually looking at that this morning.

The danger for me is that every time I look at my emacs init, I always find
things I can improve or cleanup. Not a complaint, but it is often a source
of procrastination. There are way too many weekends I've wasted tweaking my
emacs setup rather than getting on with my always large todo list!

Tim


On 2 February 2018 at 03:23, T.V Raman <raman@google.com> wrote:

> M-x emacs-init-time will show your startup time.
>
> You can find some code for more fine-grained instrumentation in
> tvr/emacs-startup.el in the emacspeak Github repo.
> --
>



-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1271 bytes --]

Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> The danger for me is that every time I look at my emacs init, I always find things I can improve or cleanup. Not a complaint, but it is often a source of procrastination.
> There are way too many weekends I've wasted tweaking my emacs setup rather than getting on with my always large todo list!

Ah, I know the feeling all too well. Hence my initial involvement in
this thread :)

Re: Loading a package applies automatically to future sessions?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> I think there should be a way to activate a package explicitly for the
  > >> current session _without_ automatically activating it for future
  > >> sessions.
  > > The failure of package.el to support this use case

  > FWIW, package.el does support this case:
  > - one way is to prevent package-initialize from activating packages,
  >   and then activating the ones you want by explicit calls to package-active.
  > - another is to set package-load-list.  E.g.

This means a user _can_ do it -- but it isn't naturak and easy.
I suggest adding a convenient interface for it.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> Since it has been more than a month with no response, I am re-posting
> the patch which fixes the problems previously discussed [1] [2] [3]
> with `package-initialize' by adding a second (optional) init-file
> `early-init.el'. This patch includes some refactoring of the code in
> `startup.el' to provide for loading any number of init-files in an
> extensible way. I hope that the change can be included in Emacs 26.2
> or Emacs 27.

Sorry for the delay, here are my comments (I doubt it's appropriate for
Emacs-26.2 since I think 26.N should be kept exclusively for bug-fixes).

Once you fix the two minor problems I mention below, looks good to me,


        Stefan


> * lisp/startup.el (early-init-file): New variable, for the filename of
>   the early init file after it has been loaded.
>
> * lisp/startup.el (load-user-init-file): New function, used to
>   eliminate duplicate code in loading the early and regular init
>   files.
>
> * lisp/startup.el (command-line): Load the early init file using
>   `load-user-init-file'. Move the check for an invalid username to
>   just before that, and move the initialization of the package system
>   to just after. Load the regular init file using
>   `load-user-init-file'.

The format for commit messages wants to combine those as follows
(notice also that we use double-spaces after "."):

    * lisp/startup.el (early-init-file): New variable, for the filename of
    the early init file after it has been loaded.
    (load-user-init-file): New function, used to eliminate duplicate code
    in loading the early and regular init files.
    (command-line): Load the early init file using `load-user-init-file'.
    Move the check for an invalid username to just before that, and move
    the initialization of the package system to just after.
    Load the regular init file using `load-user-init-file'.

> * src/lread.c (Vuser_init_file): Note change in semantics due to its
>   usage while loading the early init file.
>
> * lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
>   definition, usage, and documentation.
>
> * lisp/emacs-lisp/package.el (package--init-file-ensured): Remove
>   definition and usage.

Same here, please combine the two messages about lisp/emacs-lisp/package.el.

> * etc/NEWS: Document changes to startup and package.el.

No need to mention these changes to NEWS here.

> diff --git a/lisp/startup.el b/lisp/startup.el
> index 4575f1f94d..de85933983 100644
> --- a/lisp/startup.el
> +++ b/lisp/startup.el
> @@ -312,6 +312,15 @@ inhibit-startup-hooks
>  Currently this applies to: `emacs-startup-hook', `term-setup-hook',
>  and `window-setup-hook'.")
>  
> +(defvar early-init-file nil
> +  "File name, including directory, of user's early init file.
> +If the file loaded had extension `.elc', and the corresponding
> +source file exists, this variable contains the name of source
> +file, suitable for use by functions like `custom-save-all' which
> +edit the init file.  While Emacs loads and evaluates the init
> +file, value is the real name of the file, regardless of whether
> +or not it has the `.elc' extension.")

This duplicates the doc of user-init-file, basically.  I think it'd be
better to refer to user-init-file and just explains in which way
it's different.

Also, IIUC the above is not true: while loading the early init file,
`early-init-file` will not be bound to the file being loaded, instead it
will be `user-init-file` which will be bound to it (and I'd rather not
document that quirk).

> +(defun load-user-init-file
> +    (compute-filename &optional compute-alternate-filename load-defaults)

We usually use names like `filename-function` rather than
`compute-filename` (the function itself could be named
`compute-filename` since a function *does* something, but a variable
doesn't *do* it just contains a value, in this case a function value).

> +  "Load a user init-file.
> +COMPUTE-FILENAME is called with no arguments and should return
> +the name of the init-file to load. If this file cannot be loaded,
> +and COMPUTE-ALTERNATE-FILENAME is non-nil, then it is called with
> +no arguments and should return the name of an alternate init-file
> +to load. If LOAD-DEFAULTS is non-nil, then load default.el after
> +the init-file.
> +
> +This function sets `user-init-file' to the name of the loaded
> +init-file, or to a default value if loading is not possible."
> +  (let ((debug-on-error-from-init-file nil)
> +        (debug-on-error-should-be-set nil)
> +        (debug-on-error-initial
> +         (if (eq init-file-debug t)
> +             'startup
> +           init-file-debug))
> +        (orig-enable-multibyte (default-value 'enable-multibyte-characters)))
> +    (let ((debug-on-error debug-on-error-initial)
> +          ;; We create an anonymous function here so that we can call
> +          ;; it in different contexts depending on the value of
> +          ;; `debug-on-error'.
> +          (read-init-file
> +           (lambda ()
> +             (when init-file-user
> +               (let ((init-file-name (funcall compute-filename)))
> +
> +                 ;; If `user-init-file' is t, then `load' will store
> +                 ;; the name of the file that it loads into
> +                 ;; `user-init-file'.
> +                 (setq user-init-file t)
> +                 (load init-file-name 'noerror 'nomessage)
> +
> +                 (when (eq user-init-file t)
> +                   (let ((alt-file-name (funcall compute-alternate-filename)))
> +                     (load alt-file-name 'noerror 'nomessage)

You could directly do

    (load (funcall compute-alternate-filename) 'noerror 'nomessage)

here.  But more significantly, compute-alternate-filename is an optional
arg yet you forget to check if it's nil!

> +              ;; If a previous init-file had an error, don't forget
> +              ;; about that.
> +              (unless init-file-had-error
> +                (setq init-file-had-error nil)))

There's a problem here, since this is a nop.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 127 bytes --]

I've fixed all the problems noted by Stefan and Clément, and rebased
onto the latest master. The revised patch is attached.

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 31675 bytes --]

From 7e45f0e4ee1eb666c096ed12435f5edb434bb2bf Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable, for the filename of
the early init file after it has been loaded.
(load-user-init-file): New function, used to eliminate duplicate code
in loading the early and regular init files.
(command-line): Load the early init file using `load-user-init-file'.
Move the check for an invalid username to just before that, and move
the initialization of the package system to just after.  Load the
regular init file using `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
definition, usage, and documentation.
(package--init-file-ensured): Remove definition and usage.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
package-initialize is called, and advise against calling it in the
init file.

* doc/emacs/package.texi: Document changes to when package-initialize
is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
init file.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/package.texi     |  31 ++--
 doc/lispref/os.texi        |  14 ++
 doc/lispref/package.texi   |  16 +-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  20 +++
 lisp/emacs-lisp/package.el |  71 +--------
 lisp/startup.el            | 379 ++++++++++++++++++++++++---------------------
 src/lread.c                |   2 +-
 8 files changed, 264 insertions(+), 273 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 5f05bc0f9e..9629f41c80 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -250,31 +250,18 @@ Package Installation
 wide-ranging effects on the Emacs session.  For such information,
 consult the package's help buffer.
 
-  By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+  After a package is installed, it is automatically loaded by Emacs in
+all subsequent sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file, as the variable is read before loading the
+regular init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 0854468835..209e450fbd 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, you can customize the process of
+  loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 21dfe1c271..d5ee4cbd47 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the regular init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is set to @code{nil} in the early
+init file.
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after it in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index 762dfafdda..ef2c931b6d 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index ad31553603..97f08dc0d9 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -49,6 +49,26 @@ to reduce differences between developer and production builds.
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called 'early-init.el', in 'user-emacs-directory'.  It is
+loaded very early in the startup process: before graphical elements
+such as the tool bar are initialized, and before the package manager
+is initialized.
+
++++
+** Emacs now calls 'package-initialize' before loading the init file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to 'package-initialize' into the init
+file, which was previously done when Emacs was started.  You only need
+to make changes to your configuration if some of it needs to be run
+before 'package-initialize' is called (for example, if you set
+'package-load-list' or 'package-user-dir').  In that case, place the
+configuration that needs to be run before 'package-initialize' into
+the early init file.  Note that variables like 'package-archives' can
+be set after 'package-initialize', so they can still be customized in
+the regular init file.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..ab02d4255b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 8c36c19e82..69bc8fa781 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,12 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+See `user-init-file'.  The only difference is that
+`early-init-file' is not set during the course of evaluating the
+early init file.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +876,103 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (filename-function &optional alternate-filename-function load-defaults)
+  "Load a user init-file.
+FILENAME-FUNCTION is called with no arguments and should return
+the name of the init-file to load.  If this file cannot be
+loaded, and ALTERNATE-FILENAME-FUNCTION is non-nil, then it is
+called with no arguments and should return the name of an
+alternate init-file to load.  If LOAD-DEFAULTS is non-nil, then
+load default.el after the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall filename-function)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (and (eq user-init-file t) alternate-filename-function)
+                   (load (funcall alternate-filename-function)
+                         'noerror 'nomessage))
+
+                 ;; If we did not find the user's init file, set
+                 ;; user-init-file conclusively.  Don't let it be
+                 ;; set from default.el.
+                 (when (eq user-init-file t)
+                   (setq user-init-file init-file-name)))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (funcall read-init-file)
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1124,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1128,153 +1294,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1296,33 +1356,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 28d4bf9a4f..0f674803f2 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4897,7 +4897,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.16.1

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> I've fixed all the problems noted by Stefan and Clément, and rebased
> onto the latest master. The revised patch is attached.

LGTM
Do you have write access or do you need someone to push this for you?
Eli&John, any objection to installing this into master?


        Stefan


PS: In the mean time, some more comments to keep us busy:

> * lisp/startup.el (early-init-file): New variable, for the filename of
> the early init file after it has been loaded.

"New variable" is enough, the rest is described in the code already.

> (load-user-init-file): New function, used to eliminate duplicate code
> in loading the early and regular init files.

Same here.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 30 Jan 2018 10:02:53 -0500
> 
> > I've fixed all the problems noted by Stefan and Clément, and rebased
> > onto the latest master. The revised patch is attached.
> 
> LGTM
> Do you have write access or do you need someone to push this for you?
> Eli&John, any objection to installing this into master?

I didn't yet have time to read the patch, will do in the following
days, and get back.  I don't expect to object, but I might have some
comments.

Re: [PATCH] Fixing package-initialize, adding early init file

[Mark Oteiza]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Tue, 30 Jan 2018 10:02:53 -0500
>> 
>> > I've fixed all the problems noted by Stefan and Clément, and rebased
>> > onto the latest master. The revised patch is attached.
>> 
>> LGTM
>> Do you have write access or do you need someone to push this for you?
>> Eli&John, any objection to installing this into master?
>
> I didn't yet have time to read the patch, will do in the following
> days, and get back.  I don't expect to object, but I might have some
> comments.

ISTR there being no answer to the question posed here
https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00994.html
except for a workaround suggested by Stefan to set package-user-dir,
which implies that we have to run package-initialize twice.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

> ISTR there being no answer to the question posed here
> https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00994.html
> except for a workaround suggested by Stefan to set package-user-dir,
> which implies that we have to run package-initialize twice.

Sorry, could you elaborate on what the problem is? As it is I don't see any
problem: most configurations will work with no changes, and advanced
configurations will require only trivial changes (namely moving a few 'setq'
forms from init.el into early-init.el). People whose configurations are
advanced enough to set `package-load-list' and `package-user-dir' will find
this change trivial to make.

There's certainly no running of `package-initialize' twice in the proposed
patch. `package-initialize' is run between early-init.el and init.el, just
once.
(But this can be suppressed by setting `package-enable-at-startup' to nil in
early-init.el.)

-- Radon

[-- Attachment #2: Type: text/html, Size: 2024 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> forms from init.el into early-init.el).  People whose configurations are
> advanced enough to set `package-load-list' and `package-user-dir' will find
> this change trivial to make.

FWIW, I do set package-user-dir in my config, and I resolve the problem
by moving my whole config from .emacs.d/init.el to
.emacs.d/early-init.el.

This comes with a few constraints, but these already existed for users
of --daemon.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> forms from init.el into early-init.el).  People whose configurations are
>> advanced enough to set `package-load-list' and `package-user-dir' will find
>> this change trivial to make.
> FWIW, I do set package-user-dir in my config, and I resolve the problem
> by moving my whole config from .emacs.d/init.el to
> .emacs.d/early-init.el.

Another option is to use my package-quickstart thingy: it uses
package-load-list and package-user-dir to build one big autoloads file
which not only can be quickly loaded by Emacs at startup but it can do
so before package-load-list and package-user-dir have been set!


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> I didn't yet have time to read the patch, will do in the following
> days, and get back.  I don't expect to object, but I might have some
> comments.

Have you had the chance to take a look yet?

Thanks,
-- Radon

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Wed, 7 Feb 2018 21:54:54 -0800
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel <emacs-devel@gnu.org>
> 
> > I didn't yet have time to read the patch, will do in the following
> > days, and get back.  I don't expect to object, but I might have some
> > comments.
> 
> Have you had the chance to take a look yet?

Not yet, sorry.  I hope to find time for that during the next few
days.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1.1: Type: text/plain, Size: 442 bytes --]

> Do you have write access or do you need someone to push this for you?

The latter.

> > * lisp/startup.el (early-init-file): New variable, for the filename of
> > the early init file after it has been loaded.
>
> "New variable" is enough, the rest is described in the code already.
>
> > (load-user-init-file): New function, used to eliminate duplicate code
> > in loading the early and regular init files.
>
> Same here.

Fixed.

-- Radon

[-- Attachment #1.2: Type: text/html, Size: 1228 bytes --]

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 31531 bytes --]

From dfe5ebd007c2fdab8b6e7d72891ef3acf3f31df6 Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable.
(load-user-init-file): New function.
(command-line): Load the early init file using `load-user-init-file'.
Move the check for an invalid username to just before that, and move
the initialization of the package system to just after.  Load the
regular init file using `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
definition, usage, and documentation.
(package--init-file-ensured): Remove definition and usage.

* doc/lispref/os.texi: Document early init file.

* doc/lispref/package.texi: Document changes to when
package-initialize is called, and advise against calling it in the
init file.

* doc/emacs/package.texi: Document changes to when package-initialize
is called.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
init file.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/package.texi     |  31 ++--
 doc/lispref/os.texi        |  14 ++
 doc/lispref/package.texi   |  16 +-
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  20 +++
 lisp/emacs-lisp/package.el |  71 +--------
 lisp/startup.el            | 379 ++++++++++++++++++++++++---------------------
 src/lread.c                |   2 +-
 8 files changed, 264 insertions(+), 273 deletions(-)

diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index 5f05bc0f9e..9629f41c80 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -250,31 +250,18 @@ Package Installation
 wide-ranging effects on the Emacs session.  For such information,
 consult the package's help buffer.
 
-  By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+  After a package is installed, it is automatically loaded by Emacs in
+all subsequent sessions.  This happens at startup, before processing
+the init file but after processing the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an
+exception, Emacs does not load packages at startup if invoked with the
+@samp{-q} or @samp{--no-init-file} options (@xref{Initial Options}).
 
 @vindex package-enable-at-startup
   To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
-
-@findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+@code{package-enable-at-startup} to @code{nil}.  You must do this in
+the early init file, as the variable is read before loading the
+regular init file.  Currently it cannot be done via Customize.
 
 @vindex package-load-list
   For finer control over package loading, you can use the variable
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 9352a929a7..c76c4d80b6 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -361,6 +361,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +385,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+  @dfn{early init file}, if it exists.  This is a file named
+  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
+  home directory.  The difference is that the early init file is
+  loaded much earlier during the startup process, so you can use it to
+  customize some things that are initialized before loading the
+  regular init file.  For example, you can customize the process of
+  loading installed packages, by setting variables such as
+  @var{package-load-list} or
+  @var{package-enable-at-startup}. @xref{Package Installation,,,
+  emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 21dfe1c271..d5ee4cbd47 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -106,10 +106,12 @@ Packaging Basics
 
   Whenever Emacs starts up, it automatically calls the function
 @code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+after loading the early init file, but before loading the regular init
+file and abbrev file (if any) and before running
+@code{after-init-hook} (@pxref{Startup Summary}).  Automatic package
+loading is disabled if the user option
+@code{package-enable-at-startup} is set to @code{nil} in the early
+init file.
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
@@ -123,6 +125,12 @@ Packaging Basics
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
 loading them; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after it in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index aa3b029ab7..68aa01ca18 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index b28f284116..a440c187c4 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -49,6 +49,26 @@ to reduce differences between developer and production builds.
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called 'early-init.el', in 'user-emacs-directory'.  It is
+loaded very early in the startup process: before graphical elements
+such as the tool bar are initialized, and before the package manager
+is initialized.
+
++++
+** Emacs now calls 'package-initialize' before loading the init file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to 'package-initialize' into the init
+file, which was previously done when Emacs was started.  You only need
+to make changes to your configuration if some of it needs to be run
+before 'package-initialize' is called (for example, if you set
+'package-load-list' or 'package-user-dir').  In that case, place the
+configuration that needs to be run before 'package-initialize' into
+the early init file.  Note that variables like 'package-archives' can
+be set after 'package-initialize', so they can still be customized in
+the regular init file.
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..ab02d4255b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 8c36c19e82..69bc8fa781 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,12 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+See `user-init-file'.  The only difference is that
+`early-init-file' is not set during the course of evaluating the
+early init file.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +876,103 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (filename-function &optional alternate-filename-function load-defaults)
+  "Load a user init-file.
+FILENAME-FUNCTION is called with no arguments and should return
+the name of the init-file to load.  If this file cannot be
+loaded, and ALTERNATE-FILENAME-FUNCTION is non-nil, then it is
+called with no arguments and should return the name of an
+alternate init-file to load.  If LOAD-DEFAULTS is non-nil, then
+load default.el after the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall filename-function)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (and (eq user-init-file t) alternate-filename-function)
+                   (load (funcall alternate-filename-function)
+                         'noerror 'nomessage))
+
+                 ;; If we did not find the user's init file, set
+                 ;; user-init-file conclusively.  Don't let it be
+                 ;; set from default.el.
+                 (when (eq user-init-file t)
+                   (setq user-init-file init-file-name)))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (funcall read-init-file)
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1124,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1128,153 +1294,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1296,33 +1356,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 3b0a17c90b..ac1b3da7b7 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4886,7 +4886,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.16.1

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sun, 28 Jan 2018 11:42:53 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> I've fixed all the problems noted by Stefan and Clément, and rebased
> onto the latest master. The revised patch is attached.

Thanks, I have a few minor comments below about the documentation
parts.

> --- a/doc/emacs/package.texi
> +++ b/doc/emacs/package.texi
> @@ -250,31 +250,18 @@ Package Installation
>  wide-ranging effects on the Emacs session.  For such information,
>  consult the package's help buffer.
>  
> -  By default, Emacs also automatically loads all installed packages in
> -subsequent Emacs sessions.  This happens at startup, after processing
> -the init file (@pxref{Init File}).  As an exception, Emacs does not
> -load packages at startup if invoked with the @samp{-q} or
> -@samp{--no-init-file} options (@pxref{Initial Options}).
> +  After a package is installed, it is automatically loaded by Emacs in
> +all subsequent sessions.

Is "automatically loaded" here accurate?  Wouldn't it be better to say
"it is available in all subsequent sessions, in that its functions and
variables will be automatically loaded when needed"?

> +the init file but after processing the early init file (@pxref{Early
> +Init File,,, elisp, The Emacs Lisp Reference Manual}).  As an

We cannot document the early init file, which is a user-level feature,
only in the ELisp manual.  it should be documented in the Emacs user
manual, where the "regular" init file is described.

> -@findex package-initialize

It looks like we are removing the index entry for package-initialize,
although it is a command?

> +@code{package-enable-at-startup} to @code{nil}.  You must do this in
> +the early init file, as the variable is read before loading the
> +regular init file.

There should be a cross-reference here to where the early init file is
described.

> +@cindex early init file
> +  Emacs also attempts to load a second init file, called the
> +  @dfn{early init file}, if it exists.  This is a file named

There's indentation here which is contrary to how we write stuff in
the manual sources.

> +  @file{early-init.el} in a subdirectory named @file{.emacs.d} in your
> +  home directory.

It's better and shorter to say

  @file{early-init.el} in your @file{~/.emacs.d} directory

Btw, is it only looked up in .emacs.d, or also in the directory where
.emacs is searched (which could be just the home directory)?

>                        The difference is that the early init file is
                         ^^^^^^^^^^^^^^^^^
Difference between what and what?

> +after loading the early init file, but before loading the regular init
> +file and abbrev file (if any) and before running
                                ^
You need a comma here.

> +In most cases, you should not need to call @code{package-initialize},
> +as this is done automatically during startup.  Simply make sure to put
> +any code that should run before @code{package-initialize} in the early
> +init file, and any code that should run after it in the primary init
> +file (@xref{Init File,,, emacs, The GNU Emacs Manual}).

This is for users, so it should be in the user manual, not in ELisp
manual.

> --- a/doc/misc/org.texi
> +++ b/doc/misc/org.texi
> @@ -890,9 +890,7 @@ Installation
>  been visited, i.e., where no Org built-in function have been loaded.
>  Otherwise autoload Org functions will mess up the installation.
>  
> -Then, to make sure your Org configuration is taken into account, initialize
> -the package system with @code{(package-initialize)} in your Emacs init file
> -before setting any Org option.  If you want to use Org's package repository,
> +If you want to use Org's package repository,
>  check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.

Why was this hunk necessary?

> +** Emacs can now be configured using an early init file.
> +The file is called 'early-init.el', in 'user-emacs-directory'.  It is
> +loaded very early in the startup process: before graphical elements
> +such as the tool bar are initialized, and before the package manager
> +is initialized.

Please add here a sentence regarding the purpose of this file (to
configure package.el initialization).

> +** Emacs now calls 'package-initialize' before loading the init file.
> +This is part of a change intended to eliminate the behavior of
> +package.el inserting a call to 'package-initialize' into the init
> +file, which was previously done when Emacs was started.  You only need
> +to make changes to your configuration if some of it needs to be run
> +before 'package-initialize' is called (for example, if you set
> +'package-load-list' or 'package-user-dir').  In that case, place the
> +configuration that needs to be run before 'package-initialize' into
> +the early init file.  Note that variables like 'package-archives' can
> +be set after 'package-initialize', so they can still be customized in
> +the regular init file.

The part starting with "You only need to make changes" sounds
unrelated to what precedes it.  You should probably insert there
something which explains that calling package-initialize before
loading the init file eliminates the need for such changes in most
cases?

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> +  After a package is installed, it is automatically loaded by Emacs in
>> +all subsequent sessions.
> Is "automatically loaded" here accurate?

Indeed not.

> Wouldn't it be better to say "it is available in all subsequent
> sessions, in that its functions and variables will be automatically
> loaded when needed"?

I use the term "activated" to describe this (what activation does
depends on the package).  Whether we stick to this word or pick another,
I think it's important we pick a word for that and define it in
the manual.

>> -@findex package-initialize
> It looks like we are removing the index entry for package-initialize,
> although it is a command?

[ ...looking up package-initialize... ] OMG, you're right
it's interactive.  Does it make sense to use it interactively, really?


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> [ ...looking up package-initialize... ] OMG, you're right
> it's interactive.  Does it make sense to use it interactively, really?

Yes, I've done so dozens of times when I'm testing package.el. The
purpose of the command is to activate all installed packages, without
doing anything else. And that's what I use it for (as someone who
disables `package-enable-at-startup').

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>> [ ...looking up package-initialize... ] OMG, you're right
>> it's interactive.  Does it make sense to use it interactively, really?
> Yes, I've done so dozens of times when I'm testing package.el.

I consider that for such purposes `M-: (package-initialize) RET` (or the
same done from IELM) is quite sufficient.

"Use it interactively" here is meant for "normal users", not for someone
hacking on package.el.

> The purpose of the command is to activate all installed packages,
> without doing anything else.

Actually, it has 2 purposes.  Activating the packages is only one of
the two: in order to speed up startup by caching the autoloads like
I do in my package-fastpath patch, we will need to separate those
two purposes.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> [ ...looking up package-initialize... ] OMG, you're right
> it's interactive.  Does it make sense to use it interactively, really?

Judging from personal experience, I would guess the main users of its
interactivity are those working on or debugging packages following an
emacs -Q, which in some work sessions can happen multiple times (until
they cop on and try to automate this, that is :).

Apart from backward compatibility and M-x p-ini RET being slightly
faster than M-: M-( p-ini RET in these cases, I don't think there is
much else to be said for it.

-- 
Basil

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1.1: Type: text/plain, Size: 1444 bytes --]

Unless otherwise noted, I have integrated your comments, Eli. Let
me know if more changes are needed before merging.

I also noticed that the startup summary was not updated to reflect
the early init file and new timing of `package-initialize'. That's fixed
now.

> Btw, is it only looked up in .emacs.d, or also in the directory where
> .emacs is searched (which could be just the home directory)?

No, since .emacs is deprecated in favor of ~/.emacs.d/init.el.

> This is for users, so it should be in the user manual, not in ELisp
> manual.

While re-adding the index entry for `package-initialize', I also partially
re-added the paragraph on `package-initialize' in the user manual. The
user manual now documents `package-initialize' as only something
that is useful to call if you set `package-enable-at-startup' to nil, so
this
paragraph is not really needed in the user manual. I think it is still
useful to have it in the Elisp manual, since unlike the user manual, the
Elisp manual actually does document `package-initialize' in some
detail, so users might be misled into thinking that it is a useful function
when in fact they probably should not be using it.

> Why was this hunk necessary?

Because the description was wrong and needed to be removed. It
advised that users call `package-initialize' in their init file, which is
wrong advice if `package-initialize' is called automatically before
loading the init file.

Best,
Radon

[-- Attachment #1.2: Type: text/html, Size: 3279 bytes --]

[-- Attachment #2: 0001-Add-early-init-file-stop-package-initialize-insertio.patch --]
[-- Type: application/octet-stream, Size: 38832 bytes --]

From aaf0192d3da753df1f5e0dab1907affc719e191c Mon Sep 17 00:00:00 2001
From: Radon Rosborough <radon.neon@gmail.com>
Date: Sun, 17 Dec 2017 17:31:17 -0700
Subject: [PATCH] Add early init file, stop package-initialize insertion

* lisp/startup.el (early-init-file): New variable.
(load-user-init-file): New function.
(command-line): Load the early init file using `load-user-init-file'.
Move the check for an invalid username to just before that, and move
the initialization of the package system to just after.  Load the
regular init file using `load-user-init-file'.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
usage while loading the early init file.

* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
definition, usage, and documentation.
(package--init-file-ensured): Remove definition and usage.

* doc/emacs/custom.texi: Document early init file.

* doc/lispref/os.texi: Document early init file.  Update startup
summary.

* doc/lispref/package.texi: Document changes to when
package-initialize is called, and advise against calling it in the
init file.  Change terminology for package 'loading'.

* doc/emacs/package.texi: Document changes to when package-initialize
is called.  Change terminology for package 'loading'.

* doc/misc/org.texi: Don't recommend to call package-initialize in the
init file.

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html
---
 doc/emacs/custom.texi      |  17 ++
 doc/emacs/package.texi     |  78 +++++-----
 doc/lispref/os.texi        |  38 +++--
 doc/lispref/package.texi   |  30 ++--
 doc/misc/org.texi          |   4 +-
 etc/NEWS                   |  19 +++
 lisp/emacs-lisp/package.el |  71 +--------
 lisp/startup.el            | 379 ++++++++++++++++++++++++---------------------
 src/lread.c                |   2 +-
 9 files changed, 331 insertions(+), 307 deletions(-)

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index e27760b379..263dff7cb0 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2575,3 +2575,20 @@ Init Non-ASCII
 coding system, for your init file as well as the files you edit.  For
 example, don't mix the @samp{latin-1} and @samp{latin-9} coding
 systems.
+
+@node Early Init File
+@subsection The Early Init File
+@cindex early init file
+
+  Most customizations for Emacs can be put in the normal init file,
+@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes
+desirable to have customizations that take effect during Emacs startup
+earlier than the normal init file is processed.  Such customizations
+can be put in the early init file, @file{~/.emacs.d/early-init.el}.
+This file is loaded before the package system is initialized, so in it
+you can customize variables that affect the initialization process,
+such as @code{package-enable-at-startup} and @code{package-load-list}.
+@xref{Package Installation}.
+
+  For more information on the early init file, @pxref{Early Init
+File,,, elisp, The Emacs Lisp Reference Manual}.
diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index bc6afb7966..d5339e9124 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -241,57 +241,55 @@ Package Installation
 package is available from a higher-priority archive.  (This is
 controlled by the value of @code{package-menu-hide-low-priority}.)
 
-  Once a package is downloaded and installed, it is @dfn{loaded} into
-the current Emacs session.  Loading a package is not quite the same as
-loading a Lisp library (@pxref{Lisp Libraries}); loading a package
-adds its directory to @code{load-path} and loads its autoloads.  The
-effect of a package's autoloads varies from package to package.  Most
-packages just make some new commands available, while others have more
+  Once a package is downloaded and installed, it is made available to
+the current Emacs session.  Making a package available adds its
+directory to @code{load-path} and loads its autoloads.  The effect of
+a package's autoloads varies from package to package.  Most packages
+just make some new commands available, while others have more
 wide-ranging effects on the Emacs session.  For such information,
 consult the package's help buffer.
 
-  By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+  After a package is installed, it is automatically made available by
+Emacs in all subsequent sessions.  This happens at startup, before
+processing the init file but after processing the early init file
+(@pxref{Early Init File,,, elisp, The Emacs Lisp Reference Manual}).
+As an exception, Emacs does not make packages available at startup if
+invoked with the @samp{-q} or @samp{--no-init-file} options
+(@xref{Initial Options}).
 
 @vindex package-enable-at-startup
-  To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
+  To keep Emacs from automatically making packages available at
+startup, change the variable @code{package-enable-at-startup} to
+@code{nil}.  You must do this in the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}), as the variable
+is read before loading the regular init file.  Currently this variable
+cannot be set via Customize.
 
 @findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+  If you have set @code{package-enable-at-startup} to @code{nil}, you
+can still make packages available either during or after startup.  To
+make installed packages available during startup, call the function
+@code{package-initialize} in your init file.  To make installed
+packages available after startup, invoke the command @kbd{M-x
+package-initialize}.
 
 @vindex package-load-list
-  For finer control over package loading, you can use the variable
-@code{package-load-list}.  Its value should be a list.  A list element
-of the form @code{(@var{name} @var{version})} tells Emacs to load
-version @var{version} of the package named @var{name}.  Here,
-@var{version} should be a version string (corresponding to a specific
-version of the package), or @code{t} (which means to load any
-installed version), or @code{nil} (which means no version; this
-disables the package, preventing it from being loaded).  A list
-element can also be the symbol @code{all}, which means to load the
-latest installed version of any package not named by the other list
-elements.  The default value is just @code{'(all)}.
+  For finer control over which packages are made available at startup,
+you can use the variable @code{package-load-list}.  Its value should
+be a list.  A list element of the form @code{(@var{name}
+@var{version})} tells Emacs to make available version @var{version} of
+the package named @var{name}.  Here, @var{version} should be a version
+string (corresponding to a specific version of the package), or
+@code{t} (which means to make available any installed version), or
+@code{nil} (which means no version; this disables the package,
+preventing it from being made available).  A list element can also be
+the symbol @code{all}, which means to make available the latest
+installed version of any package not named by the other list elements.
+The default value is just @code{'(all)}.
 
   For example, if you set @code{package-load-list} to @code{'((muse
-"3.20") all)}, then Emacs only loads version 3.20 of the @samp{muse}
-package, plus any installed version of packages other than
+"3.20") all)}, then Emacs only makes available version 3.20 of the
+@samp{muse} package, plus any installed version of packages other than
 @samp{muse}.  Any other version of @samp{muse} that happens to be
 installed will be ignored.  The @samp{muse} package will be listed in
 the package menu with the @samp{held} status.
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 42be60449d..cac5bfa067 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -95,6 +95,21 @@ Startup Summary
 @item
 It does some basic parsing of the command-line arguments.
 
+@item
+It loads your early init file (@pxref{Early Init File}).  This is not
+done if the options @samp{-q}, @samp{-Q}, or @samp{--batch} were
+specified.  If the @samp{-u} option was specified, Emacs looks for the
+init file in that user's home directory instead.
+
+@item
+It calls the function @code{package-initialize} to activate any
+optional Emacs Lisp package that has been installed.  @xref{Packaging
+Basics}.  However, Emacs doesn't initialize packages when
+@code{package-enable-at-startup} is @code{nil} or when it's started
+with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
+initialize packages in the latter case, @code{package-initialize}
+should be called explicitly (e.g., via the @samp{--funcall} option).
+
 @vindex initial-window-system@r{, and startup}
 @vindex window-system-initialization-alist
 @item
@@ -154,15 +169,6 @@ Startup Summary
 (@pxref{Abbrev Files, abbrev-file-name}).  This is not done if the
 option @samp{--batch} was specified.
 
-@item
-It calls the function @code{package-initialize} to activate any
-optional Emacs Lisp package that has been installed.  @xref{Packaging
-Basics}.  However, Emacs doesn't initialize packages when
-@code{package-enable-at-startup} is @code{nil} or when it's started
-with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
-initialize packages in the latter case, @code{package-initialize}
-should be called explicitly (e.g., via the @samp{--funcall} option).
-
 @vindex after-init-time
 @item
 It sets the variable @code{after-init-time} to the value of
@@ -361,6 +367,7 @@ Init File
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +391,19 @@ Init File
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+@dfn{early init file}, if it exists.  This is a file named
+@file{early-init.el} in your @file{~/.emacs.d} directory.  The
+difference between the early init file and the regular init file is
+that the early init file is loaded much earlier during the startup
+process, so you can use it to customize some things that are
+initialized before loading the regular init file.  For example, you
+can customize the process of initializing the package system, by
+setting variables such as @var{package-load-list} or
+@var{package-enable-at-startup}. @xref{Package Installation,,,
+emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 21dfe1c271..877aaf89a3 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -105,24 +105,32 @@ Packaging Basics
 evaluates the autoload definitions in @file{@var{name}-autoloads.el}.
 
   Whenever Emacs starts up, it automatically calls the function
-@code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+@code{package-initialize} to make installed packages available to the
+current session.  This is done after loading the early init file, but
+before loading the regular init file (@pxref{Startup Summary}).
+Packages are not automatically made available if the user option
+@code{package-enable-at-startup} is set to @code{nil} in the early
+init file.
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
-installed, and loads them.  The user option @code{package-load-list}
-specifies which packages to load; by default, all installed packages
-are loaded.  If called during startup, this function also sets
+installed, and makes the packages available to the current session.
+The user option @code{package-load-list} specifies which packages to
+make available; by default, all installed packages are made available.
+If called during startup, this function also sets
 @code{package-enable-at-startup} to @code{nil}, to avoid accidentally
-loading the packages twice.  @xref{Package Installation,,, emacs, The
-GNU Emacs Manual}.
+evaluating package autoloads more than once.  @xref{Package
+Installation,,, emacs, The GNU Emacs Manual}.
 
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
-loading them; it is for internal use only.
+making them available; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after it in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index aa3b029ab7..68aa01ca18 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Installation
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive
diff --git a/etc/NEWS b/etc/NEWS
index 71569c95ad..b6b884b816 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -49,6 +49,25 @@ to reduce differences between developer and production builds.
 \f
 * Startup Changes in Emacs 27.1
 
++++
+** Emacs can now be configured using an early init file.
+The file is called 'early-init.el', in 'user-emacs-directory'.  It is
+loaded very early in the startup process: before graphical elements
+such as the tool bar are initialized, and before the package manager
+is initialized.  The primary purpose is to allow customizing how the
+package system is initialized given that initialization now happens
+before loading the regular init file (see below).
+
++++
+** Emacs now calls 'package-initialize' before loading the init file.
+This is part of a change intended to eliminate the behavior of
+package.el inserting a call to 'package-initialize' into the init
+file, which was previously done when Emacs was started.  As a result
+of this change, it is no longer necessary to call 'package-initialize'
+in your init file.  However, if your init file changes the values of
+'package-load-list' or 'package-user-dir', then that code needs to be
+moved to the early init file (see above).
+
 \f
 * Changes in Emacs 27.1
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 71d1c41ec3..ab02d4255b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1431,16 +1431,11 @@ package-read-all-archive-contents
 ;; available on disk.
 (defvar package--initialized nil)
 
-(defvar package--init-file-ensured nil
-  "Whether we know the init file has package-initialize.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
 The variable `package-load-list' controls which packages to load.
 If optional arg NO-ACTIVATE is non-nil, don't activate packages.
-If `user-init-file' does not mention `(package-initialize)', add
-it to the file.
 If called as part of loading `user-init-file', set
 `package-enable-at-startup' to nil, to prevent accidentally
 loading packages twice.
@@ -1449,13 +1444,7 @@ package-initialize
 taken care of by `package-initialize'."
   (interactive)
   (setq package-alist nil)
-  (if after-init-time
-      (package--ensure-init-file)
-    ;; If `package-initialize' is before we finished loading the init
-    ;; file, it's obvious we don't need to ensure-init.
-    (setq package--init-file-ensured t
-          ;; And likely we don't need to run it again after init.
-          package-enable-at-startup nil))
+  (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
   (unless no-activate
@@ -1872,64 +1861,6 @@ package-download-transaction
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))
 
-(defun package--ensure-init-file ()
-  "Ensure that the user's init file has `package-initialize'.
-`package-initialize' doesn't have to be called, as long as it is
-present somewhere in the file, even as a comment.  If it is not,
-add a call to it along with some explanatory comments."
-  ;; Don't mess with the init-file from "emacs -Q".
-  (when (and (stringp user-init-file)
-             (not package--init-file-ensured)
-             (file-readable-p user-init-file)
-             (file-writable-p user-init-file))
-    (let* ((buffer (find-buffer-visiting user-init-file))
-           buffer-name
-           (contains-init
-            (if buffer
-                (with-current-buffer buffer
-                  (save-excursion
-                    (save-restriction
-                      (widen)
-                      (goto-char (point-min))
-                      (re-search-forward "(package-initialize\\_>" nil 'noerror))))
-              ;; Don't visit the file if we don't have to.
-              (with-temp-buffer
-                (insert-file-contents user-init-file)
-                (goto-char (point-min))
-                (re-search-forward "(package-initialize\\_>" nil 'noerror)))))
-      (unless contains-init
-        (with-current-buffer (or buffer
-                                 (let ((delay-mode-hooks t)
-                                       (find-file-visit-truename t))
-                                   (find-file-noselect user-init-file)))
-          (when buffer
-            (setq buffer-name (buffer-file-name))
-            (set-visited-file-name (file-chase-links user-init-file)))
-          (save-excursion
-            (save-restriction
-              (widen)
-              (goto-char (point-min))
-              (while (and (looking-at-p "[[:blank:]]*\\(;\\|$\\)")
-                          (not (eobp)))
-                (forward-line 1))
-              (insert
-               "\n"
-               ";; Added by Package.el.  This must come before configurations of\n"
-               ";; installed packages.  Don't delete this line.  If you don't want it,\n"
-               ";; just comment it out by adding a semicolon to the start of the line.\n"
-               ";; You may delete these explanatory comments.\n"
-               "(package-initialize)\n")
-              (unless (looking-at-p "$")
-                (insert "\n"))
-              (let ((file-precious-flag t))
-                (save-buffer))
-              (if buffer
-                  (progn
-                    (set-visited-file-name buffer-name)
-                    (set-buffer-modified-p nil))
-                (kill-buffer (current-buffer)))))))))
-  (setq package--init-file-ensured t))
-
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
diff --git a/lisp/startup.el b/lisp/startup.el
index 8c36c19e82..69bc8fa781 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -312,6 +312,12 @@ inhibit-startup-hooks
 Currently this applies to: `emacs-startup-hook', `term-setup-hook',
 and `window-setup-hook'.")
 
+(defvar early-init-file nil
+  "File name, including directory, of user's early init file.
+See `user-init-file'.  The only difference is that
+`early-init-file' is not set during the course of evaluating the
+early init file.")
+
 (defvar keyboard-type nil
   "The brand of keyboard you are using.
 This variable is used to define the proper function and keypad
@@ -870,6 +876,103 @@ startup--setup-quote-display
           (when standard-display-table
             (aset standard-display-table char nil)))))))
 
+(defun load-user-init-file
+    (filename-function &optional alternate-filename-function load-defaults)
+  "Load a user init-file.
+FILENAME-FUNCTION is called with no arguments and should return
+the name of the init-file to load.  If this file cannot be
+loaded, and ALTERNATE-FILENAME-FUNCTION is non-nil, then it is
+called with no arguments and should return the name of an
+alternate init-file to load.  If LOAD-DEFAULTS is non-nil, then
+load default.el after the init-file.
+
+This function sets `user-init-file' to the name of the loaded
+init-file, or to a default value if loading is not possible."
+  (let ((debug-on-error-from-init-file nil)
+        (debug-on-error-should-be-set nil)
+        (debug-on-error-initial
+         (if (eq init-file-debug t)
+             'startup
+           init-file-debug)))
+    (let ((debug-on-error debug-on-error-initial)
+          ;; We create an anonymous function here so that we can call
+          ;; it in different contexts depending on the value of
+          ;; `debug-on-error'.
+          (read-init-file
+           (lambda ()
+             (when init-file-user
+               (let ((init-file-name (funcall filename-function)))
+
+                 ;; If `user-init-file' is t, then `load' will store
+                 ;; the name of the file that it loads into
+                 ;; `user-init-file'.
+                 (setq user-init-file t)
+                 (load init-file-name 'noerror 'nomessage)
+
+                 (when (and (eq user-init-file t) alternate-filename-function)
+                   (load (funcall alternate-filename-function)
+                         'noerror 'nomessage))
+
+                 ;; If we did not find the user's init file, set
+                 ;; user-init-file conclusively.  Don't let it be
+                 ;; set from default.el.
+                 (when (eq user-init-file t)
+                   (setq user-init-file init-file-name)))
+
+               ;; If we loaded a compiled file, set `user-init-file' to
+               ;; the source version if that exists.
+               (when (equal (file-name-extension user-init-file)
+                            "elc")
+                 (let* ((source (file-name-sans-extension user-init-file))
+                        (alt (concat source ".el")))
+                   (setq source (cond ((file-exists-p alt) alt)
+                                      ((file-exists-p source) source)
+                                      (t nil)))
+                   (when source
+                     (when (file-newer-than-file-p source user-init-file)
+                       (message "Warning: %s is newer than %s"
+                                source user-init-file)
+                       (sit-for 1))
+                     (setq user-init-file source))))
+
+               (when load-defaults
+
+                 ;; Prevent default.el from changing the value of
+                 ;; `inhibit-startup-screen'.
+                 (let ((inhibit-startup-screen nil))
+                   (load "default" 'noerror 'nomessage)))))))
+      ;; Now call our anonymous function.
+      (if init-file-debug
+          ;; Do this without a `condition-case' if the user wants to
+          ;; debug.
+          (funcall read-init-file)
+        (condition-case error
+            (funcall read-init-file)
+          (error
+           (display-warning
+            'initialization
+            (format-message "\
+An error occurred while loading `%s':\n\n%s%s%s\n\n\
+To ensure normal operation, you should investigate and remove the
+cause of the error in your initialization file.  Start Emacs with
+the `--debug-init' option to view a complete error backtrace."
+                            user-init-file
+                            (get (car error) 'error-message)
+                            (if (cdr error) ": " "")
+                            (mapconcat (lambda (s) (prin1-to-string s t))
+                                       (cdr error) ", "))
+            :warning)
+           (setq init-file-had-error t))))
+
+      ;; If we can tell that the init file altered debug-on-error,
+      ;; arrange to preserve the value that it set up.
+      (or (eq debug-on-error debug-on-error-initial)
+          (setq debug-on-error-should-be-set t
+                debug-on-error-from-init-file debug-on-error)))
+
+    (when debug-on-error-should-be-set
+      (setq debug-on-error debug-on-error-from-init-file))))
+
 (defun command-line ()
   "A subroutine of `normal-top-level'.
 Amongst another things, it parses the command-line arguments."
@@ -1021,6 +1124,69 @@ command-line
     (and command-line-args
          (setcdr command-line-args args)))
 
+  ;; Warn for invalid user name.
+  (when init-file-user
+    (if (string-match "[~/:\n]" init-file-user)
+        (display-warning 'initialization
+                         (format "Invalid user name %s"
+                                 init-file-user)
+                         :error)
+      (if (file-directory-p (expand-file-name
+                             ;; We don't support ~USER on MS-Windows
+                             ;; and MS-DOS except for the current
+                             ;; user, and always load .emacs from
+                             ;; the current user's home directory
+                             ;; (see below).  So always check "~",
+                             ;; even if invoked with "-u USER", or
+                             ;; if $USER or $LOGNAME are set to
+                             ;; something different.
+                             (if (memq system-type '(windows-nt ms-dos))
+                                 "~"
+                               (concat "~" init-file-user))))
+          nil
+        (display-warning 'initialization
+                         (format "User %s has no home directory"
+                                 (if (equal init-file-user "")
+                                     (user-real-login-name)
+                                   init-file-user))
+                         :error))))
+
+  ;; Load the early init file, if found.
+  (load-user-init-file
+   (lambda ()
+     (expand-file-name
+      "early-init"
+      (file-name-as-directory
+       (concat "~" init-file-user "/.emacs.d")))))
+  (setq early-init-file user-init-file)
+
+  ;; If any package directory exists, initialize the package system.
+  (and user-init-file
+       package-enable-at-startup
+       (catch 'package-dir-found
+	 (let (dirs)
+	   (if (boundp 'package-directory-list)
+	       (setq dirs package-directory-list)
+	     (dolist (f load-path)
+	       (and (stringp f)
+		    (equal (file-name-nondirectory f) "site-lisp")
+		    (push (expand-file-name "elpa" f) dirs))))
+	   (push (if (boundp 'package-user-dir)
+		     package-user-dir
+		   (locate-user-emacs-file "elpa"))
+		 dirs)
+	   (dolist (dir dirs)
+	     (when (file-directory-p dir)
+	       (dolist (subdir (directory-files dir))
+		 (when (let ((subdir (expand-file-name subdir dir)))
+                         (and (file-directory-p subdir)
+                              (file-exists-p
+                               (expand-file-name
+                                (package--description-file subdir)
+                                subdir))))
+		   (throw 'package-dir-found t)))))))
+       (package-initialize))
+
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
   ;; Initialize the window-system only after processing the command-line
@@ -1128,153 +1294,47 @@ command-line
     ;; the startup screen.
     (setq inhibit-startup-screen nil)
 
-    ;; Warn for invalid user name.
-    (when init-file-user
-      (if (string-match "[~/:\n]" init-file-user)
-	  (display-warning 'initialization
-			   (format "Invalid user name %s"
-				   init-file-user)
-			   :error)
-	(if (file-directory-p (expand-file-name
-			       ;; We don't support ~USER on MS-Windows
-			       ;; and MS-DOS except for the current
-			       ;; user, and always load .emacs from
-			       ;; the current user's home directory
-			       ;; (see below).  So always check "~",
-			       ;; even if invoked with "-u USER", or
-			       ;; if $USER or $LOGNAME are set to
-			       ;; something different.
-			       (if (memq system-type '(windows-nt ms-dos))
-				   "~"
-				 (concat "~" init-file-user))))
-	    nil
-	  (display-warning 'initialization
-			   (format "User %s has no home directory"
-				   (if (equal init-file-user "")
-				       (user-real-login-name)
-				     init-file-user))
-			   :error))))
-
     ;; Load that user's init file, or the default one, or none.
-    (let (debug-on-error-from-init-file
-	  debug-on-error-should-be-set
-	  (debug-on-error-initial
-	   (if (eq init-file-debug t) 'startup init-file-debug)))
-      (let ((debug-on-error debug-on-error-initial)
-	    ;; This function actually reads the init files.
-	    (inner
-	     (function
-	      (lambda ()
-		(if init-file-user
-		    (let ((user-init-file-1
-			   (cond
-			     ((eq system-type 'ms-dos)
-			      (concat "~" init-file-user "/_emacs"))
-			     ((not (eq system-type 'windows-nt))
-			      (concat "~" init-file-user "/.emacs"))
-			     ;; Else deal with the Windows situation
-			     ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
-			      ;; Prefer .emacs on Windows.
-			      "~/.emacs")
-			     ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
-			      ;; Also support _emacs for compatibility, but warn about it.
-			      (push `(initialization
-				      ,(format-message
-					"`_emacs' init file is deprecated, please use `.emacs'"))
-				    delayed-warnings-list)
-			      "~/_emacs")
-			     (t ;; But default to .emacs if _emacs does not exist.
-			      "~/.emacs"))))
-		      ;; This tells `load' to store the file name found
-		      ;; into user-init-file.
-		      (setq user-init-file t)
-		      (load user-init-file-1 t t)
-
-		      (when (eq user-init-file t)
-			;; If we did not find ~/.emacs, try
-			;; ~/.emacs.d/init.el.
-			(let ((otherfile
-			       (expand-file-name
-				"init"
-				(file-name-as-directory
-				 (concat "~" init-file-user "/.emacs.d")))))
-			  (load otherfile t t)
-
-			  ;; If we did not find the user's init file,
-			  ;; set user-init-file conclusively.
-			  ;; Don't let it be set from default.el.
-			  (when (eq user-init-file t)
-			    (setq user-init-file user-init-file-1))))
-
-		      ;; If we loaded a compiled file, set
-		      ;; `user-init-file' to the source version if that
-		      ;; exists.
-		      (when (and user-init-file
-				 (equal (file-name-extension user-init-file)
-					"elc"))
-			(let* ((source (file-name-sans-extension user-init-file))
-			       (alt (concat source ".el")))
-			  (setq source (cond ((file-exists-p alt) alt)
-					     ((file-exists-p source) source)
-					     (t nil)))
-			  (when source
-			    (when (file-newer-than-file-p source user-init-file)
-			      (message "Warning: %s is newer than %s"
-				       source user-init-file)
-			      (sit-for 1))
-			    (setq user-init-file source))))
-
-		      (unless inhibit-default-init
-                        (let ((inhibit-startup-screen nil))
-                          ;; Users are supposed to be told their rights.
-                          ;; (Plus how to get help and how to undo.)
-                          ;; Don't you dare turn this off for anyone
-                          ;; except yourself.
-                          (load "default" t t)))))))))
-	(if init-file-debug
-	    ;; Do this without a condition-case if the user wants to debug.
-	    (funcall inner)
-	  (condition-case error
-	      (progn
-		(funcall inner)
-		(setq init-file-had-error nil))
-	    (error
-	     (display-warning
-	      'initialization
-	      (format-message "\
-An error occurred while loading `%s':\n\n%s%s%s\n\n\
-To ensure normal operation, you should investigate and remove the
-cause of the error in your initialization file.  Start Emacs with
-the `--debug-init' option to view a complete error backtrace."
-		      user-init-file
-		      (get (car error) 'error-message)
-		      (if (cdr error) ": " "")
-		      (mapconcat (lambda (s) (prin1-to-string s t))
-				 (cdr error) ", "))
-	      :warning)
-	     (setq init-file-had-error t))))
-
-      (if (and deactivate-mark transient-mark-mode)
-	    (with-current-buffer (window-buffer)
-	      (deactivate-mark)))
-
-	;; If the user has a file of abbrevs, read it (unless -batch).
-	(when (and (not noninteractive)
-		   (file-exists-p abbrev-file-name)
-		   (file-readable-p abbrev-file-name))
-	    (quietly-read-abbrev-file abbrev-file-name))
-
-	;; If the abbrevs came entirely from the init file or the
-	;; abbrevs file, they do not need saving.
-	(setq abbrevs-changed nil)
-
-	;; If we can tell that the init file altered debug-on-error,
-	;; arrange to preserve the value that it set up.
-	(or (eq debug-on-error debug-on-error-initial)
-	    (setq debug-on-error-should-be-set t
-		  debug-on-error-from-init-file debug-on-error)))
-      (if debug-on-error-should-be-set
-	  (setq debug-on-error debug-on-error-from-init-file)))
+    (load-user-init-file
+     (lambda ()
+       (cond
+        ((eq system-type 'ms-dos)
+         (concat "~" init-file-user "/_emacs"))
+        ((not (eq system-type 'windows-nt))
+         (concat "~" init-file-user "/.emacs"))
+        ;; Else deal with the Windows situation.
+        ((directory-files "~" nil "^\\.emacs\\(\\.elc?\\)?$")
+         ;; Prefer .emacs on Windows.
+         "~/.emacs")
+        ((directory-files "~" nil "^_emacs\\(\\.elc?\\)?$")
+         ;; Also support _emacs for compatibility, but warn about it.
+         (push `(initialization
+                 ,(format-message
+                   "`_emacs' init file is deprecated, please use `.emacs'"))
+               delayed-warnings-list)
+         "~/_emacs")
+        (t ;; But default to .emacs if _emacs does not exist.
+         "~/.emacs")))
+     (lambda ()
+       (expand-file-name
+        "init"
+        (file-name-as-directory
+         (concat "~" init-file-user "/.emacs.d"))))
+     (not inhibit-default-init))
+
+    (when (and deactivate-mark transient-mark-mode)
+      (with-current-buffer (window-buffer)
+        (deactivate-mark)))
+
+    ;; If the user has a file of abbrevs, read it (unless -batch).
+    (when (and (not noninteractive)
+               (file-exists-p abbrev-file-name)
+               (file-readable-p abbrev-file-name))
+      (quietly-read-abbrev-file abbrev-file-name))
+
+    ;; If the abbrevs came entirely from the init file or the
+    ;; abbrevs file, they do not need saving.
+    (setq abbrevs-changed nil)
 
     ;; Do this here in case the init file sets mail-host-address.
     (and mail-host-address
@@ -1296,33 +1356,6 @@ command-line
 		 (eq face-ignored-fonts old-face-ignored-fonts))
       (clear-face-cache)))
 
-  ;; If any package directory exists, initialize the package system.
-  (and user-init-file
-       package-enable-at-startup
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-initialize))
-
   (setq after-init-time (current-time))
   ;; Display any accumulated warnings after all functions in
   ;; `after-init-hook' like `desktop-read' have finalized possible
diff --git a/src/lread.c b/src/lread.c
index 7cacd47d51..d009bd0cd2 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4922,7 +4922,7 @@ directory.  These file names are converted to absolute at startup.  */);
 If the file loaded had extension `.elc', and the corresponding source file
 exists, this variable contains the name of source file, suitable for use
 by functions like `custom-save-all' which edit the init file.
-While Emacs loads and evaluates the init file, value is the real name
+While Emacs loads and evaluates any init file, value is the real name
 of the file, regardless of whether or not it has the `.elc' extension.  */);
   Vuser_init_file = Qnil;
 
-- 
2.16.1

RE: [PATCH] Fixing package-initialize, adding early init file

[Drew Adams]

>> Btw, is it only looked up in .emacs.d, or also in the directory where
>> .emacs is searched (which could be just the home directory)?
>
> No, since .emacs is deprecated in favor of ~/.emacs.d/init.el.

Huh?  Deprecation does not mean desupport.
Emacs still supports ~/.emacs, AFAIK.

(And why should ~/.emacs have been deprecated?
It's fine to recommend ~/.emacs.d/init.el, of course.)

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

>>> Btw, is it only looked up in .emacs.d, or also in the directory where
>>> .emacs is searched (which could be just the home directory)?
>> No, since .emacs is deprecated in favor of ~/.emacs.d/init.el.
> Huh?  Deprecation does not mean desupport.
> Emacs still supports ~/.emacs, AFAIK.

He just means that he didn't find it necessary to provide a ~/.<foo>
alternative to the ~/.emacs,.d/early-init.el file.

FWIW I completely agree that it doesn't seem to be worthwhile to add
support for an alternate ~/.<foo> name: it would only make sense to do
that in order to provide backward compatibility with a preexisting such
~/.<foo>, but in this case there's no such preexisting file.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[John Wiegley]

>>>>> "DA" == Drew Adams <drew.adams@oracle.com> writes:

DA> Huh?  Deprecation does not mean desupport.
DA> Emacs still supports ~/.emacs, AFAIK.

DA> (And why should ~/.emacs have been deprecated?
DA> It's fine to recommend ~/.emacs.d/init.el, of course.)

I see we have an xdg.el package. Is ~/.config/emacs/init.el a possibility yet?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> DA> Huh?  Deprecation does not mean desupport.
> DA> Emacs still supports ~/.emacs, AFAIK.
> DA> (And why should ~/.emacs have been deprecated?
> DA> It's fine to recommend ~/.emacs.d/init.el, of course.)
> I see we have an xdg.el package. Is ~/.config/emacs/init.el a possibility yet?

As long as xdg.el is not preloaded in the dumped Emacs, I think not.


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> As long as xdg.el is not preloaded in the dumped Emacs, I think not.

Then may I suggest that we...

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [PATCH] Fixing package-initialize, adding early init file

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Wed, 14 Feb 2018 20:38:14 -0800
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Clément Pit-Claudel <cpitclaudel@gmail.com>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> Unless otherwise noted, I have integrated your comments, Eli. Let
> me know if more changes are needed before merging.

Thanks, I pushed this to the master branch.

A few minor nits about the docs, for the future:

 . when you add new nodes to a manual, you need to update higher-level
   menus accordingly, in the parent section/chapter and in the
   top-level menu in emacs.texi or elisp.texi
 . @xref is inappropriate for parenthesized notes; use @pxref instead
 . there were a couple of places with only one blank between
   sentences.
 . long expressions with embedded whitespace should be included in
   @w{..}, to prevent breaking them between lines, which makes them
   less readable

Re: [PATCH] Fixing package-initialize, adding early init file

[Clément Pit-Claudel]

On 2018-02-17 06:38, Eli Zaretskii wrote:
> Thanks, I pushed this to the master branch.

Radon, congratulations for the awesome work!
And thanks Eli (and everyone else) for proofreading and merging.

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]

> Thanks, I pushed this to the master branch.

[ confetti ]

> A few minor nits about the docs, for the future

Thanks, this is very helpful. I never did anything with texinfo before
this, so good to know.
​

[-- Attachment #2: Type: text/html, Size: 539 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Basil L. Contovounesios]

This feature ("Add early init file, stop package-initialize insertion",
commit 24acb31c04) seems to cause the following behaviour for me:

1. cd
2. mv .emacs.d .emacs.d.bak
   (My user-init-file lives under user-emacs-directory, so this is like
    erasing user-init-file and uninstalling all packages.)
3. emacs --no-site-file
4. M-x package-install RET bbdb RET
   (I think any package with a "dir" file will do - lmc.el does not
    cause any issues, for example.)
5. C-x C-c
6. emacs --no-site-file
7. C-h e

The first line in the *Messages* buffer reads

  Symbol’s value as variable is void: Info-default-directory-list

followed by "[N times]" where N (I guess) is proportional to the number
of packages installed in package-user-dir.

Repeating step 6 with --debug-init doesn't change anything, and neither
does Stefan's latest "Use condition-case-unless-debug" commit
eb3337cdb3.

Any ideas on what is happening and how to debug/fix this?

Thanks,

-- 
Basil

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> Any ideas on what is happening and how to debug/fix this?

I saw this error before, but for some reason concluded that it wasn't
due to my change. Thus I didn't look too much into it, but I would
assume that it is because package initialization happens earlier now,
and package.el does not take care to initialize the Info system if
it's not done already.

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> The first line in the *Messages* buffer reads
>   Symbol’s value as variable is void: Info-default-directory-list

Hmm... indeed, I see the problem: in `command-line`, we do:

- load early-init.el
- activate all packages
- do various frame initializations (e.g. create the GUI frame)
- run custom-reevaluate-setting on the custom-delayed-init-variables
- load init.el

and until we run custom-reevaluate-setting, those delayed vars are
still unbound.

I don't know if it's safe to do the custom-reevaluate-setting before the
frame initializations.  If it is, then we could move it to before
loading early-init.el, and that would be my favorite choice.

If that can't be done, then the second best option is to move the
package-activation to after we run custom-reevaluate-setting (so
early-init.el still can't (require 'info) because
Info-default-directory-list is unbound, but at least the problem at hand
should be fixed).


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Basil L. Contovounesios]

>> The first line in the *Messages* buffer reads
>>   Symbol’s value as variable is void: Info-default-directory-list

A knock-on effect is a near-twofold increase in my startup time, from
~0.4s to ~0.7s, but hopefully this is due only to the unintended 34
times the error is logged while activating 146 packages on my side.

Sorry I can't be of any help at this time; I haven't been keeping up
with this feature.

-- 
Basil

Re: [PATCH] Fixing package-initialize, adding early init file

[Basil L. Contovounesios]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

>>> The first line in the *Messages* buffer reads
>>>   Symbol’s value as variable is void: Info-default-directory-list
>
> A knock-on effect is a near-twofold increase in my startup time, from
> ~0.4s to ~0.7s, but hopefully this is due only to the unintended 34
> times the error is logged while activating 146 packages on my side.

Indeed, I spoke too soon - I think the slowdown was being caused by a
repeated call to package-initialize, which I now avoid by disabling
package-enable-at-startup in early-init-file.  Sorry about the noise.

-- 
Basil

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

> I think the slowdown was being caused by a
> repeated call to package-initialize, which I now avoid by disabling
> package-enable-at-startup in early-init-file.

Might it be a good idea to have `package-initialize' signal a warning if
it's
called multiple times? I hadn't considered this side effect of the change,
but at least it's easier to get users to remove a duplicate call than to add
a missing one.

Are there ever circumstances where calling `package-initialize' more than
once makes sense? Perhaps we can allow for those by introducing a
variable `package-warn-on-reinitialization' or somesuch, which would be
set to non-nil in `package-initialize', and then next time around produce a
warning, but then could be set back to nil to suppress the warning.

[-- Attachment #2: Type: text/html, Size: 1574 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[T.V Raman]

Also the documentation for variable 
package-enable-at-startup may need updating, present doc-string no
longer feels right:
package-enable-at-startup is a variable defined in `package.el'.
Its value is nil
Original value was t

Documentation:
Whether to activate installed packages when Emacs starts.
If non-nil, packages are activated after reading the init file
and before `after-init-hook'.  Activation is not done if
`user-init-file' is nil (e.g. Emacs was started with "-q").

Even if the value is nil, you can type M-x package-initialize to
activate the package system at any time.

You can customize this variable.

This variable was introduced, or its default value was changed, in
version 24.1 of Emacs.

-- 

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> Might it be a good idea to have `package-initialize' signal a warning
> if it's called multiple times? I hadn't considered this side effect of
> the change, but at least it's easier to get users to remove
> a duplicate call than to add a missing one.
>
> Are there ever circumstances where calling `package-initialize' more than
> once makes sense?

It can make sense if packages were added/removed/updated without
Emacs's knowledge.  So maybe we need something more targeted to
~/.emacs, by temporarily let-binding a package--within-init.el variable
(this would also allow us to not only emit a message but even skip the
execution of such a second call to package-initialize).


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> So maybe we need something more targeted to
> ~/.emacs, by temporarily let-binding a package--within-init.el variable
> (this would also allow us to not only emit a message but even skip the
> execution of such a second call to package-initialize).

So then if you actually do want to run `package-initialize', you would
let-bind `package--within-init.el' to nil? I feel like that's a little
odd, since it would require you to introduce a workaround even if
`package-initialize' was only called once (i.e. you set
`package-enable-at-startup' to nil in early-init.el). Whereas with
what I suggested, you'd only need to introduce a workaround if
`package-initialize' was actually called twice. (If you were concerned
about a warning being signaled after startup, we could just condition
it on `after-init-time' being nil.)

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> So then if you actually do want to run `package-initialize', you would
> let-bind `package--within-init.el' to nil? I feel like that's a little
> odd, since it would require you to introduce a workaround even if
> `package-initialize' was only called once (i.e. you set
> `package-enable-at-startup' to nil in early-init.el). Whereas with
> what I suggested, you'd only need to introduce a workaround if
> `package-initialize' was actually called twice.

Clearly, we would not want to warn and/or skip one the first call
(e.g. when the user set package-enable-at-startup to nil in early-init.el).

> (If you were concerned about a warning being signaled after startup,
> we could just condition it on `after-init-time' being nil.)

That would work as well, yes,


        Stefan

Re: [PATCH] Fixing package-initialize, adding early init file

[T.V Raman]

But that error re Info-default-directory is still
annoying. Interestingly, declaring it as a defvar in early-init.el
doesn't seem to help 
-- 

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> 1. For most Emacs users who do not program in Elisp, there should be
>    only a single init file that they would personalize, generally as
>    they find snippets of code they can cut and paste for their use.

The entire reason these solutions are being proposed is because we
want to avoid the situation where people cut and paste snippets into
their init file, and they fail to work, which is what used to happen.
See [1].

And, as Eli already pointed out, for most Emacs users there *is* only
a single init file. It's only if you want to do advanced programming
that you need to use the second one, which need not exist at all.

> it should be solved without requiring the use of a 2nd init file.

Many, many possible solutions have already been discussed and
dismissed; see [2]. It's nice to hope for a better one, but I don't
think this should be considered a blocker unless someone actually
comes up with a better solution.

Best,
Radon

[1]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[2]: https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html

Re: [PATCH] Fixing package-initialize, adding early init file

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

On Tue, Oct 10, 2017 at 3:03 PM, Radon Rosborough <radon.neon@gmail.com>
wrote:

>
> And, as Eli already pointed out, for most Emacs users there *is* only
> a single init file. It's only if you want to do advanced programming
> that you need to use the second one, which need not exist at all.
>

​That sounds fine.  When I mentioned 'package-initialize', I meant that
users of packages who would call that in their init files should not need
to have a second init file just to use packages and have them load properly.

Bob

[-- Attachment #2: Type: text/html, Size: 1359 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

> When I mentioned 'package-initialize', I meant that users of
> packages who would call that in their init files should not need to
> have a second init file just to use packages and have them load
> properly.

Indeed, you need only have a second init file if you would like to
customize `package-load-list', `package-enable-at-startup',
`package-user-dir', or other such low-level variables. (In particular,
`package-archives' can be set in the regular init file.)

In fact, with this patch, you don't even need to have any
package-related code at all in your init-file: not even
`package-initialize'. Packages can be installed, and they will "just
work", including if customizations are them are blindly pasted into
the regular init file.

Re: [PATCH] Fixing package-initialize, adding early init file

[John Wiegley]

>>>>> "RR" == Radon Rosborough <radon.neon@gmail.com> writes:

RR> In fact, with this patch, you don't even need to have any
RR> package-related code at all in your init-file: not even
RR> `package-initialize'. Packages can be installed, and they will "just
RR> work", including if customizations are them are blindly pasted into
RR> the regular init file.

And if I never use packages, I'll never have this second file, right?  Because
right now, every once in a while, I find (package-initialize) getting inserted
into my init.el, even though I never once have used packages. Still haven't
figured out what's doing that.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

> And if I never use packages, I'll never have this second file,
> right?

Correct.

> Because right now, every once in a while, I find
> (package-initialize) getting inserted into my init.el, even though I
> never once have used packages.

Literally the entire point of this patch is to stop that from
happening without sacrificing any existing functionality.

[-- Attachment #2: Type: text/html, Size: 551 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Sat, Oct 14, 2017 at 1:49 AM, Radon Rosborough <radon.neon@gmail.com>
wrote:
>
> > And if I never use packages, I'll never have this second file,
> > right?
>
> Correct.

A follow up: if I do use packages but never mess with its "advanced
options"
(or even customize by that matter), will I also never have this second file?

(Excuse me if the question is ignorant in the "advanced options" part, I
remember
reading something about that in these very big threads, about configuring
the
package's location and such.)

João

[-- Attachment #2: Type: text/html, Size: 864 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

> A follow up: if I do use packages but never mess with its "advanced
> options" (or even customize by that matter), will I also never have
> this second file?

Correct. If you don't have to customize `package-user-dir' or
`package-load-list', and you don't need to stop package.el from
loading packages that you already installed, then there is no need for
the early init file.

Note also that with this patch, unlike in the current situation, Emacs
doesn't do anything automatically. No files are created or modified by
Emacs.

[-- Attachment #2: Type: text/html, Size: 745 bytes --]

Re: [PATCH] Fixing package-initialize, adding early init file

[João Távora]

Radon Rosborough <radon.neon@gmail.com> writes:

>> A follow up: if I do use packages but never mess with its "advanced
>> options" (or even customize by that matter), will I also never have
>> this second file?
>
> Correct. If you don't have to customize `package-user-dir' or
> `package-load-list', and you don't need to stop package.el from
> loading packages that you already installed, then there is no need for
> the early init file.
>
> Note also that with this patch, unlike in the current situation, Emacs
> doesn't do anything automatically. No files are created or modified by
> Emacs.

Thank you. Sounds reasonable, then.

João

Re: [PATCH] Fixing package-initialize, adding early init file

[Stefan Monnier]

> A follow up: if I do use packages but never mess with its "advanced
> options" (or even customize by that matter), will I also never have
> this second file?

The core of the change is to call package-initialize *before* loading
~/.emacs.  So you'll need the other file in the specific case where you
need to do something *before* the call to package-initialize.


        Stefan

Reload file from disk

[Florian Weimer]

Some time ago, it was possible to reload the current buffer from disk
using C-x C-f RET.  This was changed to run dired instead.  The
argument at the time was you could use C-x C-f M-n RET to reload the
current buffer, and that dired was more important.  The behavior of
M-n is documented for find-file at al.:

| but the visited file name is available through the minibuffer
| history: type M-n to pull it into the minibuffer.

But over the years, C-x C-f M-n RET started doing more and more
bizarre things.  It now inserts the file name at point in the buffer,
which is particularly annoying if you try to reload a ChangeLog file.
And I just noticed that when on a domain name such as example.com, it
will apparently attempt to ping that host.

Can we please make M-n behave as documented in this context, or even
better, revert to the old C-x C-f RET behavior?  One could always
browse the current directory using C-x C-f . RET, which is why never
understood the motivation behind that change.

Re: Reload file from disk

[Eli Zaretskii]

> From: Florian Weimer <fw@deneb.enyo.de>
> Date: Sat, 11 Nov 2017 10:17:18 +0100
> 
> | but the visited file name is available through the minibuffer
> | history: type M-n to pull it into the minibuffer.
> 
> But over the years, C-x C-f M-n RET started doing more and more
> bizarre things.  It now inserts the file name at point in the buffer,
> which is particularly annoying if you try to reload a ChangeLog file.

It does?  I cannot reproduce this in "emacs -Q".  Could it be that you
have some optional package activated which does that?  If not, can you
show a recipe starting from "emacs -Q" that reproduces the behavior
you describe?

> And I just noticed that when on a domain name such as example.com, it
> will apparently attempt to ping that host.

This I can reproduce, and you can disable it by customizing
file-name-at-point-functions to a nil value.

> Can we please make M-n behave as documented in this context, or even
> better, revert to the old C-x C-f RET behavior?  One could always
> browse the current directory using C-x C-f . RET, which is why never
> understood the motivation behind that change.

The previous behavior of "C-x C-f RET" was considered confusing and
not very useful by many users, so I don't think we want to go back
there.

Re: Reload file from disk

[Eli Zaretskii]

> Date: Sat, 11 Nov 2017 12:29:24 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > But over the years, C-x C-f M-n RET started doing more and more
> > bizarre things.  It now inserts the file name at point in the buffer,
> > which is particularly annoying if you try to reload a ChangeLog file.
> 
> It does?  I cannot reproduce this in "emacs -Q".

Sorry, I see that I misunderstood your somewhat ambiguous wording.  I
see what you meant now, and the answer is the same: customize
file-name-at-point-functions to nil, and both the behaviors that
annoyed you will be gone.

Re: Reload file from disk

[Florian Weimer]

* Eli Zaretskii:

>> Date: Sat, 11 Nov 2017 12:29:24 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > But over the years, C-x C-f M-n RET started doing more and more
>> > bizarre things.  It now inserts the file name at point in the buffer,
>> > which is particularly annoying if you try to reload a ChangeLog file.
>> 
>> It does?  I cannot reproduce this in "emacs -Q".
>
> Sorry, I see that I misunderstood your somewhat ambiguous wording.  I
> see what you meant now, and the answer is the same: customize
> file-name-at-point-functions to nil, and both the behaviors that
> annoyed you will be gone.

The documentation for find-file et al. still could use some updating
and a reference to file-name-at-point-functions.  For some reason, I
was unable to find the latter.

(I still find the default behavior thoroughly bizarre, but that's
probably a matter of taste.)

Re: Reload file from disk

[Eli Zaretskii]

> From: Florian Weimer <fw@deneb.enyo.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 11 Nov 2017 11:54:57 +0100
> 
> > Sorry, I see that I misunderstood your somewhat ambiguous wording.  I
> > see what you meant now, and the answer is the same: customize
> > file-name-at-point-functions to nil, and both the behaviors that
> > annoyed you will be gone.
> 
> The documentation for find-file et al. still could use some updating
> and a reference to file-name-at-point-functions.  For some reason, I
> was unable to find the latter.

You are absolutely right, which is why I just updated the
documentation, both the doc strings and the manual, to mention this.
I was astonished to find out that this feature was first introduced in
Emacs 23.2(!), but never mentioned even in NEWS.

> (I still find the default behavior thoroughly bizarre, but that's
> probably a matter of taste.)

I'm with you, but it looks like we are the minority nowadays.

Thanks for bringing up this awful gap in documentation.

Re: Reload file from disk

[Ken Olum]

   From: Eli Zaretskii <eliz@gnu.org>
   Date: Sat, 11 Nov 2017 13:55:26 +0200

   > From: Florian Weimer <fw@deneb.enyo.de>

   > (I still find the default behavior thoroughly bizarre, but that's
   > probably a matter of taste.)

   I'm with you, but it looks like we are the minority nowadays.

I redefined find-file-read-args in my init file to reinstate the old
behavior when the new came out.  How about a variable to control this
for the possibly large minority that prefer the old behavior?

                                          Ken

Re: Reload file from disk

[Eli Zaretskii]

> From: Ken Olum <kdo@cosmos.phy.tufts.edu>
> Date: Sat, 11 Nov 2017 08:45:04 -0500
> 
>    > (I still find the default behavior thoroughly bizarre, but that's
>    > probably a matter of taste.)
> 
>    I'm with you, but it looks like we are the minority nowadays.
> 
> I redefined find-file-read-args in my init file to reinstate the old
> behavior when the new came out.  How about a variable to control this
> for the possibly large minority that prefer the old behavior?

There's already a variable: file-name-at-point-functions.  It just was
undocumented until now.  Given that it's documented, do we still need
a separate knob?

Re: Reload file from disk

[Ken Olum]

   From: Eli Zaretskii <eliz@gnu.org>
   Date: Sat, 11 Nov 2017 16:11:34 +0200

   There's already a variable: file-name-at-point-functions.  It just was
   undocumented until now.  Given that it's documented, do we still need
   a separate knob?

Sorry for the confusion.  I think we're discussing different things
here.  My goal was to have the default for find-file (without typing
M-n) be the current file, so that C-x C-f RET would reload the buffer.
I think file-name-at-point-functions affects only the question of what
you get when you type M-n.

If there are enough people who would like C-x C-f RET to reload the
buffer, perhaps there should be a variable for that.  But perhaps
there are not.  (This has little consequence for me personally, since
I already worked around it a different way.)

                                        Ken

RE: Reload file from disk

[Drew Adams]

>  My goal was to have the default for find-file (without typing
> M-n) be the current file, so that C-x C-f RET would reload the buffer.

FWIW, y'all should be talking about "finding" a Lisp file
(in the sense of `find-file'), not "loading" a Lisp file
(in the sense of `load-file' and `load-library').

I was confused for a while, because I thought you actually
meant loading (evaluating) the file.  And for loading,
`M-x load-file' _does_ use the current buffer's file as
the default.

Re: Reload file from disk

[Andreas Schwab]

On Nov 11 2017, Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Sat, 11 Nov 2017 12:29:24 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > But over the years, C-x C-f M-n RET started doing more and more
>> > bizarre things.  It now inserts the file name at point in the buffer,
>> > which is particularly annoying if you try to reload a ChangeLog file.
>> 
>> It does?  I cannot reproduce this in "emacs -Q".
>
> Sorry, I see that I misunderstood your somewhat ambiguous wording.  I
> see what you meant now, and the answer is the same: customize
> file-name-at-point-functions to nil, and both the behaviors that
> annoyed you will be gone.

Even better, bind revert-buffer to a key.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Reload file from disk

[Noam Postavsky]

On Sat, Nov 11, 2017 at 4:17 AM, Florian Weimer <fw@deneb.enyo.de> wrote:
> Some time ago, it was possible to reload the current buffer from disk
> using C-x C-f RET.  This was changed to run dired instead.  The
> argument at the time was you could use C-x C-f M-n RET to reload the
> current buffer, and that dired was more important.

I usually use C-x C-v RET for this.
Or if I want to reload because the file changed externally, SPC r
(though the buffer must be in unmodified state for that).

Re: Reload file from disk

[Joost Kremers]

On Sat, Nov 11 2017, Florian Weimer wrote:
> Some time ago, it was possible to reload the current buffer from 
> disk
> using C-x C-f RET.  This was changed to run dired instead.

Try C-x C-v (`find-alternate-file') instead: it defaults to the 
file currently being visited by the buffer, so you just need to 
type RET to reload it.


-- 
Joost Kremers
Life has its moments

Re: Reload file from disk

[Tom Tromey]

>>>>> "Joost" == Joost Kremers <joostkremers@fastmail.fm> writes:

Joost> On Sat, Nov 11 2017, Florian Weimer wrote:
>> Some time ago, it was possible to reload the current buffer from
>> disk
>> using C-x C-f RET.  This was changed to run dired instead.

Joost> Try C-x C-v (`find-alternate-file') instead: it defaults to the file
Joost> currently being visited by the buffer, so you just need to type RET to
Joost> reload it.

This isn't always as nice as the old behavior of C-x C-f RET because it
moves point to the start of the buffer.

Tom

Deleting old `:version` from defcustoms (was: master b76cdd0: Delete libraries obsolete since 23.1 and 23.2)

[Stefan Monnier]

>     Delete libraries obsolete since 23.1 and 23.2

Thanks Stefan.

I did a quick `grep` to see the functions/vars that were obsoleted in
the same time frame, and I saw instead a deluge of

    :version "23.1"

If we consider those thingies old enough to remove their obsolete
functions, shouldn't we also remove the corresponding `:version`
thingies from `defcustom`s?

I see:
- 1 defcustom with a :version of 19.29
- >200 defcustoms with a :version of 20.x
- a bit less than 200 defcustoms with a :version of 21.x
- >400 defcustoms with a :version of 22.x
- >200 defcustoms with a :version of 23.x


-- Stefan

Re: Deleting old `:version` from defcustoms (was: master b76cdd0: Delete libraries obsolete since 23.1 and 23.2)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I did a quick `grep` to see the functions/vars that were obsoleted in
> the same time frame, and I saw instead a deluge of
>
>     :version "23.1"
>
> If we consider those thingies old enough to remove their obsolete
> functions, shouldn't we also remove the corresponding `:version`
> thingies from `defcustom`s?

There are also these two libraries:

./lisp/obsolete/fast-lock.el9:;; Obsolete-since: 22.1
./lisp/obsolete/lazy-lock.el9:;; Obsolete-since: 22.1

For some reason they weren't removed in 27.1, when most other libraries
obsolete since 22.x were deleted.  This made me wonder if there was any
particular reason for their non-removal.  Does anyone know?  And could
we delete them now?

Best regards,
Stefan Kangas

[PATCH] Remove obsolete fast-lock and lazy-lock libraries (was: Re: Deleting old `:version` from defcustoms)

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

Stefan Kangas <stefankangas@gmail.com> writes:

> There are also these two libraries:
>
> ./lisp/obsolete/fast-lock.el9:;; Obsolete-since: 22.1
> ./lisp/obsolete/lazy-lock.el9:;; Obsolete-since: 22.1
>
> For some reason they weren't removed in 27.1, when most other libraries
> obsolete since 22.x were deleted.  This made me wonder if there was any
> particular reason for their non-removal.  Does anyone know?  And could
> we delete them now?

I have attached a patch to remove them below.

One thing to note is that I've kept the support for
'font-lock-support-mode' set to nil since Alan Mackenzie indicated
that this could be useful for debugging purposes here:
https://lists.gnu.org/archive/html/emacs-devel/2011-10/msg00748.html
Perhaps it should be changed into a defvar, though. Or maybe it is no
longer useful. But that could in any case be done separately, I think.

(In case anyone is wondering, the motivation here is that these
libraries are the only reason why some variables obsolete since 23.1
can't yet be removed.)

Any comments or objections?

Best regards,
Stefan Kangas

[-- Attachment #2: 0001-Remove-obsolete-fast-lock-and-lazy-lock-libraries.patch --]
[-- Type: text/x-patch, Size: 102437 bytes --]

From 7ecfac4412c0f1f126681fc9c99153ab5807e106 Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Fri, 7 Aug 2020 17:17:08 +0200
Subject: [PATCH] Remove obsolete fast-lock and lazy-lock libraries

These libraries have been obsolete since 22.1.

* lisp/obsolete/fast-lock.el:
* lisp/obsolete/lazy-lock.el: Remove libraries obsolete since 22.1.

* lisp/font-lock.el (font-lock-turn-off-thing-lock)
(font-lock-after-fontify-buffer)
(font-lock-after-unfontify-buffer)
(font-lock-default-fontify-buffer)
(font-lock-default-unfontify-buffer, font-lock-support-mode):
* lisp/mail/rmail.el (rmail-variables):
* lisp/ps-print.el (ps-print-ensure-fontified): Remove support for
'fast-lock' and 'lazy-lock'.

* lisp/font-core.el (font-lock-defaults):
* lisp/font-lock.el (font-lock-inhibit-thing-lock)
(font-lock-keywords, top-level):
* lisp/progmodes/antlr-mode.el (top-level):
* lisp/progmodes/cc-fonts.el (c-font-lock-declarations):
* lisp/progmodes/cperl-mode.el (top-level):
* src/buffer.c (Fset_buffer_modified_p): Doc fixes to remove
references to 'fast-lock' and 'lazy-lock'.

* etc/NEWS: Announce that they have been removed.
---
 etc/NEWS                     |    5 +
 lisp/font-core.el            |    2 +-
 lisp/font-lock.el            |   83 +--
 lisp/mail/rmail.el           |    3 +-
 lisp/obsolete/fast-lock.el   |  841 ---------------------------
 lisp/obsolete/lazy-lock.el   | 1053 ----------------------------------
 lisp/progmodes/antlr-mode.el |    3 -
 lisp/progmodes/cc-fonts.el   |    2 +-
 lisp/progmodes/cperl-mode.el |    2 -
 lisp/ps-print.el             |    5 +-
 src/buffer.c                 |   14 +-
 11 files changed, 41 insertions(+), 1972 deletions(-)
 delete mode 100644 lisp/obsolete/fast-lock.el
 delete mode 100644 lisp/obsolete/lazy-lock.el

diff --git a/etc/NEWS b/etc/NEWS
index 850b166069..d80ef6865e 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -771,6 +771,11 @@ have now been removed.
 ** Some libraries obsolete since Emacs 23 have been removed:
 'ledit.el', 'lmenu.el', 'lucid.el and 'old-whitespace.el'.
 
+---
+** The 'lazy-lock' and 'fast-lock' libraries have been removed.
+They have been obsolete since 22.1.  Use 'jit-lock' instead (the
+default).
+
 \f
 * Lisp Changes in Emacs 28.1
 
diff --git a/lisp/font-core.el b/lisp/font-core.el
index 098253eb16..344d7ee2ad 100644
--- a/lisp/font-core.el
+++ b/lisp/font-core.el
@@ -63,7 +63,7 @@ font-lock-defaults
 `font-lock-syntactic-keywords' and those for buffer-specialized fontification
 functions, `font-lock-fontify-buffer-function',
 `font-lock-unfontify-buffer-function', `font-lock-fontify-region-function',
-`font-lock-unfontify-region-function', and `font-lock-inhibit-thing-lock'.")
+`font-lock-unfontify-region-function'.")
 ;;;###autoload
 (put 'font-lock-defaults 'risky-local-variable t)
 (make-variable-buffer-local 'font-lock-defaults)
diff --git a/lisp/font-lock.el b/lisp/font-lock.el
index 5cda4a693d..f5adc5b768 100644
--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -47,9 +47,7 @@
 ;;
 ;; Fontification for a particular mode may be available in a number of levels
 ;; of decoration.  The higher the level, the more decoration, but the more time
-;; it takes to fontify.  See the variable `font-lock-maximum-decoration', and
-;; also the variable `font-lock-maximum-size'.  Support modes for Font Lock
-;; mode can be used to speed up Font Lock mode.  See `font-lock-support-mode'.
+;; it takes to fontify.  See the variable `font-lock-maximum-decoration'.
 \f
 ;;; How Font Lock mode fontifies:
 
@@ -471,8 +469,7 @@ font-lock-keywords
 These regular expressions can match text which spans lines,
 although it is better to avoid it if possible since updating them
 while editing text is slower, and it is not guaranteed to be
-always correct when using support modes like jit-lock or
-lazy-lock.
+always correct when using jit-lock.
 
 This variable is set by major modes via the variable
 `font-lock-defaults'.  Be careful when composing regexps for this
@@ -608,8 +605,7 @@ font-lock-unfontify-region-function
 
 (defvar font-lock-inhibit-thing-lock nil
   "List of Font Lock mode related modes that should not be turned on.
-Currently, valid mode names are `fast-lock-mode', `jit-lock-mode' and
-`lazy-lock-mode'.  This is normally set via `font-lock-defaults'.")
+This variable is now unused.")
 (make-obsolete-variable 'font-lock-inhibit-thing-lock nil "25.1")
 
 (defvar-local font-lock-multiline nil
@@ -879,25 +875,19 @@ font-lock-remove-keywords
 (defcustom font-lock-support-mode 'jit-lock-mode
   "Support mode for Font Lock mode.
 Support modes speed up Font Lock mode by being choosy about when fontification
-occurs.  The default support mode, Just-in-time Lock mode (symbol
-`jit-lock-mode'), is recommended.
-
-Other, older support modes are Fast Lock mode (symbol `fast-lock-mode') and
-Lazy Lock mode (symbol `lazy-lock-mode').  See those modes for more info.
-However, they are no longer recommended, as Just-in-time Lock mode is better.
+occurs.  The only supported mode is Just-in-time Lock mode (symbol
+`jit-lock-mode').
 
 If nil, means support for Font Lock mode is never performed.
 If a symbol, use that support mode.
 If a list, each element should be of the form (MAJOR-MODE . SUPPORT-MODE),
 where MAJOR-MODE is a symbol or t (meaning the default).  For example:
- ((c-mode . fast-lock-mode) (c++-mode . fast-lock-mode) (t . lazy-lock-mode))
-means that Fast Lock mode is used to support Font Lock mode for buffers in C or
-C++ modes, and Lazy Lock mode is used to support Font Lock mode otherwise.
+ ((c-mode . nil) (t . jit-lock-mode))
+means that no font locking is used for buffers in C or, and Jit
+Lock mode is used to support Font Lock mode otherwise.
 
 The value of this variable is used when Font Lock mode is turned on."
   :type '(choice (const :tag "none" nil)
-		 (const :tag "fast lock" fast-lock-mode)
-		 (const :tag "lazy lock" lazy-lock-mode)
 		 (const :tag "jit lock" jit-lock-mode)
 		 (repeat :menu-tag "mode specific" :tag "mode specific"
 			 :value ((t . jit-lock-mode))
@@ -907,28 +897,15 @@ font-lock-support-mode
 				      (symbol :tag "name"))
 			       (radio :tag "Support"
 				      (const :tag "none" nil)
-				      (const :tag "fast lock" fast-lock-mode)
-				      (const :tag "lazy lock" lazy-lock-mode)
 				      (const :tag "JIT lock" jit-lock-mode)))
 			 ))
   :version "21.1"
   :group 'font-lock)
 
-(defvar fast-lock-mode)
-(defvar lazy-lock-mode)
 (defvar jit-lock-mode)
 
-(declare-function fast-lock-after-fontify-buffer "fast-lock")
-(declare-function fast-lock-after-unfontify-buffer "fast-lock")
-(declare-function fast-lock-mode "fast-lock")
-(declare-function lazy-lock-after-fontify-buffer "lazy-lock")
-(declare-function lazy-lock-after-unfontify-buffer "lazy-lock")
-(declare-function lazy-lock-mode "lazy-lock")
-
 (defun font-lock-turn-on-thing-lock ()
   (pcase (font-lock-value-in-major-mode font-lock-support-mode)
-    ('fast-lock-mode (fast-lock-mode t))
-    ('lazy-lock-mode (lazy-lock-mode t))
     ('jit-lock-mode
      ;; Prepare for jit-lock
      (remove-hook 'after-change-functions
@@ -954,37 +931,29 @@ font-lock-turn-on-thing-lock
                nil t))))
 
 (defun font-lock-turn-off-thing-lock ()
-  (cond ((bound-and-true-p fast-lock-mode)
-	 (fast-lock-mode -1))
-	((bound-and-true-p jit-lock-mode)
+  (cond ((bound-and-true-p jit-lock-mode)
 	 (jit-lock-unregister 'font-lock-fontify-region)
 	 ;; Reset local vars to the non-jit-lock case.
-	 (kill-local-variable 'font-lock-fontify-buffer-function))
-	((bound-and-true-p lazy-lock-mode)
-	 (lazy-lock-mode -1))))
+	 (kill-local-variable 'font-lock-fontify-buffer-function))))
 
 (defun font-lock-after-fontify-buffer ()
-  (cond ((bound-and-true-p fast-lock-mode)
-	 (fast-lock-after-fontify-buffer))
-	;; Useless now that jit-lock intercepts font-lock-fontify-buffer.  -sm
-	;; (jit-lock-mode
-	;;  (jit-lock-after-fontify-buffer))
-	((bound-and-true-p lazy-lock-mode)
-	 (lazy-lock-after-fontify-buffer))))
+  (declare (obsolete "does nothing." "28.1"))
+  ;; Useless now that jit-lock intercepts font-lock-fontify-buffer.  -sm
+  ;; (jit-lock-mode
+  ;;  (jit-lock-after-fontify-buffer))
+  )
 
 (defun font-lock-after-unfontify-buffer ()
-  (cond ((bound-and-true-p fast-lock-mode)
-	 (fast-lock-after-unfontify-buffer))
-	;; Useless as well.  It's only called when:
-	;; - turning off font-lock: it does not matter if we leave spurious
-	;;   `fontified' text props around since jit-lock-mode is also off.
-	;; - font-lock-default-fontify-buffer fails: this is not run
-	;;   any more anyway.   -sm
-	;;
-	;; (jit-lock-mode
-	;;  (jit-lock-after-unfontify-buffer))
-	((bound-and-true-p lazy-lock-mode)
-	 (lazy-lock-after-unfontify-buffer))))
+  (declare (obsolete "does nothing." "28.1"))
+  ;; Useless as well.  It's only called when:
+  ;; - turning off font-lock: it does not matter if we leave spurious
+  ;;   `fontified' text props around since jit-lock-mode is also off.
+  ;; - font-lock-default-fontify-buffer fails: this is not run
+  ;;   any more anyway.   -sm
+  ;;
+  ;; (jit-lock-mode
+  ;;  (jit-lock-after-unfontify-buffer))
+  )
 
 ;;; End of Font Lock Support mode.
 \f
@@ -1141,7 +1110,6 @@ font-lock-default-fontify-buffer
 	    (save-excursion
 	      (save-match-data
 		(font-lock-fontify-region (point-min) (point-max) verbose)
-		(font-lock-after-fontify-buffer)
 		(setq font-lock-fontified t)))
 	  ;; We don't restore the old fontification, so it's best to unfontify.
 	  (quit (font-lock-unfontify-buffer)))))))
@@ -1152,7 +1120,6 @@ font-lock-default-unfontify-buffer
   (save-restriction
     (widen)
     (font-lock-unfontify-region (point-min) (point-max))
-    (font-lock-after-unfontify-buffer)
     (setq font-lock-fontified nil)))
 
 (defvar font-lock-dont-widen nil
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index 44cde7cb5a..bbc220cfd1 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -1518,8 +1518,7 @@ rmail-variables
 	'(rmail-font-lock-keywords
 	  t t nil nil
 	  (font-lock-maximum-size . nil)
-          (font-lock-dont-widen . t)
-	  (font-lock-inhibit-thing-lock . (lazy-lock-mode fast-lock-mode))))
+          (font-lock-dont-widen . t)))
   (make-local-variable 'require-final-newline)
   (setq require-final-newline nil)
   (make-local-variable 'version-control)
diff --git a/lisp/obsolete/fast-lock.el b/lisp/obsolete/fast-lock.el
deleted file mode 100644
index 9e0198dd59..0000000000
--- a/lisp/obsolete/fast-lock.el
+++ /dev/null
@@ -1,841 +0,0 @@
-;;; fast-lock.el --- automagic text properties caching for fast Font Lock mode
-
-;; Copyright (C) 1994-1998, 2001-2020 Free Software Foundation, Inc.
-
-;; Author: Simon Marshall <simon@gnu.org>
-;; Maintainer: emacs-devel@gnu.org
-;; Keywords: faces files
-;; Version: 3.14
-;; Obsolete-since: 22.1
-
-;; This file is part of GNU Emacs.
-
-;; GNU Emacs is free software: you can redistribute it and/or modify
-;; it under the terms of the GNU General Public License as published by
-;; the Free Software Foundation, either version 3 of the License, or
-;; (at your option) any later version.
-
-;; GNU Emacs is distributed in the hope that it will be useful,
-;; but WITHOUT ANY WARRANTY; without even the implied warranty of
-;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-;; GNU General Public License for more details.
-
-;; You should have received a copy of the GNU General Public License
-;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
-
-;;; Commentary:
-
-;; Fast Lock mode is a Font Lock support mode.
-;; It makes visiting a file in Font Lock mode faster by restoring its face text
-;; properties from automatically saved associated Font Lock cache files.
-;;
-;; See caveats and feedback below.
-;; See also the lazy-lock package.  (But don't use the two at the same time!)
-
-;; Installation:
-;;
-;; Put in your ~/.emacs:
-;;
-;; (setq font-lock-support-mode 'fast-lock-mode)
-;;
-;; Start up a new Emacs and use font-lock as usual (except that you can use the
-;; so-called "gaudier" fontification regexps on big files without frustration).
-;;
-;; When you visit a file (which has `font-lock-mode' enabled) that has a
-;; corresponding Font Lock cache file associated with it, the Font Lock cache
-;; will be loaded from that file instead of being generated by Font Lock code.
-
-;; Caveats:
-;;
-;; A cache will be saved when visiting a compressed file using crypt++, but not
-;; be read.  This is a "feature"/"consequence"/"bug" of crypt++.
-;;
-;; Version control packages are likely to stamp all over file modification
-;; times.  Therefore the act of checking out may invalidate a cache.
-\f
-;; History:
-;;
-;; 0.02--1.00:
-;; - Changed name from turbo-prop to fast-lock.  Automagic for font-lock only
-;; - Made `fast-lock-mode' a minor mode, like G. Dinesh Dutt's fss-mode
-;; 1.00--1.01:
-;; - Turn on `fast-lock-mode' only if `buffer-file-name' or `interactive-p'
-;; - Made `fast-lock-file-name' use `buffer-name' if `buffer-file-name' is nil
-;; - Moved save-all conditions to `fast-lock-save-cache'
-;; - Added `fast-lock-save-text-properties' to `kill-buffer-hook'
-;; 1.01--2.00: complete rewrite---not worth the space to document
-;; - Changed structure of text properties cache and threw out file mod checks
-;; 2.00--2.01:
-;; - Made `condition-case' forms understand `quit'.
-;; - Made `fast-lock' require `font-lock'
-;; - Made `fast-lock-cache-name' chase links (from Ben Liblit)
-;; 2.01--3.00:
-;; - Changed structure of cache to include `font-lock-keywords' (from rms)
-;; - Changed `fast-lock-cache-mechanisms' to `fast-lock-cache-directories'
-;; - Removed `fast-lock-read-others'
-;; - Made `fast-lock-read-cache' ignore cache owner
-;; - Made `fast-lock-save-cache-external' create cache directory
-;; - Made `fast-lock-save-cache-external' save `font-lock-keywords'
-;; - Made `fast-lock-cache-data' check `font-lock-keywords'
-;; 3.00--3.01: incorporated port of 2.00 to Lucid, made by Barry Warsaw
-;; - Package now provides itself
-;; - Lucid: Use `font-lock-any-extents-p' for `font-lock-any-properties-p'
-;; - Lucid: Use `list-faces' for `face-list'
-;; - Lucid: Added `set-text-properties'
-;; - Lucid: Made `turn-on-fast-lock' pass 1 not t to `fast-lock-mode'
-;; - Removed test for `fast-lock-mode' from `fast-lock-read-cache'
-;; - Lucid: Added Lucid-specific `fast-lock-get-face-properties'
-;; 3.01--3.02: now works with Lucid Emacs, thanks to Barry Warsaw
-;; - Made `fast-lock-cache-name' map ":" to ";" for OS/2 (from Serganova Vera)
-;; - Made `fast-lock-cache-name' use abbreviated file name (from Barry Warsaw)
-;; - Lucid: Separated handlers for `error' and `quit' for `condition-case'
-;; 3.02--3.03:
-;; - Changed `fast-lock-save-cache-external' to `fast-lock-save-cache-data'
-;; - Lucid: Added Lucid-specific `fast-lock-set-face-properties'
-;; 3.03--3.04:
-;; - Corrected `subrp' test of Lucid code
-;; - Replaced `font-lock-any-properties-p' with `text-property-not-all'
-;; - Lucid: Made `fast-lock-set-face-properties' put `text-prop' on extents
-;; - Made `fast-lock-cache-directories' a regexp alist (from Colin Rafferty)
-;; - Made `fast-lock-cache-directory' to return a usable cache file directory
-;; 3.04--3.05:
-;; - Lucid: Fix for XEmacs 19.11 `text-property-not-all'
-;; - Replaced `subrp' test of Lucid code with `emacs-version' `string-match'
-;; - Made `byte-compile-warnings' omit `unresolved' on compilation
-;; - Made `fast-lock-save-cache-data' use a buffer (from Rick Sladkey)
-;; - Reverted to old `fast-lock-get-face-properties' (from Rick Sladkey)
-;; 3.05--3.06: incorporated hack of 3.03, made by Jonathan Stigelman (Stig)
-;; - Reverted to 3.04 version of `fast-lock-get-face-properties'
-;; - XEmacs: Removed `list-faces' `defalias'
-;; - Made `fast-lock-mode' and `turn-on-fast-lock' succeed `autoload' cookies
-;; - Added `fast-lock-submit-bug-report'
-;; - Renamed `fast-lock-save-size' to `fast-lock-minimum-size'
-;; - Made `fast-lock-save-cache' output a message if no save ever attempted
-;; - Made `fast-lock-save-cache-data' output a message if save attempted
-;; - Made `fast-lock-cache-data' output a message if load attempted
-;; - Made `fast-lock-save-cache-data' do `condition-case' not `unwind-protect'
-;; - Made `fast-lock-save-cache' and `fast-lock-read-cache' return nothing
-;; - Made `fast-lock-save-cache' check `buffer-modified-p' (Stig)
-;; - Added `fast-lock-save-events'
-;; - Added `fast-lock-after-save-hook' to `after-save-hook' (Stig)
-;; - Added `fast-lock-kill-buffer-hook' to `kill-buffer-hook'
-;; - Changed `fast-lock-save-caches' to `fast-lock-kill-emacs-hook'
-;; - Added `fast-lock-kill-emacs-hook' to `kill-emacs-hook'
-;; - Made `fast-lock-save-cache' check `verify-visited-file-modtime' (Stig)
-;; - Made `visited-file-modtime' be the basis of the timestamp (Stig)
-;; - Made `fast-lock-save-cache-1' and `fast-lock-cache-data' use/reformat it
-;; - Added `fast-lock-cache-filename' to keep track of the cache file name
-;; - Added `fast-lock-after-fontify-buffer'
-;; - Added `fast-lock-save-faces' list of faces to save (idea from Stig/Tibor)
-;; - Made `fast-lock-get-face-properties' functions use it
-;; - XEmacs: Made `fast-lock-set-face-properties' do extents the Font Lock way
-;; - XEmacs: Removed fix for `text-property-not-all' (19.11 support dropped)
-;; - Made `fast-lock-mode' ensure `font-lock-mode' is on
-;; - Made `fast-lock-save-cache' do `cdr-safe' not `cdr' (from Dave Foster)
-;; - Made `fast-lock-save-cache' do `set-buffer' first (from Dave Foster)
-;; - Made `fast-lock-save-cache' loop until saved or quit (from Georg Nikodym)
-;; - Made `fast-lock-cache-data' check `buffer-modified-p'
-;; - Made `fast-lock-cache-data' do `font-lock-compile-keywords' if necessary
-;; - XEmacs: Made `font-lock-compile-keywords' `defalias'
-;; 3.06--3.07:
-;; - XEmacs: Add `fast-lock-after-fontify-buffer' to the Font Lock hook
-;; - Made `fast-lock-cache-name' explain the use of `directory-abbrev-alist'
-;; - Made `fast-lock-mode' use `buffer-file-truename' not `buffer-file-name'
-;; 3.07--3.08:
-;; - Made `fast-lock-read-cache' set `fast-lock-cache-filename'
-;; 3.08--3.09:
-;; - Made `fast-lock-save-cache' cope if `fast-lock-minimum-size' is a list
-;; - Made `fast-lock-mode' respect the value of `font-lock-inhibit-thing-lock'
-;; - Added `fast-lock-after-unfontify-buffer'
-;; 3.09--3.10:
-;; - Rewrite for Common Lisp macros
-;; - Made fast-lock.el barf on a crap 8+3 pseudo-OS (Eli Zaretskii help)
-;; - XEmacs: Made `add-minor-mode' succeed `autoload' cookie
-;; - XEmacs: Made `fast-lock-save-faces' default to `font-lock-face-list'
-;; - Made `fast-lock-save-cache' use `font-lock-value-in-major-mode'
-;; - Wrap with `save-buffer-state' (Ray Van Tassle report)
-;; - Made `fast-lock-mode' wrap `font-lock-support-mode'
-;; 3.10--3.11:
-;; - Made `fast-lock-get-face-properties' cope with face lists
-;; - Added `fast-lock-verbose'
-;; - XEmacs: Add `font-lock-value-in-major-mode' if necessary
-;; - Removed `fast-lock-submit-bug-report' and bade farewell
-;; 3.11--3.12:
-;; - Added Custom support (Hrvoje Nikšić help)
-;; - Made `save-buffer-state' wrap `inhibit-point-motion-hooks'
-;; - Made `fast-lock-cache-data' simplify calls of `font-lock-compile-keywords'
-;; 3.12--3.13:
-;; - Removed `byte-*' variables from `eval-when-compile' (Erik Naggum hint)
-;; - Changed structure of cache to include `font-lock-syntactic-keywords'
-;; - Made `fast-lock-save-cache-1' save syntactic fontification data
-;; - Made `fast-lock-cache-data' take syntactic fontification data
-;; - Added `fast-lock-get-syntactic-properties'
-;; - Renamed `fast-lock-set-face-properties' to `fast-lock-add-properties'
-;; - Made `fast-lock-add-properties' add syntactic and face fontification data
-;; 3.13--3.14:
-;; - Made `fast-lock-cache-name' cope with `windowsnt' (Geoff Voelker fix)
-;; - Made `fast-lock-verbose' use `other' widget (Andreas Schwab fix)
-;; - Used `with-temp-message' where possible to make messages temporary.
-\f
-;;; Code:
-
-(require 'font-lock)
-
-(declare-function msdos-long-file-names "msdos.c")
-
-;; Make sure fast-lock.el is supported.
-(if (and (eq system-type 'ms-dos) (not (msdos-long-file-names)))
-    (error "`fast-lock' was written for long file name systems"))
-
-(defvar font-lock-face-list)
-
-(eval-when-compile
- ;; We use this to preserve or protect things when modifying text properties.
- (defmacro save-buffer-state (varlist &rest body)
-   "Bind variables according to VARLIST and eval BODY restoring buffer state."
-   `(let* (,@(append varlist
-                     '((modified (buffer-modified-p)) (buffer-undo-list t)
-                       (inhibit-read-only t) (inhibit-point-motion-hooks t)
-                       (inhibit-modification-hooks t)
-                       deactivate-mark buffer-file-name buffer-file-truename)))
-     ,@body
-     (when (and (not modified) (buffer-modified-p))
-       (set-buffer-modified-p nil))))
- (put 'save-buffer-state 'lisp-indent-function 1)
- ;;
- ;; We use this to verify that a face should be saved.
- (defmacro fast-lock-save-facep (face)
-   "Return non-nil if FACE is one of `fast-lock-save-faces'."
-   `(or (null fast-lock-save-faces)
-     (if (symbolp ,face)
-         (memq ,face fast-lock-save-faces)
-         (let ((faces ,face))
-           (while (unless (memq (car faces) fast-lock-save-faces)
-                    (setq faces (cdr faces))))
-           faces)))))
-
-(defgroup fast-lock nil
-  "Font Lock support mode to cache fontification."
-  :load 'fast-lock
-  :group 'font-lock)
-
-(defvar fast-lock-mode nil)		; Whether we are turned on.
-(defvar fast-lock-cache-timestamp nil)	; For saving/reading.
-(defvar fast-lock-cache-filename nil)	; For deleting.
-\f
-;; User Variables:
-
-(defcustom fast-lock-minimum-size 25600
-  "Minimum size of a buffer for cached fontification.
-Only buffers more than this can have associated Font Lock cache files saved.
-If nil, means cache files are never created.
-If a list, each element should be a cons pair of the form (MAJOR-MODE . SIZE),
-where MAJOR-MODE is a symbol or t (meaning the default).  For example:
- ((c-mode . 25600) (c++-mode . 25600) (rmail-mode . 1048576))
-means that the minimum size is 25K for buffers in C or C++ modes, one megabyte
-for buffers in Rmail mode, and size is irrelevant otherwise."
-  :type '(choice (const :tag "none" nil)
-		 (integer :tag "size")
-		 (repeat :menu-tag "mode specific" :tag "mode specific"
-			 :value ((t . nil))
-			 (cons :tag "Instance"
-			       (radio :tag "Mode"
-				      (const :tag "all" t)
-				      (symbol :tag "name"))
-			       (radio :tag "Size"
-				      (const :tag "none" nil)
-				      (integer :tag "size")))))
-  :group 'fast-lock)
-
-(defcustom fast-lock-cache-directories '("~/.emacs-flc")
-; - `internal', keep each file's Font Lock cache file in the same file.
-; - `external', keep each file's Font Lock cache file in the same directory.
-  "Directories in which Font Lock cache files are saved and read.
-Each item should be either DIR or a cons pair of the form (REGEXP . DIR) where
-DIR is a directory name (relative or absolute) and REGEXP is a regexp.
-
-An attempt will be made to save or read Font Lock cache files using these items
-until one succeeds (i.e., until a readable or writable one is found).  If an
-item contains REGEXP, DIR is used only if the buffer file name matches REGEXP.
-For example:
-
- (let ((home (expand-file-name (abbreviate-file-name (file-truename \"~/\")))))
-   (list (cons (concat \"^\" (regexp-quote home)) \".\") \"~/.emacs-flc\"))
-    =>
- ((\"^/your/true/home/directory/\" . \".\") \"~/.emacs-flc\")
-
-would cause a file's current directory to be used if the file is under your
-home directory hierarchy, or otherwise the absolute directory `~/.emacs-flc'.
-For security reasons, it is not advisable to use the file's current directory
-to avoid the possibility of using the cache of another user."
-  :type '(repeat (radio (directory :tag "directory")
-			(cons :tag "Matching"
-			      (regexp :tag "regexp")
-			      (directory :tag "directory"))))
-  :group 'fast-lock)
-(put 'fast-lock-cache-directories 'risky-local-variable t)
-
-(defcustom fast-lock-save-events '(kill-buffer kill-emacs)
-  "Events under which caches will be saved.
-Valid events are `save-buffer', `kill-buffer' and `kill-emacs'.
-If concurrent editing sessions use the same associated cache file for a file's
-buffer, then you should add `save-buffer' to this list."
-  :type '(set (const :tag "buffer saving" save-buffer)
-	      (const :tag "buffer killing" kill-buffer)
-	      (const :tag "emacs killing" kill-emacs))
-  :group 'fast-lock)
-
-(defcustom fast-lock-save-others t
-  "If non-nil, save Font Lock cache files irrespective of file owner.
-If nil, means only buffer files known to be owned by you can have associated
-Font Lock cache files saved.  Ownership may be unknown for networked files."
-  :type 'boolean
-  :group 'fast-lock)
-
-(defcustom fast-lock-verbose font-lock-verbose
-  "If non-nil, means show status messages for cache processing.
-If a number, only buffers greater than this size have processing messages."
-  :type '(choice (const :tag "never" nil)
-		 (other :tag "always" t)
-		 (integer :tag "size"))
-  :group 'fast-lock)
-
-(defvar fast-lock-save-faces
-  (when (featurep 'xemacs)
-    ;; XEmacs uses extents for everything, so we have to pick the right ones.
-    font-lock-face-list)
-  "Faces that will be saved in a Font Lock cache file.
-If nil, means information for all faces will be saved.")
-\f
-;; User Functions:
-
-;;;###autoload
-(defun fast-lock-mode (&optional arg)
-  "Toggle Fast Lock mode.
-With arg, turn Fast Lock mode on if and only if arg is positive and the buffer
-is associated with a file.  Enable it automatically in your `~/.emacs' by:
-
- (setq font-lock-support-mode \\='fast-lock-mode)
-
-If Fast Lock mode is enabled, and the current buffer does not contain any text
-properties, any associated Font Lock cache is used if its timestamp matches the
-buffer's file, and its `font-lock-keywords' match those that you are using.
-
-Font Lock caches may be saved:
-- When you save the file's buffer.
-- When you kill an unmodified file's buffer.
-- When you exit Emacs, for all unmodified or saved buffers.
-Depending on the value of `fast-lock-save-events'.
-See also the commands `fast-lock-read-cache' and `fast-lock-save-cache'.
-
-Use \\[font-lock-fontify-buffer] to fontify the buffer if the cache is bad.
-
-Various methods of control are provided for the Font Lock cache.  In general,
-see variable `fast-lock-cache-directories' and function `fast-lock-cache-name'.
-For saving, see variables `fast-lock-minimum-size', `fast-lock-save-events',
-`fast-lock-save-others' and `fast-lock-save-faces'."
-  (interactive "P")
-  ;; Only turn on if we are visiting a file.  We could use `buffer-file-name',
-  ;; but many packages temporarily wrap that to nil when doing their own thing.
-  (set (make-local-variable 'fast-lock-mode)
-       (and buffer-file-truename
-	    (not (memq 'fast-lock-mode font-lock-inhibit-thing-lock))
-	    (if arg (> (prefix-numeric-value arg) 0) (not fast-lock-mode))))
-  (if (and fast-lock-mode (not font-lock-mode))
-      ;; Turned on `fast-lock-mode' rather than `font-lock-mode'.
-      (progn
-        (message "Use font-lock-support-mode rather than calling fast-lock-mode")
-        (sit-for 2))
-    ;; Let's get down to business.
-    (set (make-local-variable 'fast-lock-cache-timestamp) nil)
-    (set (make-local-variable 'fast-lock-cache-filename) nil)
-    (when (and fast-lock-mode (not font-lock-fontified))
-      (fast-lock-read-cache))))
-
-(defun fast-lock-read-cache ()
-  "Read the Font Lock cache for the current buffer.
-
-The following criteria must be met for a Font Lock cache file to be read:
-- Fast Lock mode must be turned on in the buffer.
-- The buffer must not be modified.
-- The buffer's `font-lock-keywords' must match the cache's.
-- The buffer file's timestamp must match the cache's.
-- Criteria imposed by `fast-lock-cache-directories'.
-
-See `fast-lock-mode'."
-  (interactive)
-  (let ((directories fast-lock-cache-directories)
-	(modified (buffer-modified-p)) (inhibit-read-only t)
-	(fontified font-lock-fontified))
-    (set (make-local-variable 'font-lock-fontified) nil)
-    ;; Keep trying directories until fontification is turned off.
-    (while (and directories (not font-lock-fontified))
-      (let ((directory (fast-lock-cache-directory (car directories) nil)))
-	(condition-case nil
-	    (when directory
-	      (setq fast-lock-cache-filename (fast-lock-cache-name directory))
-	      (when (file-readable-p fast-lock-cache-filename)
-		(load fast-lock-cache-filename t t t)))
-	  (error nil) (quit nil))
-	(setq directories (cdr directories))))
-    ;; Unset `fast-lock-cache-filename', and restore `font-lock-fontified', if
-    ;; we don't use a cache.  (Note that `fast-lock-cache-data' sets the value
-    ;; of `fast-lock-cache-timestamp'.)
-    (set-buffer-modified-p modified)
-    (unless font-lock-fontified
-      (setq fast-lock-cache-filename nil font-lock-fontified fontified))))
-
-(defun fast-lock-save-cache (&optional buffer)
-  "Save the Font Lock cache of BUFFER or the current buffer.
-
-The following criteria must be met for a Font Lock cache file to be saved:
-- Fast Lock mode must be turned on in the buffer.
-- The event must be one of `fast-lock-save-events'.
-- The buffer must be at least `fast-lock-minimum-size' bytes long.
-- The buffer file must be owned by you, or `fast-lock-save-others' must be t.
-- The buffer must contain at least one `face' text property.
-- The buffer must not be modified.
-- The buffer file's timestamp must be the same as the file's on disk.
-- The on disk file's timestamp must be different than the buffer's cache.
-- Criteria imposed by `fast-lock-cache-directories'.
-
-See `fast-lock-mode'."
-  (interactive)
-  (save-excursion
-    (when buffer
-      (set-buffer buffer))
-    (let ((min-size (font-lock-value-in-major-mode fast-lock-minimum-size))
-	  (file-timestamp (visited-file-modtime)) (saved nil))
-      (when (and fast-lock-mode
-	     ;;
-	     ;; "Only save if the buffer matches the file, the file has
-	     ;; changed, and it was changed by the current emacs session."
-	     ;;
-	     ;; Only save if the buffer is not modified,
-	     ;; (i.e., so we don't save for something not on disk)
-	     (not (buffer-modified-p))
-	     ;; and the file's timestamp is the same as the buffer's,
-	     ;; (i.e., someone else hasn't written the file in the meantime)
-	     (verify-visited-file-modtime (current-buffer))
-	     ;; and the file's timestamp is different from the cache's.
-	     ;; (i.e., a save has occurred since the cache was read)
-	     (not (equal fast-lock-cache-timestamp file-timestamp))
-	     ;;
-	     ;; Only save if user's restrictions are satisfied.
-	     (and min-size (>= (buffer-size) min-size))
-	     (or fast-lock-save-others
-		 (eq (user-uid) (file-attribute-user-id
-				 (file-attributes buffer-file-name))))
-	     ;;
-	     ;; Only save if there are `face' properties to save.
-	     (text-property-not-all (point-min) (point-max) 'face nil))
-	;;
-	;; Try each directory until we manage to save or the user quits.
-	(let ((directories fast-lock-cache-directories))
-	  (while (and directories (memq saved '(nil error)))
-	    (let* ((dir (fast-lock-cache-directory (car directories) t))
-		   (file (and dir (fast-lock-cache-name dir))))
-	      (when (and file (file-writable-p file))
-		(setq saved (fast-lock-save-cache-1 file file-timestamp)))
-	      (setq directories (cdr directories)))))))))
-
-;;;###autoload
-(defun turn-on-fast-lock ()
-  "Unconditionally turn on Fast Lock mode."
-  (fast-lock-mode t))
-\f
-;;; API Functions:
-
-(defun fast-lock-after-fontify-buffer ()
-  ;; Delete the Font Lock cache file used to restore fontification, if any.
-  (when fast-lock-cache-filename
-    (if (file-writable-p fast-lock-cache-filename)
-	(delete-file fast-lock-cache-filename)
-      (message "File %s font lock cache cannot be deleted" (buffer-name))))
-  ;; Flag so that a cache will be saved later even if the file is never saved.
-  (setq fast-lock-cache-timestamp nil))
-
-(defalias 'fast-lock-after-unfontify-buffer
-  'ignore)
-\f
-;; Miscellaneous Functions:
-
-(defun fast-lock-save-cache-after-save-file ()
-  ;; Do `fast-lock-save-cache' if `save-buffer' is on `fast-lock-save-events'.
-  (when (memq 'save-buffer fast-lock-save-events)
-    (fast-lock-save-cache)))
-
-(defun fast-lock-save-cache-before-kill-buffer ()
-  ;; Do `fast-lock-save-cache' if `kill-buffer' is on `fast-lock-save-events'.
-  (when (memq 'kill-buffer fast-lock-save-events)
-    (fast-lock-save-cache)))
-
-(defun fast-lock-save-caches-before-kill-emacs ()
-  ;; Do `fast-lock-save-cache's if `kill-emacs' is on `fast-lock-save-events'.
-  (when (memq 'kill-emacs fast-lock-save-events)
-    (mapcar 'fast-lock-save-cache (buffer-list))))
-
-(defun fast-lock-cache-directory (directory create)
-  "Return usable directory based on DIRECTORY.
-Returns nil if the directory does not exist, or, if CREATE non-nil, cannot be
-created.  DIRECTORY may be a string or a cons pair of the form (REGEXP . DIR).
-See `fast-lock-cache-directories'."
-  (let ((dir
-	 (cond ((not buffer-file-name)
-		;; Should never be nil, but `crypt++' screws it up.
-		nil)
-	       ((stringp directory)
-		;; Just a directory.
-		directory)
-	       (t
-		;; A directory if the file name matches the regexp.
-		(let ((bufile (expand-file-name buffer-file-truename))
-		      (case-fold-search nil))
-		  (when (save-match-data (string-match (car directory) bufile))
-		    (cdr directory)))))))
-    (cond ((not dir)
-	   nil)
-	  ((file-accessible-directory-p dir)
-	   dir)
-	  (create
-	   (condition-case nil
-	       (progn (make-directory dir t) dir)
-	     (error nil))))))
-
-;; If you are wondering why we only hash if the directory is not ".", rather
-;; than if `file-name-absolute-p', it is because if we just appended ".flc" for
-;; relative cache directories (that are not ".") then it is possible that more
-;; than one file would have the same cache name in that directory, if the luser
-;; made a link from one relative cache directory to another.  (Phew!)
-(defun fast-lock-cache-name (directory)
-  "Return full cache file name using caching DIRECTORY.
-If DIRECTORY is `.', the file name is the buffer file name appended with `.flc'.
-Otherwise, the file name is constructed from DIRECTORY and the buffer's true
-abbreviated file name, with all `/' characters in the name replaced with `#'
-characters, and appended with `.flc'.
-
-If the same file has different cache file names when edited on different
-machines, e.g., on one machine the cache file name has the prefix `#home',
-perhaps due to automount, try putting in your `~/.emacs' something like:
-
- (setq directory-abbrev-alist (cons \\='(\"^/home/\" . \"/\") directory-abbrev-alist))
-
-Emacs automagically removes the common `/tmp_mnt' automount prefix by default.
-
-See `fast-lock-cache-directory'."
-  (if (string-equal directory ".")
-      (concat buffer-file-name ".flc")
-    (let* ((bufile (expand-file-name buffer-file-truename))
-	   (chars-alist
-	    (if (memq system-type '(windows-nt cygwin))
-		'((?/ . (?#)) (?# . (?# ?#)) (?: . (?\;)) (?\; . (?\; ?\;)))
-	      '((?/ . (?#)) (?# . (?# ?#)))))
-	   (mapchars
-	    (function (lambda (c) (or (cdr (assq c chars-alist)) (list c))))))
-      (concat
-       (file-name-as-directory (expand-file-name directory))
-       (mapconcat 'char-to-string (apply 'append (mapcar mapchars bufile)) "")
-       ".flc"))))
-\f
-;; Font Lock Cache Processing Functions:
-
-;; The version 3 format of the cache is:
-;;
-;; (fast-lock-cache-data VERSION TIMESTAMP
-;;  font-lock-syntactic-keywords SYNTACTIC-PROPERTIES
-;;  font-lock-keywords FACE-PROPERTIES)
-
-(defun fast-lock-save-cache-1 (file timestamp)
-  ;; Save the FILE with the TIMESTAMP plus fontification data.
-  ;; Returns non-nil if a save was attempted to a writable cache file.
-  (let ((tpbuf (generate-new-buffer " *fast-lock*"))
-	(verbose (if (numberp fast-lock-verbose)
-		     (> (buffer-size) fast-lock-verbose)
-		   fast-lock-verbose))
-	(saved t))
-    (with-temp-message
-	(when verbose
-	  (format "Saving %s font lock cache..." (buffer-name)))
-      (condition-case nil
-	  (save-excursion
-	    (print (list 'fast-lock-cache-data 3
-			 (list 'quote timestamp)
-			 (list 'quote font-lock-syntactic-keywords)
-			 (list 'quote (fast-lock-get-syntactic-properties))
-			 (list 'quote font-lock-keywords)
-			 (list 'quote (fast-lock-get-face-properties)))
-		   tpbuf)
-	    (set-buffer tpbuf)
-	    (write-region (point-min) (point-max) file nil 'quietly)
-	    (setq fast-lock-cache-timestamp timestamp
-		  fast-lock-cache-filename file))
-	(error (setq saved 'error)) (quit (setq saved 'quit)))
-      (kill-buffer tpbuf))
-    (cond ((eq saved 'quit)
-	   (message "Saving %s font lock cache...quit" (buffer-name)))
-	  ((eq saved 'error)
-	   (message "Saving %s font lock cache...failed" (buffer-name))))
-    ;; We return non-nil regardless of whether a failure occurred.
-    saved))
-
-(defun fast-lock-cache-data (version timestamp
-			     syntactic-keywords syntactic-properties
-			     keywords face-properties
-			     &rest ignored)
-  ;; Find value of syntactic keywords in case it is a symbol.
-  (setq font-lock-syntactic-keywords (font-lock-eval-keywords
-				      font-lock-syntactic-keywords))
-  ;; Compile all keywords in case some are and some aren't.
-  (when font-lock-syntactic-keywords
-    (setq font-lock-syntactic-keywords (font-lock-compile-keywords
-					font-lock-syntactic-keywords t)))
-  (when syntactic-keywords
-    (setq syntactic-keywords (font-lock-compile-keywords syntactic-keywords t)))
-  (setq font-lock-keywords (font-lock-compile-keywords font-lock-keywords)
-	keywords (font-lock-compile-keywords keywords))
-  ;; Use the Font Lock cache SYNTACTIC-PROPERTIES and FACE-PROPERTIES if we're
-  ;; using cache VERSION format 3, the current buffer's file timestamp matches
-  ;; the TIMESTAMP, the current buffer's `font-lock-syntactic-keywords' are the
-  ;; same as SYNTACTIC-KEYWORDS, and the current buffer's `font-lock-keywords'
-  ;; are the same as KEYWORDS.
-  (let ((buf-timestamp (visited-file-modtime))
-	(verbose (if (numberp fast-lock-verbose)
-		     (> (buffer-size) fast-lock-verbose)
-		   fast-lock-verbose))
-	(loaded t))
-    (if (or (/= version 3)
-	    (buffer-modified-p)
-	    (not (equal timestamp buf-timestamp))
-	    (not (equal syntactic-keywords font-lock-syntactic-keywords))
-	    (not (equal keywords font-lock-keywords)))
-	(setq loaded nil)
-      (with-temp-message
-	  (when verbose
-	    (format "Loading %s font lock cache..." (buffer-name)))
-	(condition-case nil
-	    (fast-lock-add-properties syntactic-properties face-properties)
-	  (error (setq loaded 'error)) (quit (setq loaded 'quit))))
-      (cond ((eq loaded 'quit)
-	     (message "Loading %s font lock cache...quit" (buffer-name)))
-	    ((eq loaded 'error)
-	     (message "Loading %s font lock cache...failed" (buffer-name)))))
-    (setq font-lock-fontified (eq loaded t)
-	  fast-lock-cache-timestamp (and (eq loaded t) timestamp))))
-\f
-;; Text Properties Processing Functions:
-
-;; This is fast, but fails if adjacent characters have different `face' text
-;; properties.  Maybe that's why I dropped it in the first place?
-;(defun fast-lock-get-face-properties ()
-;  "Return a list of `face' text properties in the current buffer.
-;Each element of the list is of the form (VALUE START1 END1 START2 END2 ...)
-;where VALUE is a `face' property value and STARTx and ENDx are positions."
-;  (save-restriction
-;    (widen)
-;    (let ((start (text-property-not-all (point-min) (point-max) 'face nil))
-;	  (limit (point-max)) end properties value cell)
-;      (while start
-;	(setq end (next-single-property-change start 'face nil limit)
-;	      value (get-text-property start 'face))
-;	;; Make, or add to existing, list of regions with same `face'.
-;	(if (setq cell (assq value properties))
-;	    (setcdr cell (cons start (cons end (cdr cell))))
-;	  (setq properties (cons (list value start end) properties)))
-;	(setq start (next-single-property-change end 'face)))
-;      properties)))
-
-;; This is slow, but copes if adjacent characters have different `face' text
-;; properties, but fails if they are lists.
-;(defun fast-lock-get-face-properties ()
-;  "Return a list of `face' text properties in the current buffer.
-;Each element of the list is of the form (VALUE START1 END1 START2 END2 ...)
-;where VALUE is a `face' property value and STARTx and ENDx are positions.
-;Only those `face' VALUEs in `fast-lock-save-faces' are returned."
-;  (save-restriction
-;    (widen)
-;    (let ((faces (or fast-lock-save-faces (face-list))) (limit (point-max))
-;	  properties regions face start end)
-;      (while faces
-;	(setq face (car faces) faces (cdr faces) regions () end (point-min))
-;	;; Make a list of start/end regions with `face' property face.
-;	(while (setq start (text-property-any end limit 'face face))
-;	  (setq end (or (text-property-not-all start limit 'face face) limit)
-;		regions (cons start (cons end regions))))
-;	;; Add `face' face's regions, if any, to properties.
-;	(when regions
-;	  (push (cons face regions) properties)))
-;      properties)))
-
-(defun fast-lock-get-face-properties ()
-  "Return a list of `face' text properties in the current buffer.
-Each element of the list is of the form (VALUE START1 END1 START2 END2 ...)
-where VALUE is a `face' property value and STARTx and ENDx are positions."
-  (save-restriction
-    (widen)
-    (let ((start (text-property-not-all (point-min) (point-max) 'face nil))
-	  end properties value cell)
-      (while start
-	(setq end (next-single-property-change start 'face nil (point-max))
-	      value (get-text-property start 'face))
-	;; Make, or add to existing, list of regions with same `face'.
-	(cond ((setq cell (assoc value properties))
-	       (setcdr cell (cons start (cons end (cdr cell)))))
-	      ((fast-lock-save-facep value)
-	       (push (list value start end) properties)))
-	(setq start (text-property-not-all end (point-max) 'face nil)))
-      properties)))
-
-(defun fast-lock-get-syntactic-properties ()
-  "Return a list of `syntax-table' text properties in the current buffer.
-See `fast-lock-get-face-properties'."
-  (save-restriction
-    (widen)
-    (let ((start (text-property-not-all (point-min) (point-max) 'syntax-table
-					nil))
-	  end properties value cell)
-      (while start
-	(setq end (next-single-property-change start 'syntax-table nil
-					       (point-max))
-	      value (get-text-property start 'syntax-table))
-	;; Make, or add to existing, list of regions with same `syntax-table'.
-	(if (setq cell (assoc value properties))
-	    (setcdr cell (cons start (cons end (cdr cell))))
-	  (push (list value start end) properties))
-	(setq start (text-property-not-all end (point-max) 'syntax-table nil)))
-      properties)))
-
-(defun fast-lock-add-properties (syntactic-properties face-properties)
-  "Add `syntax-table' and `face' text properties to the current buffer.
-Any existing `syntax-table' and `face' text properties are removed first.
-See `fast-lock-get-face-properties'."
-  (save-buffer-state (plist regions)
-    (save-restriction
-      (widen)
-      (font-lock-unfontify-region (point-min) (point-max))
-      ;;
-      ;; Set the `syntax-table' property for each start/end region.
-      (while syntactic-properties
-	(setq plist (list 'syntax-table (car (car syntactic-properties)))
-	      regions (cdr (car syntactic-properties))
-	      syntactic-properties (cdr syntactic-properties))
-	(while regions
-	  (add-text-properties (nth 0 regions) (nth 1 regions) plist)
-	  (setq regions (nthcdr 2 regions))))
-      ;;
-      ;; Set the `face' property for each start/end region.
-      (while face-properties
-	(setq plist (list 'face (car (car face-properties)))
-	      regions (cdr (car face-properties))
-	      face-properties (cdr face-properties))
-	(while regions
-	  (add-text-properties (nth 0 regions) (nth 1 regions) plist)
-	  (setq regions (nthcdr 2 regions)))))))
-\f
-;; Functions for XEmacs:
-
-(when (featurep 'xemacs)
-  ;;
-  ;; It would be better to use XEmacs' `map-extents' over extents with a
-  ;; `font-lock' property, but `face' properties are on different extents.
-  (defun fast-lock-get-face-properties ()
-    "Return a list of `face' text properties in the current buffer.
-Each element of the list is of the form (VALUE START1 END1 START2 END2 ...)
-where VALUE is a `face' property value and STARTx and ENDx are positions.
-Only those `face' VALUEs in `fast-lock-save-faces' are returned."
-    (save-restriction
-      (widen)
-      (let ((properties ()) cell)
-	(map-extents
-	 (function (lambda (extent ignore)
-	    (let ((value (extent-face extent)))
-	      ;; We're only interested if it's one of `fast-lock-save-faces'.
-	      (when (and value (fast-lock-save-facep value))
-		(let ((start (extent-start-position extent))
-		      (end (extent-end-position extent)))
-		  ;; Make or add to existing list of regions with the same
-		  ;; `face' property value.
-		  (if (setq cell (assoc value properties))
-		      (setcdr cell (cons start (cons end (cdr cell))))
-		    (push (list value start end) properties))))
-	      ;; Return nil to keep `map-extents' going.
-	      nil))))
-	properties)))
-  ;;
-  ;; XEmacs does not support the `syntax-table' text property.
-  (defalias 'fast-lock-get-syntactic-properties
-    'ignore)
-  ;;
-  ;; Make extents just like XEmacs' font-lock.el does.
-  (defun fast-lock-add-properties (syntactic-properties face-properties)
-    "Set `face' text properties in the current buffer.
-Any existing `face' text properties are removed first.
-See `fast-lock-get-face-properties'."
-    (save-restriction
-      (widen)
-      (font-lock-unfontify-region (point-min) (point-max))
-      ;; Set the `face' property, etc., for each start/end region.
-      (while face-properties
-	(let ((face (car (car face-properties)))
-	      (regions (cdr (car face-properties))))
-	  (while regions
-	    (font-lock-set-face (nth 0 regions) (nth 1 regions) face)
-	    (setq regions (nthcdr 2 regions)))
-	  (setq face-properties (cdr face-properties))))
-      ;; XEmacs does not support the `syntax-table' text property.
-      ))
-  ;;
-  ;; XEmacs 19.12 font-lock.el's `font-lock-fontify-buffer' runs a hook.
-  (add-hook 'font-lock-after-fontify-buffer-hook
-	    'fast-lock-after-fontify-buffer))
-
-(unless (boundp 'font-lock-syntactic-keywords)
-  (defvar font-lock-syntactic-keywords nil))
-
-(unless (boundp 'font-lock-inhibit-thing-lock)
-  (defvar font-lock-inhibit-thing-lock nil))
-
-(unless (fboundp 'font-lock-compile-keywords)
-  (defalias 'font-lock-compile-keywords 'identity))
-
-(unless (fboundp 'font-lock-eval-keywords)
-  (defun font-lock-eval-keywords (keywords)
-    (if (symbolp keywords)
-	(font-lock-eval-keywords (if (fboundp keywords)
-				     (funcall keywords)
-				   (eval keywords)))
-      keywords)))
-
-(unless (fboundp 'font-lock-value-in-major-mode)
-  (defun font-lock-value-in-major-mode (alist)
-    (if (consp alist)
-	(cdr (or (assq major-mode alist) (assq t alist)))
-      alist)))
-
-(unless (fboundp 'current-message)
-  (defun current-message ()
-    ""))
-\f
-;; Install ourselves:
-
-(add-hook 'after-save-hook 'fast-lock-save-cache-after-save-file)
-(add-hook 'kill-buffer-hook 'fast-lock-save-cache-before-kill-buffer)
-(unless noninteractive
-  (add-hook 'kill-emacs-hook 'fast-lock-save-caches-before-kill-emacs))
-
-;;;###autoload
-(when (fboundp 'add-minor-mode)
-  (defvar fast-lock-mode nil)
-  (add-minor-mode 'fast-lock-mode nil))
-;;;###dont-autoload
-(unless (assq 'fast-lock-mode minor-mode-alist)
-  (setq minor-mode-alist (append minor-mode-alist '((fast-lock-mode nil)))))
-
-;; Provide ourselves:
-
-(provide 'fast-lock)
-
-;;; fast-lock.el ends here
-
-;; Local Variables:
-;; byte-compile-warnings: (not obsolete)
-;; End:
diff --git a/lisp/obsolete/lazy-lock.el b/lisp/obsolete/lazy-lock.el
deleted file mode 100644
index 694188ff23..0000000000
--- a/lisp/obsolete/lazy-lock.el
+++ /dev/null
@@ -1,1053 +0,0 @@
-;;; lazy-lock.el --- lazy demand-driven fontification for fast Font Lock mode
-
-;; Copyright (C) 1994-1998, 2001-2020 Free Software Foundation, Inc.
-
-;; Author: Simon Marshall <simon@gnu.org>
-;; Maintainer: emacs-devel@gnu.org
-;; Keywords: faces files
-;; Version: 2.11
-;; Obsolete-since: 22.1
-
-;; This file is part of GNU Emacs.
-
-;; GNU Emacs is free software: you can redistribute it and/or modify
-;; it under the terms of the GNU General Public License as published by
-;; the Free Software Foundation, either version 3 of the License, or
-;; (at your option) any later version.
-
-;; GNU Emacs is distributed in the hope that it will be useful,
-;; but WITHOUT ANY WARRANTY; without even the implied warranty of
-;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-;; GNU General Public License for more details.
-
-;; You should have received a copy of the GNU General Public License
-;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
-
-;;; Commentary:
-
-;; Purpose:
-;;
-;; Lazy Lock mode is a Font Lock support mode.
-;; It makes visiting buffers in Font Lock mode faster by making fontification
-;; be demand-driven, deferred and stealthy, so that fontification only occurs
-;; when, and where, necessary.
-;;
-;; See caveats and feedback below.
-;; See also the fast-lock package.  (But don't use them at the same time!)
-
-;; Installation:
-;;
-;; Put in your ~/.emacs:
-;;
-;; (setq font-lock-support-mode 'lazy-lock-mode)
-;;
-;; Start up a new Emacs and use font-lock as usual (except that you can use the
-;; so-called "gaudier" fontification regexps on big files without frustration).
-;;
-;; In a buffer (which has `font-lock-mode' enabled) which is at least
-;; `lazy-lock-minimum-size' characters long, buffer fontification will not
-;; occur and only the visible portion of the buffer will be fontified.  Motion
-;; around the buffer will fontify those visible portions not previously
-;; fontified.  If stealth fontification is enabled, buffer fontification will
-;; occur in invisible parts of the buffer after `lazy-lock-stealth-time'
-;; seconds of idle time.  If on-the-fly fontification is deferred, on-the-fly
-;; fontification will occur after `lazy-lock-defer-time' seconds of idle time.
-
-;; User-visible differences with version 1:
-;;
-;; - Version 2 can defer on-the-fly fontification.  Therefore you need not, and
-;; should not, use defer-lock.el with this version of lazy-lock.el.
-;;
-;; A number of variables have changed meaning:
-;;
-;; - A value of nil for the variable `lazy-lock-minimum-size' means never turn
-;; on demand-driven fontification.  In version 1 this meant always turn on
-;; demand-driven fontification.  If you really want demand-driven fontification
-;; regardless of buffer size, set this variable to 0.
-;;
-;; - The variable `lazy-lock-stealth-lines' cannot have a nil value.  In
-;; version 1 this meant use `window-height' as the maximum number of lines to
-;; fontify as a stealth chunk.  This makes no sense; stealth fontification is
-;; of a buffer, not a window.
-
-;; Implementation differences with version 1:
-;;
-;; - Version 1 of lazy-lock.el is a bit of a hack.  Version 1 demand-driven
-;; fontification, the core feature of lazy-lock.el, is implemented by placing a
-;; function on `post-command-hook'.  This function fontifies where necessary,
-;; i.e., where a window scroll has occurred.  However, there are a number of
-;; problems with using `post-command-hook':
-;;
-;; (a) As the name suggests, `post-command-hook' is run after every command,
-;;     i.e., frequently and regardless of whether scrolling has occurred.
-;; (b) Scrolling can occur during a command, when `post-command-hook' is not
-;;     run, i.e., it is not necessarily run after scrolling has occurred.
-;; (c) When `post-command-hook' is run, there is nothing to suggest where
-;;     scrolling might have occurred, i.e., which windows have scrolled.
-;;
-;; Thus lazy-lock.el's function is called almost as often as possible, usually
-;; when it need not be called, yet it is not always called when it is needed.
-;; Also, lazy-lock.el's function must check each window to see if a scroll has
-;; occurred there.  Worse still, lazy-lock.el's function must fontify a region
-;; twice as large as necessary to make sure the window is completely fontified.
-;; Basically, `post-command-hook' is completely inappropriate for lazy-lock.el.
-;;
-;; Ideally, we want to attach lazy-lock.el's function to a hook that is run
-;; only when scrolling occurs, e.g., `window-start' has changed, and tells us
-;; as much information as we need, i.e., the window and its new buffer region.
-;; Richard Stallman implemented a `window-scroll-functions' for Emacs 19.30.
-;; Functions on it are run when `window-start' has changed, and are supplied
-;; with the window and the window's new `window-start' position.  (It would be
-;; better if it also supplied the window's new `window-end' position, but that
-;; is calculated as part of the redisplay process, and the functions on
-;; `window-scroll-functions' are run before redisplay has finished.)  Thus, the
-;; hook deals with the above problems (a), (b) and (c).
-;;
-;; If only life was that easy.  Version 2 demand-driven fontification is mostly
-;; implemented by placing a function on `window-scroll-functions'.  However,
-;; not all scrolling occurs when `window-start' has changed.  A change in
-;; window size, e.g., via C-x 1, or a significant deletion, e.g., of a number
-;; of lines, causes text previously invisible (i.e., after `window-end') to
-;; become visible without changing `window-start'.  Arguably, these events are
-;; not scrolling events, but fontification must occur for lazy-lock.el to work.
-;; Hooks `window-size-change-functions' and `redisplay-end-trigger-functions'
-;; were added for these circumstances.
-;;
-;; (Ben Wing thinks these hooks are "horribly horribly kludgy", and implemented
-;; a `pre-idle-hook', a `mother-of-all-post-command-hooks', for XEmacs 19.14.
-;; He then hacked up a version 1 lazy-lock.el to use `pre-idle-hook' rather
-;; than `post-command-hook'.  Whereas functions on `post-command-hook' are
-;; called almost as often as possible, functions on `pre-idle-hook' really are
-;; called as often as possible, even when the mouse moves and, on some systems,
-;; while XEmacs is idle.  Thus, the hook deals with the above problem (b), but
-;; unfortunately it makes (a) worse and does not address (c) at all.
-;;
-;; I freely admit that `redisplay-end-trigger-functions' and, to a much lesser
-;; extent, `window-size-change-functions' are not pretty.  However, I feel that
-;; a `window-scroll-functions' feature is cleaner than a `pre-idle-hook', and
-;; the result is faster and smaller, less intrusive and more targeted, code.
-;; Since `pre-idle-hook' is pretty much like `post-command-hook', there is no
-;; point in making this version of lazy-lock.el work with it.  Anyway, that's
-;; Lit 30 of my humble opinion.
-;;
-;; - Version 1 stealth fontification is also implemented by placing a function
-;; on `post-command-hook'.  This function waits for a given amount of time,
-;; and, if Emacs remains idle, fontifies where necessary.  Again, there are a
-;; number of problems with using `post-command-hook':
-;;
-;; (a) Functions on `post-command-hook' are run sequentially, so this function
-;;     can interfere with other functions on the hook, and vice versa.
-;; (b) This function waits for a given amount of time, so it can interfere with
-;;     various features that are dealt with by Emacs after a command, e.g.,
-;;     region highlighting, asynchronous updating and keystroke echoing.
-;; (c) Fontification may be required during a command, when `post-command-hook'
-;;     is not run.  (Version 2 deferred fontification only.)
-;;
-;; Again, `post-command-hook' is completely inappropriate for lazy-lock.el.
-;; Richard Stallman and Morten Welinder implemented internal Timers and Idle
-;; Timers for Emacs 19.31.  Functions can be run independently at given times
-;; or after given amounts of idle time.  Thus, the feature deals with the above
-;; problems (a), (b) and (c).  Version 2 deferral and stealth are implemented
-;; by functions on Idle Timers.  (A function on XEmacs' `pre-idle-hook' is
-;; similar to an Emacs Idle Timer function with a fixed zero second timeout.)
-
-;; - Version 1 has the following problems (relative to version 2):
-;;
-;; (a) It is slow when it does its job.
-;; (b) It does not always do its job when it should.
-;; (c) It slows all interaction (when it doesn't need to do its job).
-;; (d) It interferes with other package functions on `post-command-hook'.
-;; (e) It interferes with Emacs things within the read-eval loop.
-;;
-;; Ben's hacked-up lazy-lock.el 1.14 almost solved (b) but made (c) worse.
-;;
-;; - Version 2 has the following additional features (relative to version 1):
-;;
-;; (a) It can defer fontification (both on-the-fly and on-scrolling).
-;; (b) It can fontify contextually (syntactically true on-the-fly).
-
-;; Caveats:
-;;
-;; Lazy Lock mode does not work efficiently with Outline mode.
-;; This is because when in Outline mode, although text may be not visible to
-;; you in the window, the text is visible to Emacs Lisp code (not surprisingly)
-;; and Lazy Lock fontifies it mercilessly.  Maybe it will be fixed one day.
-;;
-;; Because buffer text is not necessarily fontified, other packages that expect
-;; buffer text to be fontified in Font Lock mode either might not work as
-;; expected, or might not display buffer text as expected.  An example of the
-;; latter is `occur', which copies lines of buffer text into another buffer.
-;;
-;; In Emacs 19.30, Lazy Lock mode does not ensure that an existing buffer is
-;; fontified if it is made visible via a minibuffer-less command that replaces
-;; an existing window's buffer (e.g., via the Buffers menu).  Upgrade!
-;;
-;; In Emacs 19.30, Lazy Lock mode does not work well with Transient Mark mode
-;; or modes based on Comint mode (e.g., Shell mode), and also interferes with
-;; the echoing of keystrokes in the minibuffer.  This is because of the way
-;; deferral and stealth have to be implemented for Emacs 19.30.  Upgrade!
-;;
-;; Currently XEmacs does not have the features to support this version of
-;; lazy-lock.el.  Maybe it will one day.
-\f
-;; History:
-;;
-;; 1.15--2.00:
-;; - Rewrite for Emacs 19.30 and the features rms added to support lazy-lock.el
-;;   so that it could work correctly and efficiently.
-;; - Many thanks to those who reported bugs, fixed bugs, made suggestions or
-;;   otherwise contributed in the version 1 cycle; Jari Aalto, Kevin Broadey,
-;;   Ulrik Dickow, Bill Dubuque, Bob Glickstein, Boris Goldowsky,
-;;   Jonas Jarnestrom, David Karr, Michael Kifer, Erik Naggum, Rick Sladkey,
-;;   Jim Thompson, Ben Wing, Ilya Zakharevich, and Richard Stallman.
-;; 2.00--2.01:
-;; - Made `lazy-lock-fontify-after-command' always `sit-for' and so redisplay
-;; - Use `buffer-name' not `buffer-live-p' (Bill Dubuque hint)
-;; - Made `lazy-lock-install' do `add-to-list' not `setq' of `current-buffer'
-;; - Made `lazy-lock-fontify-after-install' loop over buffer list
-;; - Made `lazy-lock-arrange-before-change' to arrange `window-end' triggering
-;; - Made `lazy-lock-let-buffer-state' wrap both `befter-change-functions'
-;; - Made `lazy-lock-fontify-region' do `condition-case' (Hyman Rosen report)
-;; 2.01--2.02:
-;; - Use `buffer-live-p' as `buffer-name' can barf (Richard Stanton report)
-;; - Made `lazy-lock-install' set `font-lock-fontified' (Kevin Davidson report)
-;; - Made `lazy-lock-install' add hooks only if needed
-;; - Made `lazy-lock-unstall' add `font-lock-after-change-function' if needed
-;; 2.02--2.03:
-;; - Made `lazy-lock-fontify-region' do `condition-case' for `quit' too
-;; - Made `lazy-lock-mode' respect the value of `font-lock-inhibit-thing-lock'
-;; - Added `lazy-lock-after-unfontify-buffer'
-;; - Removed `lazy-lock-fontify-after-install' hack
-;; - Made `lazy-lock-fontify-after-scroll' not `set-buffer' to `window-buffer'
-;; - Made `lazy-lock-fontify-after-trigger' not `set-buffer' to `window-buffer'
-;; - Made `lazy-lock-fontify-after-idle' be interruptible (Scott Burson hint)
-;; 2.03--2.04:
-;; - Rewrite for Emacs 19.31 idle timers
-;; - Renamed `buffer-windows' to `get-buffer-window-list'
-;; - Removed `buffer-live-p'
-;; - Made `lazy-lock-defer-after-change' always save `current-buffer'
-;; - Made `lazy-lock-fontify-after-defer' just process buffers
-;; - Made `lazy-lock-install-hooks' add hooks correctly (Kevin Broadey report)
-;; - Made `lazy-lock-install' cope if `lazy-lock-defer-time' is a list
-;; 2.04--2.05:
-;; - Rewrite for Common Lisp macros
-;; - Added `do-while' macro
-;; - Renamed `lazy-lock-let-buffer-state' macro to `save-buffer-state'
-;; - Returned `lazy-lock-fontify-after-install' hack (Darren Hall hint)
-;; - Added `lazy-lock-defer-on-scrolling' functionality (Scott Byer hint)
-;; - Made `lazy-lock-mode' wrap `font-lock-support-mode'
-;; 2.05--2.06:
-;; - Made `lazy-lock-fontify-after-defer' swap correctly (Scott Byer report)
-;; 2.06--2.07:
-;; - Added `lazy-lock-stealth-load' functionality (Rob Hooft hint)
-;; - Made `lazy-lock-unstall' call `lazy-lock-fontify-region' if needed
-;; - Made `lazy-lock-mode' call `lazy-lock-unstall' only if needed
-;; - Made `lazy-lock-defer-after-scroll' do `set-window-redisplay-end-trigger'
-;; - Added `lazy-lock-defer-contextually' functionality
-;; - Added `lazy-lock-defer-on-the-fly' from `lazy-lock-defer-time'
-;; - Renamed `lazy-lock-defer-driven' to `lazy-lock-defer-on-scrolling'
-;; - Removed `lazy-lock-submit-bug-report' and bade farewell
-;; 2.07--2.08:
-;; - Made `lazy-lock-fontify-conservatively' fontify around `window-point'
-;; - Made `save-buffer-state' wrap `inhibit-point-motion-hooks'
-;; - Added Custom support
-;; 2.08--2.09:
-;; - Removed `byte-*' variables from `eval-when-compile' (Erik Naggum hint)
-;; - Made various wrapping `inhibit-point-motion-hooks' (Vinicius Latorre hint)
-;; - Made `lazy-lock-fontify-after-idle' wrap `minibuffer-auto-raise'
-;; - Made `lazy-lock-fontify-after-defer' paranoid about deferred buffers
-;; 2.09--2.10:
-;; - Use `window-end' UPDATE arg for Emacs 20.4 and later.
-;; - Made deferral `widen' before unfontifying (Dan Nicolaescu report)
-;; - Use `lazy-lock-fontify-after-visage' for hideshow.el (Dan Nicolaescu hint)
-;; - Use `other' widget where possible (Andreas Schwab fix)
-;; 2.10--2.11:
-;; - Used `with-temp-message' where possible to make messages temporary.
-\f
-;;; Code:
-
-(require 'font-lock)
-(eval-when-compile (require 'cl-lib))
-
-(eval-when-compile
- ;; We use this to preserve or protect things when modifying text properties.
- (defmacro save-buffer-state (varlist &rest body)
-   "Bind variables according to VARLIST and eval BODY restoring buffer state."
-   `(let* (,@(append varlist
-		   '((modified (buffer-modified-p))
-		     (buffer-undo-list t)
-		     (inhibit-read-only t)
-		     (inhibit-point-motion-hooks t)
-		     (inhibit-modification-hooks t)
-		     deactivate-mark
-		     buffer-file-name
-		     buffer-file-truename)))
-     ,@body
-     (when (and (not modified) (buffer-modified-p))
-       (restore-buffer-modified-p nil))))
- (put 'save-buffer-state 'lisp-indent-function 1)
- ;;
- ;; We use this for clarity and speed.  Naughty but nice.
- (defmacro do-while (test &rest body)
-   "(do-while TEST BODY...): eval BODY... and repeat if TEST yields non-nil.
-The order of execution is thus BODY, TEST, BODY, TEST and so on
-until TEST returns nil."
-   `(while (progn ,@body ,test)))
- (put 'do-while 'lisp-indent-function (get 'while 'lisp-indent-function)))
-
-(defgroup lazy-lock nil
-  "Font Lock support mode to fontify lazily."
-  :group 'font-lock)
-
-(defvar lazy-lock-mode nil)			; Whether we are turned on.
-(defvar lazy-lock-buffers nil)			; For deferral.
-(defvar lazy-lock-timers (cons nil nil))	; For deferral and stealth.
-\f
-;; User Variables:
-
-(defcustom lazy-lock-minimum-size 25600
-  "Minimum size of a buffer for demand-driven fontification.
-On-demand fontification occurs if the buffer size is greater than this value.
-If nil, means demand-driven fontification is never performed.
-If a list, each element should be a cons pair of the form (MAJOR-MODE . SIZE),
-where MAJOR-MODE is a symbol or t (meaning the default).  For example:
- ((c-mode . 25600) (c++-mode . 25600) (rmail-mode . 1048576))
-means that the minimum size is 25K for buffers in C or C++ modes, one megabyte
-for buffers in Rmail mode, and size is irrelevant otherwise.
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "none" nil)
-		 (integer :tag "size")
-		 (repeat :menu-tag "mode specific" :tag "mode specific"
-			 :value ((t . nil))
-			 (cons :tag "Instance"
-			       (radio :tag "Mode"
-				      (const :tag "all" t)
-				      (symbol :tag "name"))
-			       (radio :tag "Size"
-				      (const :tag "none" nil)
-				      (integer :tag "size")))))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-defer-on-the-fly t
-  "If non-nil, means fontification after a change should be deferred.
-If nil, means on-the-fly fontification is performed.  This means when changes
-occur in the buffer, those areas are immediately fontified.
-If a list, it should be a list of `major-mode' symbol names for which deferred
-fontification should occur.  The sense of the list is negated if it begins with
-`not'.  For example:
- (c-mode c++-mode)
-means that on-the-fly fontification is deferred for buffers in C and C++ modes
-only, and deferral does not occur otherwise.
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "never" nil)
-		 (const :tag "always" t)
-		 (set :menu-tag "mode specific" :tag "modes"
-		      :value (not)
-		      (const :tag "Except" not)
-		      (repeat :inline t (symbol :tag "mode"))))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-defer-on-scrolling nil
-  "If non-nil, means fontification after a scroll should be deferred.
-If nil, means demand-driven fontification is performed.  This means when
-scrolling into unfontified areas of the buffer, those areas are immediately
-fontified.  Thus scrolling never presents unfontified areas.  However, since
-fontification occurs during scrolling, scrolling may be slow.
-If t, means defer-driven fontification is performed.  This means fontification
-of those areas is deferred.  Thus scrolling may present momentarily unfontified
-areas.  However, since fontification does not occur during scrolling, scrolling
-will be faster than demand-driven fontification.
-If any other value, e.g., `eventually', means demand-driven fontification is
-performed until the buffer is fontified, then buffer fontification becomes
-defer-driven.  Thus scrolling never presents unfontified areas until the buffer
-is first fontified, after which subsequent scrolling may present future buffer
-insertions momentarily unfontified.  However, since fontification does not
-occur during scrolling after the buffer is first fontified, scrolling will
-become faster.  (But, since contextual changes continually occur, such a value
-makes little sense if `lazy-lock-defer-contextually' is non-nil.)
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "never" nil)
-		 (const :tag "always" t)
-		 (other :tag "eventually" eventually))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-defer-contextually 'syntax-driven
-  "If non-nil, means deferred fontification should be syntactically true.
-If nil, means deferred fontification occurs only on those lines modified.  This
-means where modification on a line causes syntactic change on subsequent lines,
-those subsequent lines are not refontified to reflect their new context.
-If t, means deferred fontification occurs on those lines modified and all
-subsequent lines.  This means those subsequent lines are refontified to reflect
-their new syntactic context, either immediately or when scrolling into them.
-If any other value, e.g., `syntax-driven', means deferred syntactically true
-fontification occurs only if syntactic fontification is performed using the
-buffer mode's syntax table, i.e., only if `font-lock-keywords-only' is nil.
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "never" nil)
-		 (const :tag "always" t)
-		 (other :tag "syntax-driven" syntax-driven))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-defer-time 0.25
-  "Time in seconds to delay before beginning deferred fontification.
-Deferred fontification occurs if there is no input within this time.
-If nil, means fontification is never deferred, regardless of the values of the
-variables `lazy-lock-defer-on-the-fly', `lazy-lock-defer-on-scrolling' and
-`lazy-lock-defer-contextually'.
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "never" nil)
-		 (number :tag "seconds"))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-stealth-time 30
-  "Time in seconds to delay before beginning stealth fontification.
-Stealth fontification occurs if there is no input within this time.
-If nil, means stealth fontification is never performed.
-
-The value of this variable is used when Lazy Lock mode is turned on."
-  :type '(choice (const :tag "never" nil)
-		 (number :tag "seconds"))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-stealth-lines (if font-lock-maximum-decoration 100 250)
-  "Maximum size of a chunk of stealth fontification.
-Each iteration of stealth fontification can fontify this number of lines.
-To speed up input response during stealth fontification, at the cost of stealth
-taking longer to fontify, you could reduce the value of this variable."
-  :type '(integer :tag "lines")
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-stealth-load
-  (if (condition-case nil (load-average) (error)) 200)
-  "Load in percentage above which stealth fontification is suspended.
-Stealth fontification pauses when the system short-term load average (as
-returned by the function `load-average' if supported) goes above this level,
-thus reducing the demand that stealth fontification makes on the system.
-If nil, means stealth fontification is never suspended.
-To reduce machine load during stealth fontification, at the cost of stealth
-taking longer to fontify, you could reduce the value of this variable.
-See also `lazy-lock-stealth-nice'."
-  :type (if (condition-case nil (load-average) (error))
-	    '(choice (const :tag "never" nil)
-		     (integer :tag "load"))
-	  '(const :format "%t: unsupported\n" nil))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-stealth-nice 0.125
-  "Time in seconds to pause between chunks of stealth fontification.
-Each iteration of stealth fontification is separated by this amount of time,
-thus reducing the demand that stealth fontification makes on the system.
-If nil, means stealth fontification is never paused.
-To reduce machine load during stealth fontification, at the cost of stealth
-taking longer to fontify, you could increase the value of this variable.
-See also `lazy-lock-stealth-load'."
-  :type '(choice (const :tag "never" nil)
-		 (number :tag "seconds"))
-  :group 'lazy-lock)
-
-(defcustom lazy-lock-stealth-verbose
-  (and (not lazy-lock-defer-contextually) (not (null font-lock-verbose)))
-  "If non-nil, means stealth fontification should show status messages."
-  :type 'boolean
-  :group 'lazy-lock)
-\f
-;; User Functions:
-
-;;;###autoload
-(defun lazy-lock-mode (&optional arg)
-  "Toggle Lazy Lock mode.
-With arg, turn Lazy Lock mode on if and only if arg is positive.  Enable it
-automatically in your `~/.emacs' by:
-
- (setq font-lock-support-mode \\='lazy-lock-mode)
-
-For a newer font-lock support mode with similar functionality, see
-`jit-lock-mode'.  Eventually, Lazy Lock mode will be deprecated in
-JIT Lock's favor.
-
-When Lazy Lock mode is enabled, fontification can be lazy in a number of ways:
-
-- Demand-driven buffer fontification if `lazy-lock-minimum-size' is non-nil.
-  This means initial fontification does not occur if the buffer is greater than
-  `lazy-lock-minimum-size' characters in length.  Instead, fontification occurs
-  when necessary, such as when scrolling through the buffer would otherwise
-  reveal unfontified areas.  This is useful if buffer fontification is too slow
-  for large buffers.
-
-- Deferred scroll fontification if `lazy-lock-defer-on-scrolling' is non-nil.
-  This means demand-driven fontification does not occur as you scroll.
-  Instead, fontification is deferred until after `lazy-lock-defer-time' seconds
-  of Emacs idle time, while Emacs remains idle.  This is useful if
-  fontification is too slow to keep up with scrolling.
-
-- Deferred on-the-fly fontification if `lazy-lock-defer-on-the-fly' is non-nil.
-  This means on-the-fly fontification does not occur as you type.  Instead,
-  fontification is deferred until after `lazy-lock-defer-time' seconds of Emacs
-  idle time, while Emacs remains idle.  This is useful if fontification is too
-  slow to keep up with your typing.
-
-- Deferred context fontification if `lazy-lock-defer-contextually' is non-nil.
-  This means fontification updates the buffer corresponding to true syntactic
-  context, after `lazy-lock-defer-time' seconds of Emacs idle time, while Emacs
-  remains idle.  Otherwise, fontification occurs on modified lines only, and
-  subsequent lines can remain fontified corresponding to previous syntactic
-  contexts.  This is useful where strings or comments span lines.
-
-- Stealthy buffer fontification if `lazy-lock-stealth-time' is non-nil.
-  This means remaining unfontified areas of buffers are fontified if Emacs has
-  been idle for `lazy-lock-stealth-time' seconds, while Emacs remains idle.
-  This is useful if any buffer has any deferred fontification.
-
-Basic Font Lock mode on-the-fly fontification behavior fontifies modified
-lines only.  Thus, if `lazy-lock-defer-contextually' is non-nil, Lazy Lock mode
-on-the-fly fontification may fontify differently, albeit correctly.  In any
-event, to refontify some lines you can use \\[font-lock-fontify-block].
-
-Stealth fontification only occurs while the system remains unloaded.
-If the system load rises above `lazy-lock-stealth-load' percent, stealth
-fontification is suspended.  Stealth fontification intensity is controlled via
-the variable `lazy-lock-stealth-nice' and `lazy-lock-stealth-lines', and
-verbosity is controlled via the variable `lazy-lock-stealth-verbose'."
-  (interactive "P")
-  (let* ((was-on lazy-lock-mode)
-	 (now-on (unless (memq 'lazy-lock-mode font-lock-inhibit-thing-lock)
-		   (if arg (> (prefix-numeric-value arg) 0) (not was-on)))))
-    (cond ((and now-on (not font-lock-mode))
-	   ;; Turned on `lazy-lock-mode' rather than `font-lock-mode'.
-           (message "Use font-lock-support-mode rather than calling lazy-lock-mode")
-           (sit-for 2))
-	  (now-on
-	   ;; Turn ourselves on.
-	   (set (make-local-variable 'lazy-lock-mode) t)
-	   (lazy-lock-install))
-	  (was-on
-	   ;; Turn ourselves off.
-	   (set (make-local-variable 'lazy-lock-mode) nil)
-	   (lazy-lock-unstall)))))
-
-;;;###autoload
-(defun turn-on-lazy-lock ()
-  "Unconditionally turn on Lazy Lock mode."
-  (lazy-lock-mode t))
-
-(defun lazy-lock-install ()
-  (let ((min-size (font-lock-value-in-major-mode lazy-lock-minimum-size))
-	(defer-change (and lazy-lock-defer-time lazy-lock-defer-on-the-fly))
-	(defer-scroll (and lazy-lock-defer-time lazy-lock-defer-on-scrolling))
-	(defer-context (and lazy-lock-defer-time lazy-lock-defer-contextually
-			    (or (eq lazy-lock-defer-contextually t)
-				(null font-lock-keywords-only)))))
-    ;;
-    ;; Tell Font Lock whether Lazy Lock will do fontification.
-    (make-local-variable 'font-lock-fontified)
-    (setq font-lock-fontified (and min-size (>= (buffer-size) min-size)))
-    ;;
-    ;; Add the text properties and fontify.
-    (if (not font-lock-fontified)
-	(lazy-lock-after-fontify-buffer)
-      ;; Make sure we fontify in any existing windows showing the buffer.
-      (let ((windows (get-buffer-window-list (current-buffer) 'nomini t)))
-	(lazy-lock-after-unfontify-buffer)
-	(while windows
-	  (lazy-lock-fontify-conservatively (car windows))
-	  (setq windows (cdr windows)))))
-    ;;
-    ;; Add the fontification hooks.
-    (lazy-lock-install-hooks
-     font-lock-fontified
-     (cond ((eq (car-safe defer-change) 'not)
-	    (not (memq major-mode (cdr defer-change))))
-	   ((listp defer-change)
-	    (memq major-mode defer-change))
-	   (t
-	    defer-change))
-     (eq defer-scroll t)
-     defer-context)
-    ;;
-    ;; Add the fontification timers.
-    (lazy-lock-install-timers
-     (if (or defer-change defer-scroll defer-context) lazy-lock-defer-time)
-     lazy-lock-stealth-time)))
-
-(defun lazy-lock-install-hooks (fontifying
-				defer-change defer-scroll defer-context)
-  ;;
-  ;; Add hook if lazy-lock.el is fontifying on scrolling or is deferring.
-  (when (or fontifying defer-change defer-scroll defer-context)
-    (add-hook 'window-scroll-functions (if defer-scroll
-					   'lazy-lock-defer-after-scroll
-					 'lazy-lock-fontify-after-scroll)
-	      nil t))
-  ;;
-  ;; Add hook if lazy-lock.el is fontifying and is not deferring changes.
-  (when (and fontifying (not defer-change) (not defer-context))
-    (add-hook 'before-change-functions 'lazy-lock-arrange-before-change nil t))
-  ;;
-  ;; Replace Font Lock mode hook.
-  (remove-hook 'after-change-functions 'font-lock-after-change-function t)
-  (add-hook 'after-change-functions
-	    (cond ((and defer-change defer-context)
-		   'lazy-lock-defer-rest-after-change)
-		  (defer-change
-		   'lazy-lock-defer-line-after-change)
-		  (defer-context
-		   'lazy-lock-fontify-rest-after-change)
-		  (t
-		   'lazy-lock-fontify-line-after-change))
-	    nil t)
-  ;;
-  ;; Add package-specific hook.
-  (add-hook 'outline-view-change-hook 'lazy-lock-fontify-after-visage nil t)
-  (add-hook 'hs-hide-hook 'lazy-lock-fontify-after-visage nil t))
-
-(defun lazy-lock-install-timers (dtime stime)
-  ;; Schedule or re-schedule the deferral and stealth timers.
-  ;; The layout of `lazy-lock-timers' is:
-  ;;  ((DEFER-TIME . DEFER-TIMER) (STEALTH-TIME . STEALTH-TIMER)
-  ;; If an idle timeout has changed, cancel the existing idle timer (if there
-  ;; is one) and schedule a new one (if the new idle timeout is non-nil).
-  (unless (eq dtime (car (car lazy-lock-timers)))
-    (let ((defer (car lazy-lock-timers)))
-      (when (cdr defer)
-	(cancel-timer (cdr defer)))
-      (setcar lazy-lock-timers (cons dtime (and dtime
-	      (run-with-idle-timer dtime t 'lazy-lock-fontify-after-defer))))))
-  (unless (eq stime (car (cdr lazy-lock-timers)))
-    (let ((stealth (cdr lazy-lock-timers)))
-      (when (cdr stealth)
-	(cancel-timer (cdr stealth)))
-      (setcdr lazy-lock-timers (cons stime (and stime
-	      (run-with-idle-timer stime t 'lazy-lock-fontify-after-idle)))))))
-
-(defun lazy-lock-unstall ()
-  ;;
-  ;; If Font Lock mode is still enabled, make sure that the buffer is
-  ;; fontified, and reinstall its hook.  We must do this first.
-  (when font-lock-mode
-    (when (lazy-lock-unfontified-p)
-      (let ((verbose (if (numberp font-lock-verbose)
-			 (> (buffer-size) font-lock-verbose)
-		       font-lock-verbose)))
-	(with-temp-message
-	    (when verbose
-	      (format "Fontifying %s..." (buffer-name)))
-	  ;; Make sure we fontify etc. in the whole buffer.
-	  (save-restriction
-	    (widen)
-	    (lazy-lock-fontify-region (point-min) (point-max))))))
-    (add-hook 'after-change-functions 'font-lock-after-change-function nil t))
-  ;;
-  ;; Remove the text properties.
-  (lazy-lock-after-unfontify-buffer)
-  ;;
-  ;; Remove the fontification hooks.
-  (remove-hook 'window-scroll-functions 'lazy-lock-fontify-after-scroll t)
-  (remove-hook 'window-scroll-functions 'lazy-lock-defer-after-scroll t)
-  (remove-hook 'before-change-functions 'lazy-lock-arrange-before-change t)
-  (remove-hook 'after-change-functions 'lazy-lock-fontify-line-after-change t)
-  (remove-hook 'after-change-functions 'lazy-lock-fontify-rest-after-change t)
-  (remove-hook 'after-change-functions 'lazy-lock-defer-line-after-change t)
-  (remove-hook 'after-change-functions 'lazy-lock-defer-rest-after-change t)
-  (remove-hook 'outline-view-change-hook 'lazy-lock-fontify-after-visage t)
-  (remove-hook 'hs-hide-hook 'lazy-lock-fontify-after-visage t))
-\f
-;; Hook functions.
-
-;; Lazy Lock mode intervenes when (1) a previously invisible buffer region
-;; becomes visible, i.e., for demand- or defer-driven on-the-scroll
-;; fontification, (2) a buffer modification occurs, i.e., for defer-driven
-;; on-the-fly fontification, (3) Emacs becomes idle, i.e., for fontification of
-;; deferred fontification and stealth fontification, and (4) other special
-;; occasions.
-
-;; 1.  There are three ways whereby this can happen.
-;;
-;; (a) Scrolling the window, either explicitly (e.g., `scroll-up') or
-;;     implicitly (e.g., `search-forward').  Here, `window-start' changes.
-;;     Fontification occurs by adding `lazy-lock-fontify-after-scroll' (for
-;;     demand-driven fontification) or `lazy-lock-defer-after-scroll' (for
-;;     defer-driven fontification) to the hook `window-scroll-functions'.
-
-(defun lazy-lock-fontify-after-scroll (window window-start)
-  ;; Called from `window-scroll-functions'.
-  ;; Fontify WINDOW from WINDOW-START following the scroll.
-  (let ((inhibit-point-motion-hooks t))
-    (lazy-lock-fontify-region window-start (window-end window t)))
-  ;; A prior deletion that did not cause scrolling, followed by a scroll, would
-  ;; result in an unnecessary trigger after this if we did not cancel it now.
-  (set-window-redisplay-end-trigger window nil))
-
-(defun lazy-lock-defer-after-scroll (window window-start)
-  ;; Called from `window-scroll-functions'.
-  ;; Defer fontification following the scroll.  Save the current buffer so that
-  ;; we subsequently fontify in all windows showing the buffer.
-  (unless (memq (current-buffer) lazy-lock-buffers)
-    (push (current-buffer) lazy-lock-buffers))
-  ;; A prior deletion that did not cause scrolling, followed by a scroll, would
-  ;; result in an unnecessary trigger after this if we did not cancel it now.
-  (set-window-redisplay-end-trigger window nil))
-
-;; (b) Resizing the window, either explicitly (e.g., `enlarge-window') or
-;;     implicitly (e.g., `delete-other-windows').  Here, `window-end' changes.
-;;     Fontification occurs by adding `lazy-lock-fontify-after-resize' to the
-;;     hook `window-size-change-functions'.
-
-(defun lazy-lock-fontify-after-resize (frame)
-  ;; Called from `window-size-change-functions'.
-  ;; Fontify windows in FRAME following the resize.  We cannot use
-  ;; `window-start' or `window-end' so we fontify conservatively.
-  (save-excursion
-    (save-selected-window
-      (select-frame frame)
-      (walk-windows (function (lambda (window)
-		       (set-buffer (window-buffer window))
-		       (when lazy-lock-mode
-			 (lazy-lock-fontify-conservatively window))
-		       (set-window-redisplay-end-trigger window nil)))
-		    'nomini frame))))
-
-;; (c) Deletion in the buffer.  Here, a `window-end' marker can become visible.
-;;     Fontification occurs by adding `lazy-lock-arrange-before-change' to
-;;     `before-change-functions' and `lazy-lock-fontify-after-trigger' to the
-;;     hook `redisplay-end-trigger-functions'.  Before every deletion, the
-;;     marker `window-redisplay-end-trigger' position is set to the soon-to-be
-;;     changed `window-end' position.  If the marker becomes visible,
-;;     `lazy-lock-fontify-after-trigger' gets called.  Ouch.  Note that we only
-;;     have to deal with this eventuality if there is no on-the-fly deferral.
-
-(defun lazy-lock-arrange-before-change (beg end)
-  ;; Called from `before-change-functions'.
-  ;; Arrange that if text becomes visible it will be fontified (if a deletion
-  ;; is pending, text might become visible at the bottom).
-  (unless (eq beg end)
-    (let ((windows (get-buffer-window-list (current-buffer) 'nomini t)) window)
-      (while windows
-	(setq window (car windows))
-	(unless (markerp (window-redisplay-end-trigger window))
-	  (set-window-redisplay-end-trigger window (make-marker)))
-	(set-marker (window-redisplay-end-trigger window) (window-end window))
-	(setq windows (cdr windows))))))
-
-(defun lazy-lock-fontify-after-trigger (window trigger-point)
-  ;; Called from `redisplay-end-trigger-functions'.
-  ;; Fontify WINDOW from TRIGGER-POINT following the redisplay.
-  ;; We could probably just use `lazy-lock-fontify-after-scroll' without loss:
-  ;;  (inline (lazy-lock-fontify-after-scroll window (window-start window)))
-  (let ((inhibit-point-motion-hooks t))
-    (lazy-lock-fontify-region trigger-point (window-end window t))))
-
-;; 2.  Modified text must be marked as unfontified so it can be identified and
-;;     fontified later when Emacs is idle.  Deferral occurs by adding one of
-;;     `lazy-lock-fontify-*-after-change' (for on-the-fly fontification) or
-;;     `lazy-lock-defer-*-after-change' (for deferred fontification) to the
-;;     hook `after-change-functions'.
-
-(defalias 'lazy-lock-fontify-line-after-change
-  ;; Called from `after-change-functions'.
-  ;; Fontify the current change.
-  'font-lock-after-change-function)
-
-(defun lazy-lock-fontify-rest-after-change (beg end old-len)
-  ;; Called from `after-change-functions'.
-  ;; Fontify the current change and defer fontification of the rest of the
-  ;; buffer.  Save the current buffer so that we subsequently fontify in all
-  ;; windows showing the buffer.
-  (lazy-lock-fontify-line-after-change beg end old-len)
-  (save-buffer-state nil
-    (unless (memq (current-buffer) lazy-lock-buffers)
-      (push (current-buffer) lazy-lock-buffers))
-    (save-restriction
-      (widen)
-      (remove-text-properties end (point-max) '(lazy-lock nil)))))
-
-(defun lazy-lock-defer-line-after-change (beg end old-len)
-  ;; Called from `after-change-functions'.
-  ;; Defer fontification of the current change.  Save the current buffer so
-  ;; that we subsequently fontify in all windows showing the buffer.
-  (save-buffer-state nil
-    (unless (memq (current-buffer) lazy-lock-buffers)
-      (push (current-buffer) lazy-lock-buffers))
-    (remove-text-properties (max (1- beg) (point-min))
-			    (min (1+ end) (point-max))
-			    '(lazy-lock nil))))
-
-(defun lazy-lock-defer-rest-after-change (beg end old-len)
-  ;; Called from `after-change-functions'.
-  ;; Defer fontification of the rest of the buffer.  Save the current buffer so
-  ;; that we subsequently fontify in all windows showing the buffer.
-  (save-buffer-state nil
-    (unless (memq (current-buffer) lazy-lock-buffers)
-      (push (current-buffer) lazy-lock-buffers))
-    (save-restriction
-      (widen)
-      (remove-text-properties (max (1- beg) (point-min))
-			      (point-max)
-			      '(lazy-lock nil)))))
-
-;; 3.  Deferred fontification and stealth fontification are done from these two
-;;     functions.  They are set up as Idle Timers.
-
-(defun lazy-lock-fontify-after-defer ()
-  ;; Called from `timer-idle-list'.
-  ;; Fontify all windows where deferral has occurred for its buffer.
-  (save-excursion
-    (while (and lazy-lock-buffers (not (input-pending-p)))
-      (let ((buffer (car lazy-lock-buffers)) windows)
-	;; Paranoia: check that the buffer is still live and Lazy Lock mode on.
-	(when (buffer-live-p buffer)
-	  (set-buffer buffer)
-	  (when lazy-lock-mode
-	    (setq windows (get-buffer-window-list buffer 'nomini t))
-	    (while windows
-	      (lazy-lock-fontify-window (car windows))
-	      (setq windows (cdr windows)))))
-	(setq lazy-lock-buffers (cdr lazy-lock-buffers)))))
-  ;; Add hook if fontification should now be defer-driven in this buffer.
-  (when (and lazy-lock-mode lazy-lock-defer-on-scrolling
-	     (memq 'lazy-lock-fontify-after-scroll window-scroll-functions)
-	     (not (or (input-pending-p) (lazy-lock-unfontified-p))))
-    (remove-hook 'window-scroll-functions 'lazy-lock-fontify-after-scroll t)
-    (add-hook 'window-scroll-functions 'lazy-lock-defer-after-scroll nil t)))
-
-(defun lazy-lock-fontify-after-idle ()
-  ;; Called from `timer-idle-list'.
-  ;; Fontify all buffers that need it, stealthily while idle.
-  (unless (or executing-kbd-macro (window-minibuffer-p (selected-window)))
-    ;; Loop over all buffers, fontify stealthily for each if necessary.
-    (let ((buffers (buffer-list)) (continue t)
-	  message message-log-max minibuffer-auto-raise)
-      (save-excursion
-	(do-while (and buffers continue)
-	  (set-buffer (car buffers))
-	  (if (not (and lazy-lock-mode (lazy-lock-unfontified-p)))
-	      (setq continue (not (input-pending-p)))
-	    ;; Fontify regions in this buffer while there is no input.
-	    (with-temp-message
-		(when lazy-lock-stealth-verbose
-		  "Fontifying stealthily...")
-	      (do-while (and (lazy-lock-unfontified-p) continue)
-		(if (and lazy-lock-stealth-load
-			 (> (car (load-average)) lazy-lock-stealth-load))
-		    ;; Wait a while before continuing with the loop.
-		    (progn
-		      (when message
-			(message "Fontifying stealthily...suspended")
-			(setq message nil))
-		      (setq continue (sit-for (or lazy-lock-stealth-time 30))))
-		  ;; Fontify a chunk.
-		  (when lazy-lock-stealth-verbose
-		    (if message
-			(message "Fontifying stealthily... %2d%% of %s"
-				 (lazy-lock-percent-fontified) (buffer-name))
-		      (message "Fontifying stealthily...")
-		      (setq message t)))
-		  ;; Current buffer may have changed during `sit-for'.
-		  (set-buffer (car buffers))
-		  (lazy-lock-fontify-chunk)
-		  (setq continue (sit-for (or lazy-lock-stealth-nice 0)))))))
-	  (setq buffers (cdr buffers)))))))
-
-;; 4.  Special circumstances.
-
-(defun lazy-lock-fontify-after-visage ()
-  ;; Called from `outline-view-change-hook' and `hs-hide-hook'.
-  ;; Fontify windows showing the current buffer, as its visibility has changed.
-  ;; This is a conspiracy hack between lazy-lock.el, outline.el and
-  ;; hideshow.el.
-  (let ((windows (get-buffer-window-list (current-buffer) 'nomini t)))
-    (while windows
-      (lazy-lock-fontify-conservatively (car windows))
-      (setq windows (cdr windows)))))
-
-(defun lazy-lock-after-fontify-buffer ()
-  ;; Called from `font-lock-after-fontify-buffer'.
-  ;; Mark the current buffer as fontified.
-  ;; This is a conspiracy hack between lazy-lock.el and font-lock.el.
-  (save-buffer-state nil
-    (add-text-properties (point-min) (point-max) '(lazy-lock t))))
-
-(defun lazy-lock-after-unfontify-buffer ()
-  ;; Called from `font-lock-after-unfontify-buffer'.
-  ;; Mark the current buffer as unfontified.
-  ;; This is a conspiracy hack between lazy-lock.el and font-lock.el.
-  (save-buffer-state nil
-    (remove-text-properties (point-min) (point-max) '(lazy-lock nil))))
-\f
-;; Fontification functions.
-
-;; If packages want to ensure that some region of the buffer is fontified, they
-;; should use this function.  For an example, see ps-print.el.
-(defun lazy-lock-fontify-region (beg end)
-  ;; Fontify between BEG and END, where necessary, in the current buffer.
-  (save-restriction
-    (widen)
-    (when (setq beg (text-property-any beg end 'lazy-lock nil))
-      (save-excursion
-	(save-match-data
-	  (save-buffer-state
-	   (next)
-	   ;; Find successive unfontified regions between BEG and END.
-	   (condition-case data
-	       (do-while beg
-			 (setq next (or (text-property-any beg end 'lazy-lock t) end))
-	  ;; Make sure the region end points are at beginning of line.
-			 (goto-char beg)
-			 (unless (bolp)
-			   (beginning-of-line)
-			   (setq beg (point)))
-			 (goto-char next)
-			 (unless (bolp)
-			   (forward-line)
-			   (setq next (point)))
-		     ;; Fontify the region, then flag it as fontified.
-			 (font-lock-fontify-region beg next)
-			 (add-text-properties beg next '(lazy-lock t))
-			 (setq beg (text-property-any next end 'lazy-lock nil)))
-	     ((error quit) (message "Fontifying region...%s" data)))))))))
-
-(defun lazy-lock-fontify-chunk ()
-  ;; Fontify the nearest chunk, for stealth, in the current buffer.
-  (let ((inhibit-point-motion-hooks t))
-    (save-excursion
-      (save-restriction
-	(widen)
-	;; Move to end of line in case the character at point is not fontified.
-	(end-of-line)
-	;; Find where the previous (next) unfontified regions end (begin).
-	(let ((prev (previous-single-property-change (point) 'lazy-lock))
-	      (next (text-property-any (point) (point-max) 'lazy-lock nil)))
-	  ;; Fontify from the nearest unfontified position.
-	  (if (or (null prev) (and next (< (- next (point)) (- (point) prev))))
-	      ;; The next, or neither, region is the nearest not fontified.
-	      (lazy-lock-fontify-region
-	       (progn (goto-char (or next (point-min)))
-		      (beginning-of-line)
-		      (point))
-	       (progn (goto-char (or next (point-min)))
-		      (forward-line lazy-lock-stealth-lines)
-		      (point)))
-	    ;; The previous region is the nearest not fontified.
-	    (lazy-lock-fontify-region
-	     (progn (goto-char prev)
-		    (forward-line (- lazy-lock-stealth-lines))
-		    (point))
-	     (progn (goto-char prev)
-		    (forward-line)
-		    (point)))))))))
-
-(defun lazy-lock-fontify-window (window)
-  ;; Fontify in WINDOW between `window-start' and `window-end'.
-  ;; We can only do this when we can use `window-start' and `window-end'.
-  (with-current-buffer (window-buffer window)
-    (lazy-lock-fontify-region (window-start window) (window-end window))))
-
-(defun lazy-lock-fontify-conservatively (window)
-  ;; Fontify in WINDOW conservatively around point.
-  ;; Where we cannot use `window-start' and `window-end' we do `window-height'
-  ;; lines around point.  That way we guarantee to have done enough.
-  (with-current-buffer (window-buffer window)
-    (let ((inhibit-point-motion-hooks t))
-      (lazy-lock-fontify-region
-       (save-excursion
-	 (goto-char (window-point window))
-	 (vertical-motion (- (window-height window)) window) (point))
-       (save-excursion
-	 (goto-char (window-point window))
-	 (vertical-motion (window-height window) window) (point))))))
-
-(defun lazy-lock-unfontified-p ()
-  ;; Return non-nil if there is anywhere still to be fontified.
-  (save-restriction
-    (widen)
-    (text-property-any (point-min) (point-max) 'lazy-lock nil)))
-
-(defun lazy-lock-percent-fontified ()
-  ;; Return the percentage (of characters) of the buffer that are fontified.
-  (save-restriction
-    (widen)
-    (let ((beg (point-min)) (size 0) next)
-      ;; Find where the next fontified region begins.
-      (while (setq beg (text-property-any beg (point-max) 'lazy-lock t))
-	(setq next (or (text-property-any beg (point-max) 'lazy-lock nil)
-		       (point-max)))
-	(cl-incf size (- next beg))
-	(setq beg next))
-      ;; Float because using integer multiplication will frequently overflow.
-      (truncate (* (/ (float size) (point-max)) 100)))))
-\f
-;; Version dependent workarounds and fixes.
-
-(when (consp lazy-lock-defer-time)
-  ;;
-  ;; In 2.06.04 and below, `lazy-lock-defer-time' could specify modes and time.
-  (with-output-to-temp-buffer "*Help*"
-    (princ "The value of the variable `lazy-lock-defer-time' was\n ")
-    (princ lazy-lock-defer-time)
-    (princ "\n")
-    (princ "This variable cannot now be a list of modes and time,\n")
-    (princ "so instead use ")
-    (princ (substitute-command-keys "\\[customize-option]"))
-    (princ " to modify the variables, or put the forms:\n")
-    (princ " (setq lazy-lock-defer-time ")
-    (princ (cdr lazy-lock-defer-time))
-    (princ ")\n")
-    (princ " (setq lazy-lock-defer-on-the-fly '")
-    (princ (car lazy-lock-defer-time))
-    (princ ")\n")
-    (princ "in your ~/.emacs.  ")
-    (princ "The above forms have been evaluated for this editor session,\n")
-    (princ "but you should use ")
-    (princ (substitute-command-keys "\\[customize-option]"))
-    (princ " or change your ~/.emacs now."))
-  (setq lazy-lock-defer-on-the-fly (car lazy-lock-defer-time)
-	lazy-lock-defer-time (cdr lazy-lock-defer-time)))
-
-(when (boundp 'lazy-lock-defer-driven)
-  ;;
-  ;; In 2.06.04 and below, `lazy-lock-defer-driven' was the variable name.
-  (with-output-to-temp-buffer "*Help*"
-    (princ "The value of the variable `lazy-lock-defer-driven' is set to ")
-    (if (memq lazy-lock-defer-driven '(nil t))
-	(princ lazy-lock-defer-driven)
-      (princ "`")
-      (princ lazy-lock-defer-driven)
-      (princ "'"))
-    (princ ".\n")
-    (princ "This variable is now called `lazy-lock-defer-on-scrolling',\n")
-    (princ "so instead use ")
-    (princ (substitute-command-keys "\\[customize-option]"))
-    (princ " to modify the variable, or put the form:\n")
-    (princ " (setq lazy-lock-defer-on-scrolling ")
-    (unless (memq lazy-lock-defer-driven '(nil t))
-      (princ "'"))
-    (princ lazy-lock-defer-driven)
-    (princ ")\n")
-    (princ "in your ~/.emacs.  ")
-    (princ "The above form has been evaluated for this editor session,\n")
-    (princ "but you should use ")
-    (princ (substitute-command-keys "\\[customize-option]"))
-    (princ " or change your ~/.emacs now."))
-  (setq lazy-lock-defer-on-scrolling lazy-lock-defer-driven))
-\f
-;; Install ourselves:
-
-(add-hook 'window-size-change-functions 'lazy-lock-fontify-after-resize)
-(add-hook 'redisplay-end-trigger-functions 'lazy-lock-fontify-after-trigger)
-
-(unless (assq 'lazy-lock-mode minor-mode-alist)
-  (setq minor-mode-alist (append minor-mode-alist '((lazy-lock-mode nil)))))
-
-;; Provide ourselves:
-
-(provide 'lazy-lock)
-
-;; Local Variables:
-;; byte-compile-warnings: (not obsolete)
-;; End:
-
-;;; lazy-lock.el ends here
diff --git a/lisp/progmodes/antlr-mode.el b/lisp/progmodes/antlr-mode.el
index bf56a7ee49..7864485895 100644
--- a/lisp/progmodes/antlr-mode.el
+++ b/lisp/progmodes/antlr-mode.el
@@ -75,9 +75,6 @@
 ;;   (add-hook 'speedbar-load-hook  ; would be too late in antlr-mode.el
 ;;	       (lambda () (speedbar-add-supported-extension ".g")))
 
-;; I strongly recommend to use font-lock with a support mode like fast-lock,
-;; lazy-lock or better jit-lock (Emacs-21.1+) / lazy-shot (XEmacs).
-
 ;; To customize, use menu item "Antlr" -> "Customize Antlr".
 
 ;;; Code:
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index 386cc2f16f..15f303c756 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -1430,7 +1430,7 @@ c-font-lock-declarations
 	     ))
 
 	;; Below we fontify a whole declaration even when it crosses the limit,
-	;; to avoid gaps when jit/lazy-lock fontifies the file a block at a
+	;; to avoid gaps when jit-lock fontifies the file a block at a
 	;; time.  That is however annoying during editing, e.g. the following is
 	;; a common situation while the first line is being written:
 	;;
diff --git a/lisp/progmodes/cperl-mode.el b/lisp/progmodes/cperl-mode.el
index 6122caf518..648fbf43c3 100644
--- a/lisp/progmodes/cperl-mode.el
+++ b/lisp/progmodes/cperl-mode.el
@@ -49,8 +49,6 @@
 ;; `cperl-praise', `cperl-speed'.				   <<<<<<
 
 ;; The mode information (on C-h m) provides some customization help.
-;; If you use font-lock feature of this mode, it is advisable to use
-;; either lazy-lock-mode or fast-lock-mode.  I prefer lazy-lock.
 
 ;; Faces used now: three faces for first-class and second-class keywords
 ;; and control flow words, one for each: comments, string, labels,
diff --git a/lisp/ps-print.el b/lisp/ps-print.el
index ace3001781..367495f5bd 100644
--- a/lisp/ps-print.el
+++ b/lisp/ps-print.el
@@ -6331,14 +6331,11 @@ ps-screen-to-bit-face
 
 
 (declare-function jit-lock-fontify-now "jit-lock" (&optional start end))
-(declare-function lazy-lock-fontify-region "lazy-lock" (beg end))
 
 ;; to avoid compilation gripes
 (defun ps-print-ensure-fontified (start end)
   (cond ((and (boundp 'jit-lock-mode) (symbol-value 'jit-lock-mode))
-	 (jit-lock-fontify-now start end))
-	((and (boundp 'lazy-lock-mode) (symbol-value 'lazy-lock-mode))
-	 (lazy-lock-fontify-region start end))))
+	 (jit-lock-fontify-now start end))))
 
 
 (defun ps-generate-postscript-with-faces (from to)
diff --git a/src/buffer.c b/src/buffer.c
index 241f2d43a9..e4eb6d6741 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -1398,13 +1398,13 @@ DEFUN ("set-buffer-modified-p", Fset_buffer_modified_p, Sset_buffer_modified_p,
 {
   Frestore_buffer_modified_p (flag);
 
-  /* Set update_mode_lines only if buffer is displayed in some window.
-     Packages like jit-lock or lazy-lock preserve a buffer's modified
-     state by recording/restoring the state around blocks of code.
-     Setting update_mode_lines makes redisplay consider all windows
-     (on all frames).  Stealth fontification of buffers not displayed
-     would incur additional redisplay costs if we'd set
-     update_modes_lines unconditionally.
+  /* Set update_mode_lines only if buffer is displayed in some
+     window. Packages like jit-lock preserve a buffer's modified state
+     by recording/restoring the state around blocks of code. Setting
+     update_mode_lines makes redisplay consider all windows (on all
+     frames).  Stealth fontification of buffers not displayed would
+     incur additional redisplay costs if we'd set update_modes_lines
+     unconditionally.
 
      Ideally, I think there should be another mechanism for fontifying
      buffers without "modifying" buffers, or redisplay should be
-- 
2.27.0

Re: [PATCH] Remove obsolete fast-lock and lazy-lock libraries

[Stefan Monnier]

>> There are also these two libraries:
>> ./lisp/obsolete/fast-lock.el9:;; Obsolete-since: 22.1
>> ./lisp/obsolete/lazy-lock.el9:;; Obsolete-since: 22.1
> I have attached a patch to remove them below.

LGTM.

>  ;; to avoid compilation gripes
>  (defun ps-print-ensure-fontified (start end)
>    (cond ((and (boundp 'jit-lock-mode) (symbol-value 'jit-lock-mode))
> -	 (jit-lock-fontify-now start end))
> -	((and (boundp 'lazy-lock-mode) (symbol-value 'lazy-lock-mode))
> -	 (lazy-lock-fontify-region start end))))
> +	 (jit-lock-fontify-now start end))))

I think this can be replaced by `font-lock-ensure`, but it probably
requires additional tests to be sure, so maybe keep the code as is and
just add a FIXME.


        Stefan

Delete variables obsolete since Emacs 23

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

I've attached a patch which removes most variables declared obsolete in
Emacs 23.1.  I tried following the style in Glenn's commit f1c48b0ec5
which does the same for Emacs 22.1.

1. There is a small number of variables declared obsolete in 23.2, 23.3
   and 23.4.  I think we can remove them in a subsequent patch.

2. I was not able to untangle how to remove this, and so left it alone
   for now:

   * lisp/net/newst-backend.el (newsticker-cache-filename)

3. Note the added FIXME in `vc-default-working-revision', where I'm not
   sure if it should be declared obsolete or not.  See Stefan M's commit
   6e5d0e9e73.

4. The variable `translation-table-for-input' was declared obsolete with
   in Emacs 23.1 and has the following comment.  I'm not sure what to do
   about it, if anything.

   ;; This was introduced in 21.4 for pre-unicode unification.  That
   ;; usage was rendered obsolete in 23.1, which uses Unicode internally.
   ;; Other uses are possible, so this variable is not _really_ obsolete,
   ;; but Stefan insists to mark it so.

Other than that, does the attached patch look okay for master?

Best regards,
Stefan Kangas

[-- Attachment #2: 0001-Remove-many-items-obsolete-since-Emacs-23.1.patch --]
[-- Type: text/x-diff, Size: 50121 bytes --]

From 993fd6697972f99d503385aec1f5012a573af41c Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Sat, 16 May 2020 14:16:24 +0200
Subject: [PATCH] Remove many items obsolete since Emacs 23.1

Emacs 23.1 was five major releases and over a decade ago.
This list can be reviewed before to the next release, but for now
hopefully this motivates any needed external updates.

* lisp/abbrev.el (pre-abbrev-expand-hook):
* lisp/bookmark.el (bookmark-read-annotation-text-func)
(bookmark-jump-noselect):
* lisp/buff-menu.el (buffer-menu-mode-hook):
* lisp/cus-edit.el (custom-mode-hook, custom-mode):
* lisp/dirtrack.el (dirtrack-debug-toggle, dirtrack-debug):
* lisp/emacs-lisp/crm.el (crm-minibuffer-complete)
(crm-minibuffer-completion-help)
(crm-minibuffer-complete-and-exit):
* lisp/emacs-lisp/easymenu.el
(easy-menu-precalculate-equivalent-keybindings):
* lisp/emacs-lisp/lisp-mode.el (lisp-mode-auto-fill):
* lisp/epa.el (epa-display-verify-result):
* lisp/epg.el (epg-passphrase-callback-function):
* lisp/eshell/eshell.el (eshell-report-bug):
* lisp/ffap.el (ffap-bug, ffap-submit-bug):
* lisp/files.el (locate-file-completion):
* lisp/hi-lock.el (hi-lock-face-history, hi-lock-regexp-history):
* lisp/hilit-chg.el (highlight-changes-initial-state)
(highlight-changes-active-string)
(highlight-changes-passive-string, global-highlight-changes):
* lisp/international/mule-cmds.el (nonascii-insert-offset)
(nonascii-translation-table):
* lisp/international/mule-diag.el (non-iso-charset-alist):
* lisp/international/mule-util.el (detect-coding-with-priority):
* lisp/international/mule.el (charset-id, charset-bytes)
(charset-list, char-valid-p, generic-char-p)
(char-coding-system-table, make-coding-system)
(set-coding-priority)
* lisp/mail/rmail.el (rmail-message-filter):
* lisp/minibuffer.el (complete-in-turn, dynamic-completion-table)
(completion-common-substring)
(minibuffer-local-must-match-filename-map):
* lisp/mouse.el (mouse-major-mode-menu, mouse-popup-menubar)
(mouse-popup-menubar-stuff):
* lisp/net/newst-treeview.el (newsticker-groups-filename):
* lisp/obsolete/tpu-edt.el (tpu-have-ispell, GOLD-map):
* lisp/obsolete/vc-arch.el (vc-arch-command):
* lisp/password-cache.el (password-read-and-add):
* lisp/shell.el (shell-dirtrack-toggle):
* lisp/subr.el (forward-point, define-key-rebound-commands)
(redisplay-end-trigger-functions, window-redisplay-end-trigger)
(set-window-redisplay-end-trigger, process-filter-multibyte-p)
(set-process-filter-multibyte):
* lisp/t-mouse.el (t-mouse-mode):
* lisp/term/w32-win.el (w32-focus-frame, w32-select-font):
* lisp/textmodes/ispell.el (ispell-aspell-supports-utf8):
* lisp/textmodes/remember.el (remember-buffer):
* lisp/tooltip.el (tooltip-hook):
* lisp/url/url-util.el (url-generate-unique-filename):
* lisp/url/url-vars.el (url-temporary-directory):
* lisp/vc/vc-hooks.el (vc-workfile-version):
* lisp/vc/vc-mtn.el (vc-mtn-command):
* lisp/vc/vc.el (vc-revert-buffer):
* lisp/vcursor.el (vcursor-toggle-vcursor-map):
Remove items, obsolete since Emacs 23.1.
* lisp/abbrev.el (expand-abbrev):
Don't run removed hook 'pre-abbrev-expand-hook'.
* lisp/cedet/semantic/wisent/wisent.el:
* lisp/progmodes/idlwave.el:
Update reference to removed function 'char-valid-p'.
* lisp/gnus/mml2015.el (epg-encrypt-string):
* lisp/gnus/mml1991.el (epg-make-context):
* lisp/gnus/mml-smime.el (autoload):
Remove autoload of removed 'epg-passphrase-callback-function'.
* lisp/minibuffer.el (completion-extra-properties):
Remove support for `completion-common-substring'.
* lisp/obsolete/tpu-edt.el (tpu-toggle-overwrite-mode)
Remove support for removed `spell' package.
; * etc/NEWS: List removed items.
---
 etc/NEWS                             |  34 +++++++
 lisp/abbrev.el                       |  10 --
 lisp/bookmark.el                     |  13 ---
 lisp/buff-menu.el                    |   3 -
 lisp/cedet/semantic/wisent/wisent.el |   6 +-
 lisp/cus-edit.el                     |   4 -
 lisp/dirtrack.el                     |   3 -
 lisp/emacs-lisp/crm.el               |   6 --
 lisp/emacs-lisp/easymenu.el          |  10 --
 lisp/emacs-lisp/lisp-mode.el         |   2 -
 lisp/epa.el                          |   4 -
 lisp/epg.el                          |  13 ---
 lisp/eshell/eshell.el                |   9 --
 lisp/ffap.el                         |   6 --
 lisp/files.el                        |   8 --
 lisp/gnus/mml-smime.el               |   1 -
 lisp/gnus/mml1991.el                 |   1 -
 lisp/gnus/mml2015.el                 |   1 -
 lisp/hi-lock.el                      |   6 --
 lisp/hilit-chg.el                    |  16 ----
 lisp/international/mule-cmds.el      |   5 -
 lisp/international/mule-diag.el      |   4 -
 lisp/international/mule-util.el      |   9 --
 lisp/international/mule.el           | 134 ---------------------------
 lisp/mail/rmail.el                   |  19 ----
 lisp/minibuffer.el                   |  18 +---
 lisp/mouse.el                        |  28 ------
 lisp/net/newst-treeview.el           |  30 +-----
 lisp/obsolete/tpu-edt.el             |  12 +--
 lisp/obsolete/vc-arch.el             |   2 -
 lisp/password-cache.el               |  16 ----
 lisp/progmodes/idlwave.el            |   2 -
 lisp/shell.el                        |   3 -
 lisp/subr.el                         |  11 ---
 lisp/t-mouse.el                      |   2 -
 lisp/term/w32-win.el                 |   4 -
 lisp/textmodes/ispell.el             |   9 --
 lisp/textmodes/remember.el           |   3 -
 lisp/tooltip.el                      |   2 -
 lisp/url/url-util.el                 |  25 -----
 lisp/url/url-vars.el                 |   7 --
 lisp/vc/vc-hooks.el                  |   3 +-
 lisp/vc/vc-mtn.el                    |   1 -
 lisp/vc/vc.el                        |   3 -
 lisp/vcursor.el                      |   3 -
 45 files changed, 41 insertions(+), 470 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index b93199f362..0e3791e13a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -401,6 +401,40 @@ Use macro 'with-current-buffer-window' with action alist entry 'body-function'.
 ** Some libraries obsolete since Emacs 23 have been removed:
 'ledit.el', 'lmenu.el', 'lucid.el and 'old-whitespace.el'.
 
+---
+** Some functions and variables obsolete since Emacs 23 have been removed:
+'GOLD-map', 'bookmark-jump-noselect',
+'bookmark-read-annotation-text-func', 'buffer-menu-mode-hook',
+'char-coding-system-table', 'char-valid-p', 'charset-bytes',
+'charset-id', 'charset-list', 'complete-in-turn',
+'completion-common-substring', 'crm-minibuffer-complete',
+'crm-minibuffer-complete-and-exit', 'crm-minibuffer-completion-help',
+'custom-mode', 'custom-mode-hook', 'define-key-rebound-commands',
+'detect-coding-with-priority', 'dirtrack-debug',
+'dirtrack-debug-toggle', 'dynamic-completion-table',
+'easy-menu-precalculate-equivalent-keybindings',
+'epa-display-verify-result', 'epg-passphrase-callback-function',
+'eshell-report-bug', 'ffap-bug', 'ffap-submit-bug', 'forward-point',
+'generic-char-p', 'global-highlight-changes', 'hi-lock-face-history',
+'hi-lock-regexp-history', 'highlight-changes-active-string',
+'highlight-changes-initial-state', 'highlight-changes-passive-string',
+'ispell-aspell-supports-utf8', 'lisp-mode-auto-fill',
+'locate-file-completion', 'make-coding-system',
+'minibuffer-local-must-match-filename-map', 'mouse-major-mode-menu',
+'mouse-popup-menubar', 'mouse-popup-menubar-stuff',
+'newsticker-groups-filename', 'non-iso-charset-alist',
+'nonascii-insert-offset', 'nonascii-translation-table',
+'password-read-and-add', 'pre-abbrev-expand-hook',
+'process-filter-multibyte-p', 'redisplay-end-trigger-functions',
+'remember-buffer', 'rmail-message-filter', 'set-coding-priority',
+'set-process-filter-multibyte', 'set-window-redisplay-end-trigger',
+'shell-dirtrack-toggle', 't-mouse-mode', 'tooltip-hook',
+'tpu-have-ispell', 'url-generate-unique-filename',
+'url-temporary-directory', 'vc-arch-command', 'vc-mtn-command',
+'vc-revert-buffer', 'vc-workfile-version',
+'vcursor-toggle-vcursor-map', 'w32-focus-frame', 'w32-select-font',
+'window-redisplay-end-trigger'.
+
 \f
 * Lisp Changes in Emacs 28.1
 
diff --git a/lisp/abbrev.el b/lisp/abbrev.el
index 190b3504fa..6fcaa52d46 100644
--- a/lisp/abbrev.el
+++ b/lisp/abbrev.el
@@ -521,14 +521,6 @@ last-abbrev-location
 ;;   "Local (mode-specific) abbrev table of current buffer.")
 ;; (make-variable-buffer-local 'local-abbrev-table)
 
-(defcustom pre-abbrev-expand-hook nil
-  "Function or functions to be called before abbrev expansion is done.
-This is the first thing that `expand-abbrev' does, and so this may change
-the current abbrev table before abbrev lookup happens."
-  :type 'hook
-  :group 'abbrev-mode)
-(make-obsolete-variable 'pre-abbrev-expand-hook 'abbrev-expand-function "23.1")
-
 (defun clear-abbrev-table (table)
   "Undefine all abbrevs in abbrev table TABLE, leaving it empty."
   (setq abbrevs-changed t)
@@ -840,12 +832,10 @@ abbrev-expand-function
 (defun expand-abbrev ()
   "Expand the abbrev before point, if there is an abbrev there.
 Effective when explicitly called even when `abbrev-mode' is nil.
-Before doing anything else, runs `pre-abbrev-expand-hook'.
 Calls the value of `abbrev-expand-function' with no argument to do
 the work, and returns whatever it does.  (That return value should
 be the abbrev symbol if expansion occurred, else nil.)"
   (interactive)
-  (run-hooks 'pre-abbrev-expand-hook)
   (funcall abbrev-expand-function))
 
 (defun abbrev--default-expand ()
diff --git a/lisp/bookmark.el b/lisp/bookmark.el
index 0fa77ed322..f4580cd36e 100644
--- a/lisp/bookmark.el
+++ b/lisp/bookmark.el
@@ -922,8 +922,6 @@ bookmark-default-annotation-text
 	  "#  Date:    " (current-time-string) "\n"))
 
 
-(define-obsolete-variable-alias 'bookmark-read-annotation-text-func
-  'bookmark-edit-annotation-text-func "23.1")
 (defvar bookmark-edit-annotation-text-func 'bookmark-default-annotation-text
   "Function to return default text to use for a bookmark annotation.
 It takes one argument, the name of the bookmark, as a string.")
@@ -1142,17 +1140,6 @@ bookmark-jump-other-frame
   (let ((pop-up-frames t))
     (bookmark-jump-other-window bookmark)))
 
-(defun bookmark-jump-noselect (bookmark)
-  "Return the location pointed to by BOOKMARK (see `bookmark-jump').
-The return value has the form (BUFFER . POINT).
-
-Note: this function is deprecated and is present for Emacs 22
-compatibility only."
-  (declare (obsolete bookmark-handle-bookmark "23.1"))
-  (save-excursion
-    (bookmark-handle-bookmark bookmark)
-    (cons (current-buffer) (point))))
-
 (defun bookmark-handle-bookmark (bookmark-name-or-record)
   "Call BOOKMARK-NAME-OR-RECORD's handler or `bookmark-default-handler'
 if it has none.  This changes current buffer and point and returns nil,
diff --git a/lisp/buff-menu.el b/lisp/buff-menu.el
index 655a76a713..9f8cdce4a2 100644
--- a/lisp/buff-menu.el
+++ b/lisp/buff-menu.el
@@ -214,9 +214,6 @@ Buffer-menu-mode-map
     map)
   "Local keymap for `Buffer-menu-mode' buffers.")
 
-(define-obsolete-variable-alias 'buffer-menu-mode-hook
-  'Buffer-menu-mode-hook "23.1")
-
 (define-derived-mode Buffer-menu-mode tabulated-list-mode "Buffer Menu"
   "Major mode for Buffer Menu buffers.
 The Buffer Menu is invoked by the commands \\[list-buffers],
diff --git a/lisp/cedet/semantic/wisent/wisent.el b/lisp/cedet/semantic/wisent/wisent.el
index d8a35d3e7d..3ec1cae020 100644
--- a/lisp/cedet/semantic/wisent/wisent.el
+++ b/lisp/cedet/semantic/wisent/wisent.el
@@ -55,10 +55,10 @@ wisent
 ;;;; Runtime stuff
 ;;;; -------------
 
-;;; Compatibility
+;;; XEmacs Compatibility
 (eval-and-compile
-  (if (fboundp 'char-valid-p)
-      (defalias 'wisent-char-p 'char-valid-p)
+  (if (fboundp 'characterp)
+      (defalias 'wisent-char-p 'characterp)
     (defalias 'wisent-char-p 'char-or-char-int-p)))
 
 ;;; Printed representation of terminals and nonterminals
diff --git a/lisp/cus-edit.el b/lisp/cus-edit.el
index 1ec2708550..171c27c945 100644
--- a/lisp/cus-edit.el
+++ b/lisp/cus-edit.el
@@ -4864,8 +4864,6 @@ Custom-goto-parent
 	       (parent (downcase (widget-get  button :tag))))
 	  (customize-group parent)))))
 
-(define-obsolete-variable-alias 'custom-mode-hook 'Custom-mode-hook "23.1")
-
 (defcustom Custom-mode-hook nil
   "Hook called when entering Custom mode."
   :type 'hook
@@ -4936,8 +4934,6 @@ Custom-mode
 
 (put 'Custom-mode 'mode-class 'special)
 
-(define-obsolete-function-alias 'custom-mode 'Custom-mode "23.1")
-
 ;;; The End.
 
 (provide 'cus-edit)
diff --git a/lisp/dirtrack.el b/lisp/dirtrack.el
index 3a0bbd2c9c..ad0c18d1b3 100644
--- a/lisp/dirtrack.el
+++ b/lisp/dirtrack.el
@@ -196,9 +196,6 @@ dirtrack-mode
     (remove-hook 'comint-preoutput-filter-functions 'dirtrack t)))
 
 
-(define-obsolete-function-alias 'dirtrack-debug-toggle 'dirtrack-debug-mode
-  "23.1")
-(define-obsolete-variable-alias 'dirtrack-debug 'dirtrack-debug-mode "23.1")
 (define-minor-mode dirtrack-debug-mode
   "Toggle Dirtrack debugging."
   nil nil nil
diff --git a/lisp/emacs-lisp/crm.el b/lisp/emacs-lisp/crm.el
index 65483d0813..89d106ee48 100644
--- a/lisp/emacs-lisp/crm.el
+++ b/lisp/emacs-lisp/crm.el
@@ -270,12 +270,6 @@ completing-read-multiple
     (remove-hook 'choose-completion-string-functions
 		 'crm--choose-completion-string)))
 
-(define-obsolete-function-alias 'crm-minibuffer-complete 'crm-complete "23.1")
-(define-obsolete-function-alias
-  'crm-minibuffer-completion-help 'crm-completion-help "23.1")
-(define-obsolete-function-alias
-  'crm-minibuffer-complete-and-exit 'crm-complete-and-exit "23.1")
-
 ;; testing and debugging
 ;; (defun crm-init-test-environ ()
 ;;   "Set up some variables for testing."
diff --git a/lisp/emacs-lisp/easymenu.el b/lisp/emacs-lisp/easymenu.el
index 6ba8b997f8..73dabef3fa 100644
--- a/lisp/emacs-lisp/easymenu.el
+++ b/lisp/emacs-lisp/easymenu.el
@@ -29,16 +29,6 @@
 
 ;;; Code:
 
-(defvar easy-menu-precalculate-equivalent-keybindings nil
-  "Determine when equivalent key bindings are computed for easy-menu menus.
-It can take some time to calculate the equivalent key bindings that are shown
-in a menu.  If the variable is on, then this calculation gives a (maybe
-noticeable) delay when a mode is first entered.  If the variable is off, then
-this delay will come when a menu is displayed the first time.  If you never use
-menus, turn this variable off, otherwise it is probably better to keep it on.")
-(make-obsolete-variable
- 'easy-menu-precalculate-equivalent-keybindings nil "23.1")
-
 (defsubst easy-menu-intern (s)
   (if (stringp s) (intern s) s))
 
diff --git a/lisp/emacs-lisp/lisp-mode.el b/lisp/emacs-lisp/lisp-mode.el
index 7098a41f27..8d3b357a7f 100644
--- a/lisp/emacs-lisp/lisp-mode.el
+++ b/lisp/emacs-lisp/lisp-mode.el
@@ -785,8 +785,6 @@ lisp-comment-indent
             nil)))
       (comment-indent-default)))
 
-(define-obsolete-function-alias 'lisp-mode-auto-fill 'do-auto-fill "23.1")
-
 (defcustom lisp-indent-offset nil
   "If non-nil, indent second line of expressions that many more columns."
   :group 'lisp
diff --git a/lisp/epa.el b/lisp/epa.el
index 8ec4218735..68c472f3f5 100644
--- a/lisp/epa.el
+++ b/lisp/epa.el
@@ -631,10 +631,6 @@ epa-display-error
 	  (goto-char (point-min)))
 	(display-buffer buffer)))))
 
-(defun epa-display-verify-result (verify-result)
-  (declare (obsolete epa-display-info "23.1"))
-  (epa-display-info (epg-verify-result-to-string verify-result)))
-
 (defun epa-passphrase-callback-function (context key-id handback)
   (if (eq key-id 'SYM)
       (read-passwd
diff --git a/lisp/epg.el b/lisp/epg.el
index 222fd913e1..603415f0ec 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -1234,19 +1234,6 @@ epg--status-IMPORT_RES
 			     (epg-context-result-for context 'import-status)))
     (epg-context-set-result-for context 'import-status nil)))
 
-(defun epg-passphrase-callback-function (context key-id _handback)
-  (declare (obsolete epa-passphrase-callback-function "23.1"))
-  (if (eq key-id 'SYM)
-      (read-passwd "Passphrase for symmetric encryption: "
-		   (eq (epg-context-operation context) 'encrypt))
-    (read-passwd
-     (if (eq key-id 'PIN)
-	"Passphrase for PIN: "
-       (let ((entry (assoc key-id epg-user-id-alist)))
-	 (if entry
-	     (format "Passphrase for %s %s: " key-id (cdr entry))
-	   (format "Passphrase for %s: " key-id)))))))
-
 (defun epg--list-keys-1 (context name mode)
   (let ((args (append (if (epg-context-home-directory context)
 			  (list "--homedir"
diff --git a/lisp/eshell/eshell.el b/lisp/eshell/eshell.el
index 2a63882ff0..7f94c9ba63 100644
--- a/lisp/eshell/eshell.el
+++ b/lisp/eshell/eshell.el
@@ -380,15 +380,6 @@ eshell-command-result
 	      (set status-var eshell-last-command-status))
 	  (cadr result))))))
 
-;;;_* Reporting bugs
-;;
-;; If you do encounter a bug, on any system, please report
-;; it -- in addition to any particular oddities in your configuration
-;; -- so that the problem may be corrected for the benefit of others.
-
-;;;###autoload
-(define-obsolete-function-alias 'eshell-report-bug 'report-emacs-bug "23.1")
-
 ;;; Code:
 
 (defun eshell-unload-all-modules ()
diff --git a/lisp/ffap.el b/lisp/ffap.el
index d656b37372..c26fb5420b 100644
--- a/lisp/ffap.el
+++ b/lisp/ffap.el
@@ -1814,12 +1814,6 @@ ffap-literally
 
 (defalias 'find-file-literally-at-point 'ffap-literally)
 
-\f
-;;; Bug Reporter:
-
-(define-obsolete-function-alias 'ffap-bug 'report-emacs-bug "23.1")
-(define-obsolete-function-alias 'ffap-submit-bug 'report-emacs-bug "23.1")
-
 \f
 ;;; Hooks for Gnus, VM, Rmail:
 ;;
diff --git a/lisp/files.el b/lisp/files.el
index dba704f7a4..c2f37c178d 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -972,14 +972,6 @@ locate-file-completion-table
       (completion-table-with-context
        string-dir names string-file pred action)))))
 
-(defun locate-file-completion (string path-and-suffixes action)
-  "Do completion for file names passed to `locate-file'.
-PATH-AND-SUFFIXES is a pair of lists, (DIRECTORIES . SUFFIXES)."
-  (declare (obsolete locate-file-completion-table "23.1"))
-  (locate-file-completion-table (car path-and-suffixes)
-                                (cdr path-and-suffixes)
-                                string nil action))
-
 (defvar locate-dominating-stop-dir-regexp
   (purecopy "\\`\\(?:[\\/][\\/][^\\/]+[\\/]\\|/\\(?:net\\|afs\\|\\.\\.\\.\\)/\\)\\'")
   "Regexp of directory names that stop the search in `locate-dominating-file'.
diff --git a/lisp/gnus/mml-smime.el b/lisp/gnus/mml-smime.el
index 4754f37a2d..acddb30033 100644
--- a/lisp/gnus/mml-smime.el
+++ b/lisp/gnus/mml-smime.el
@@ -329,7 +329,6 @@ password-cache-expiry
   (autoload 'epg-verify-string "epg")
   (autoload 'epg-sign-string "epg")
   (autoload 'epg-encrypt-string "epg")
-  (autoload 'epg-passphrase-callback-function "epg")
   (autoload 'epg-context-set-passphrase-callback "epg")
   (autoload 'epg-sub-key-fingerprint "epg")
   (autoload 'epg-configuration "epg-config")
diff --git a/lisp/gnus/mml1991.el b/lisp/gnus/mml1991.el
index 8be1b84e52..88864ea357 100644
--- a/lisp/gnus/mml1991.el
+++ b/lisp/gnus/mml1991.el
@@ -242,7 +242,6 @@ mml1991-pgg-encrypt
 (defvar epg-user-id-alist)
 
 (autoload 'epg-make-context "epg")
-(autoload 'epg-passphrase-callback-function "epg")
 (autoload 'epa-select-keys "epa")
 (autoload 'epg-list-keys "epg")
 (autoload 'epg-context-set-armor "epg")
diff --git a/lisp/gnus/mml2015.el b/lisp/gnus/mml2015.el
index d1d150ad2e..45c9bbfe90 100644
--- a/lisp/gnus/mml2015.el
+++ b/lisp/gnus/mml2015.el
@@ -712,7 +712,6 @@ inhibit-redisplay
 (autoload 'epg-verify-string "epg")
 (autoload 'epg-sign-string "epg")
 (autoload 'epg-encrypt-string "epg")
-(autoload 'epg-passphrase-callback-function "epg")
 (autoload 'epg-context-set-passphrase-callback "epg")
 (autoload 'epg-key-sub-key-list "epg")
 (autoload 'epg-sub-key-capability "epg")
diff --git a/lisp/hi-lock.el b/lisp/hi-lock.el
index a18310322a..27bd316621 100644
--- a/lisp/hi-lock.el
+++ b/lisp/hi-lock.el
@@ -237,17 +237,11 @@ hi-lock-interactive-lighters
   "Human-readable lighters for `hi-lock-interactive-patterns'.")
 (put 'hi-lock-interactive-lighters 'permanent-local t)
 
-(define-obsolete-variable-alias 'hi-lock-face-history
-                                'hi-lock-face-defaults "23.1")
 (defvar hi-lock-face-defaults
   '("hi-yellow" "hi-pink" "hi-green" "hi-blue" "hi-salmon" "hi-aquamarine"
     "hi-black-b" "hi-blue-b" "hi-red-b" "hi-green-b" "hi-black-hb")
   "Default faces for hi-lock interactive functions.")
 
-(define-obsolete-variable-alias 'hi-lock-regexp-history
-                                'regexp-history
-                                "23.1")
-
 (defvar hi-lock-file-patterns-prefix "Hi-lock"
   "String used to identify hi-lock patterns at the start of files.")
 
diff --git a/lisp/hilit-chg.el b/lisp/hilit-chg.el
index 04a5ccd8d5..ae97bb008a 100644
--- a/lisp/hilit-chg.el
+++ b/lisp/hilit-chg.el
@@ -224,9 +224,6 @@ highlight-changes-colors
 ;; When you invoke highlight-changes-mode, should highlight-changes-visible-mode
 ;; be on or off?
 
-(define-obsolete-variable-alias 'highlight-changes-initial-state
-  'highlight-changes-visibility-initial-state "23.1")
-
 (defcustom highlight-changes-visibility-initial-state t
   "Controls whether changes are initially visible in Highlight Changes mode.
 
@@ -236,13 +233,7 @@ highlight-changes-visibility-initial-state
   :type 'boolean
   :group 'highlight-changes)
 
-;; highlight-changes-global-initial-state has been removed
-
-
-
 ;; These are the strings displayed in the mode-line for the minor mode:
-(define-obsolete-variable-alias 'highlight-changes-active-string
-  'highlight-changes-visible-string "23.1")
 
 (defcustom highlight-changes-visible-string " +Chg"
   "The string used when in Highlight Changes mode and changes are visible.
@@ -252,9 +243,6 @@ highlight-changes-visible-string
 		 (const :tag "None"  nil))
   :group 'highlight-changes)
 
-(define-obsolete-variable-alias 'highlight-changes-passive-string
-  'highlight-changes-invisible-string "23.1")
-
 (defcustom highlight-changes-invisible-string " -Chg"
   "The string used when in Highlight Changes mode and changes are hidden.
 This should be set to nil if no indication is desired, or to
@@ -957,10 +945,6 @@ hilit-chg-get-diff-list-hk
 (define-globalized-minor-mode global-highlight-changes-mode
   highlight-changes-mode highlight-changes-mode-turn-on)
 
-(define-obsolete-function-alias
- 'global-highlight-changes
-  'global-highlight-changes-mode "23.1")
-
 (defun highlight-changes-mode-turn-on ()
   "See if Highlight Changes mode should be turned on for this buffer.
 This is called when `global-highlight-changes-mode' is turned on."
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 7714a778fc..5fe931dd9b 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -2968,11 +2968,6 @@ unify-8859-on-decoding-mode
 ;; Doc said "obsolete" in 23.1, this statement only added in 24.1.
 (make-obsolete 'unify-8859-on-decoding-mode "don't use it." "23.1")
 
-(defvar nonascii-insert-offset 0)
-(make-obsolete-variable 'nonascii-insert-offset "do not use it." "23.1")
-(defvar nonascii-translation-table nil)
-(make-obsolete-variable 'nonascii-translation-table "do not use it." "23.1")
-
 (defvar ucs-names nil
   "Hash table of cached CHAR-NAME keys to CHAR-CODE values.")
 
diff --git a/lisp/international/mule-diag.el b/lisp/international/mule-diag.el
index 80e78ef787..b13bde58ca 100644
--- a/lisp/international/mule-diag.el
+++ b/lisp/international/mule-diag.el
@@ -200,10 +200,6 @@ list-character-sets-2
 ;;;		   (charset-iso-graphic-plane charset)
 		   (charset-description charset)))))
 
-(defvar non-iso-charset-alist nil
-  "Obsolete.")
-(make-obsolete-variable 'non-iso-charset-alist "no longer relevant." "23.1")
-
 ;; A variable to hold charset input history.
 (defvar charset-history nil)
 
diff --git a/lisp/international/mule-util.el b/lisp/international/mule-util.el
index 5cc10b1315..660ac58e02 100644
--- a/lisp/international/mule-util.el
+++ b/lisp/international/mule-util.el
@@ -274,15 +274,6 @@ with-coding-priority
 ;;;###autoload(put 'with-coding-priority 'lisp-indent-function 1)
 (put 'with-coding-priority 'edebug-form-spec t)
 
-;;;###autoload
-(defmacro detect-coding-with-priority (from to priority-list)
-  "Detect a coding system of the text between FROM and TO with PRIORITY-LIST.
-PRIORITY-LIST is an alist of coding categories vs the corresponding
-coding systems ordered by priority."
-  (declare (obsolete with-coding-priority "23.1"))
-  `(with-coding-priority (mapcar #'cdr ,priority-list)
-     (detect-coding-region ,from ,to)))
-
 ;;;###autoload
 (defun detect-coding-with-language-environment (from to lang-env)
   "Detect a coding system for the text between FROM and TO with LANG-ENV.
diff --git a/lisp/international/mule.el b/lisp/international/mule.el
index 72e8cad9d6..411cc6f41d 100644
--- a/lisp/international/mule.el
+++ b/lisp/international/mule.el
@@ -408,16 +408,6 @@ charset-info
 ;; because that makes a bootstrapping problem
 ;; if you need to recompile all the Lisp files using interpreted code.
 
-(defun charset-id (_charset)
-  "Always return 0.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  0)
-
-(defmacro charset-bytes (_charset)
-  "Always return 0.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  0)
-
 (defun get-charset-property (charset propname)
   "Return the value of CHARSET's PROPNAME property.
 This is the last value stored with
@@ -463,19 +453,8 @@ charset-long-name
   "Return long name of CHARSET."
   (plist-get (charset-plist charset) :long-name))
 
-(defun charset-list ()
-  "Return list of all charsets ever defined."
-  (declare (obsolete charset-list "23.1"))
-  charset-list)
-
 \f
 ;;; CHARACTER
-(define-obsolete-function-alias 'char-valid-p 'characterp "23.1")
-
-(defun generic-char-p (_char)
-  "Always return nil.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  nil)
 
 (defun make-char-internal (charset-id &optional code1 code2)
   (let ((charset (aref emacs-mule-charset-table charset-id)))
@@ -1084,10 +1063,6 @@ coding-system-list
 	      (setq codings (cons alias codings))))))
     codings))
 
-(defconst char-coding-system-table nil
-  "It exists just for backward compatibility, and the value is always nil.")
-(make-obsolete-variable 'char-coding-system-table nil "23.1")
-
 (defun transform-make-coding-system-args (name type &optional doc-string props)
   "For internal use only.
 Transform XEmacs style args for `make-coding-system' to Emacs style.
@@ -1169,106 +1144,6 @@ transform-make-coding-system-args
       (error "unsupported XEmacs style make-coding-style arguments: %S"
 	     `(,name ,type ,doc-string ,props))))))
 
-(defun make-coding-system (coding-system type mnemonic doc-string
-					 &optional
-					 flags
-					 properties
-					 eol-type)
-  "Define a new coding system CODING-SYSTEM (symbol).
-This function is provided for backward compatibility."
-  (declare (obsolete define-coding-system "23.1"))
-  ;; For compatibility with XEmacs, we check the type of TYPE.  If it
-  ;; is a symbol, perhaps, this function is called with XEmacs-style
-  ;; arguments.  Here, try to transform that kind of arguments to
-  ;; Emacs style.
-  (if (symbolp type)
-      (let ((args (transform-make-coding-system-args coding-system type
-						     mnemonic doc-string)))
-	(setq coding-system (car args)
-	      type (nth 1 args)
-	      mnemonic (nth 2 args)
-	      doc-string (nth 3 args)
-	      flags (nth 4 args)
-	      properties (nth 5 args)
-	      eol-type (nth 6 args))))
-
-  (setq type
-	(cond ((eq type 0) 'emacs-mule)
-	      ((eq type 1) 'shift-jis)
-	      ((eq type 2) 'iso2022)
-	      ((eq type 3) 'big5)
-	      ((eq type 4) 'ccl)
-	      ((eq type 5) 'raw-text)
-	      (t
-	       (error "Invalid coding system type: %s" type))))
-
-  (setq properties
-	(let ((plist nil) key)
-	  (dolist (elt properties)
-	    (setq key (car elt))
-	    (cond ((eq key 'post-read-conversion)
-		   (setq key :post-read-conversion))
-		  ((eq key 'pre-write-conversion)
-		   (setq key :pre-write-conversion))
-		  ((eq key 'translation-table-for-decode)
-		   (setq key :decode-translation-table))
-		  ((eq key 'translation-table-for-encode)
-		   (setq key :encode-translation-table))
-		  ((eq key 'safe-charsets)
-		   (setq key :charset-list))
-		  ((eq key 'mime-charset)
-		   (setq key :mime-charset))
-		  ((eq key 'valid-codes)
-		   (setq key :valids)))
-	    (setq plist (plist-put plist key (cdr elt))))
-	  plist))
-  (setq properties (plist-put properties :mnemonic mnemonic))
-  (plist-put properties :coding-type type)
-  (cond ((eq eol-type 0) (setq eol-type 'unix))
-	((eq eol-type 1) (setq eol-type 'dos))
-	((eq eol-type 2) (setq eol-type 'mac))
-	((vectorp eol-type) (setq eol-type nil)))
-  (plist-put properties :eol-type eol-type)
-
-  (cond
-   ((eq type 'iso2022)
-    (plist-put properties :flags
-	       (list (and (or (consp (nth 0 flags))
-			      (consp (nth 1 flags))
-			      (consp (nth 2 flags))
-			      (consp (nth 3 flags))) 'designation)
-		     (or (nth 4 flags) 'long-form)
-		     (and (nth 5 flags) 'ascii-at-eol)
-		     (and (nth 6 flags) 'ascii-at-cntl)
-		     (and (nth 7 flags) '7-bit)
-		     (and (nth 8 flags) 'locking-shift)
-		     (and (nth 9 flags) 'single-shift)
-		     (and (nth 10 flags) 'use-roman)
-		     (and (nth 11 flags) 'use-oldjis)
-		     (or (nth 12 flags) 'direction)
-		     (and (nth 13 flags) 'init-at-bol)
-		     (and (nth 14 flags) 'designate-at-bol)
-		     (and (nth 15 flags) 'safe)
-		     (and (nth 16 flags) 'latin-extra)))
-    (plist-put properties :designation
-	       (let ((vec (make-vector 4 nil)))
-		 (dotimes (i 4)
-		   (let ((spec (nth i flags)))
-		     (if (eq spec t)
-			 (aset vec i '(94 96))
-		     (if (consp spec)
-			 (progn
-			   (if (memq t spec)
-			       (setq spec (append (delq t spec) '(94 96))))
-			   (aset vec i spec))))))
-		 vec)))
-
-   ((eq type 'ccl)
-    (plist-put properties :ccl-decoder (car flags))
-    (plist-put properties :ccl-encoder (cdr flags))))
-
-  (apply 'define-coding-system coding-system doc-string properties))
-
 (defun merge-coding-systems (first second)
   "Fill in any unspecified aspects of coding system FIRST from SECOND.
 Return the resulting coding system."
@@ -1615,15 +1490,6 @@ set-next-selection-coding-system
 
   (setq next-selection-coding-system coding-system))
 
-(defun set-coding-priority (arg)
-  "Set priority of coding categories according to ARG.
-ARG is a list of coding categories ordered by priority.
-
-This function is provided for backward compatibility."
-  (declare (obsolete set-coding-system-priority "23.1"))
-  (apply 'set-coding-system-priority
-	 (mapcar #'(lambda (x) (symbol-value x)) arg)))
-
 ;;; X selections
 
 (defvar ctext-non-standard-encodings-alist
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index 44cde7cb5a..312baffb90 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -521,25 +521,6 @@ rmail-mmdf-delim1
 (defvar rmail-mmdf-delim2 "^\001\001\001\001\n"
   "Regexp marking the end of an mmdf message.")
 
-;; FIXME Post-mbox, this is now unused.
-;; In Emacs-22, this was called:
-;;  i) the very first time a message was shown.
-;; ii) when toggling the headers to the normal state, every time.
-;; It's not clear what it should do now, since there is nothing that
-;; records when a message is shown for the first time (unseen is not
-;; necessarily the same thing).
-;; See https://lists.gnu.org/r/emacs-devel/2009-03/msg00013.html
-(defcustom rmail-message-filter nil
-  "If non-nil, a filter function for new messages in RMAIL.
-Called with region narrowed to the message, including headers,
-before obeying `rmail-ignored-headers'."
-  :group 'rmail-headers
-  :type '(choice (const nil) function))
-
-(make-obsolete-variable 'rmail-message-filter
-			"it is not used (try `rmail-show-message-hook')."
-			"23.1")
-
 (defcustom rmail-automatic-folder-directives nil
   "List of directives specifying how to automatically file messages.
 Whenever Rmail shows a message in the folder that `rmail-file-name'
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index d2c3f9045e..0d99f4687c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -685,13 +685,6 @@ completion--twq-all
                completions)
        qboundary))))
 
-;; (defmacro complete-in-turn (a b) `(completion-table-in-turn ,a ,b))
-;; (defmacro dynamic-completion-table (fun) `(completion-table-dynamic ,fun))
-(define-obsolete-function-alias
-  'complete-in-turn #'completion-table-in-turn "23.1")
-(define-obsolete-function-alias
-  'dynamic-completion-table #'completion-table-dynamic "23.1")
-
 ;;; Minibuffer completion
 
 (defgroup minibuffer nil
@@ -1770,9 +1763,6 @@ completion--insert-strings
 			      ;; Round up to a whole number of columns.
 			      (* colwidth (ceiling length colwidth))))))))))))
 
-(defvar completion-common-substring nil)
-(make-obsolete-variable 'completion-common-substring nil "23.1")
-
 (defvar completion-setup-hook nil
   "Normal hook run at the end of setting up a completion list buffer.
 When this hook is run, the current buffer is the one in which the
@@ -1864,11 +1854,7 @@ display-completion-list
         (insert "Possible completions are:\n")
         (completion--insert-strings completions))))
 
-  ;; The hilit used to be applied via completion-setup-hook, so there
-  ;; may still be some code that uses completion-common-substring.
-  (with-no-warnings
-    (let ((completion-common-substring common-substring))
-      (run-hooks 'completion-setup-hook)))
+  (run-hooks 'completion-setup-hook)
   nil)
 
 (defvar completion-extra-properties nil
@@ -2374,8 +2360,6 @@ minibuffer-local-filename-completion-map
 Gets combined either with `minibuffer-local-completion-map' or
 with `minibuffer-local-must-match-map'.")
 
-(define-obsolete-variable-alias 'minibuffer-local-must-match-filename-map
-  'minibuffer-local-filename-must-match-map "23.1")
 (defvar minibuffer-local-filename-must-match-map (make-sparse-keymap))
 (make-obsolete-variable 'minibuffer-local-filename-must-match-map nil "24.1")
 
diff --git a/lisp/mouse.el b/lisp/mouse.el
index 795b4da19e..ae6ebf4dfc 100644
--- a/lisp/mouse.el
+++ b/lisp/mouse.el
@@ -271,34 +271,6 @@ mouse-menu-bar-map
            local-menu
            minor-mode-menus)))
 
-(defun mouse-major-mode-menu (event &optional prefix)
-  "Pop up a mode-specific menu of mouse commands.
-Default to the Edit menu if the major mode doesn't define a menu."
-  (declare (obsolete mouse-menu-major-mode-map "23.1"))
-  (interactive "@e\nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu (mouse-menu-major-mode-map) event prefix))
-
-(defun mouse-popup-menubar (event prefix)
-  "Pop up a menu equivalent to the menu bar for keyboard EVENT with PREFIX.
-The contents are the items that would be in the menu bar whether or
-not it is actually displayed."
-  (declare (obsolete mouse-menu-bar-map "23.1"))
-  (interactive "@e \nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu (mouse-menu-bar-map) (unless (integerp event) event) prefix))
-
-(defun mouse-popup-menubar-stuff (event prefix)
-  "Popup a menu like either `mouse-major-mode-menu' or `mouse-popup-menubar'.
-Use the former if the menu bar is showing, otherwise the latter."
-  (declare (obsolete nil "23.1"))
-  (interactive "@e\nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu
-   (if (zerop (or (frame-parameter nil 'menu-bar-lines) 0))
-       (mouse-menu-bar-map)
-     (mouse-menu-major-mode-map))
-   event prefix))
 \f
 ;; Commands that operate on windows.
 
diff --git a/lisp/net/newst-treeview.el b/lisp/net/newst-treeview.el
index 1bed61f3e7..ff8a447c7c 100644
--- a/lisp/net/newst-treeview.el
+++ b/lisp/net/newst-treeview.el
@@ -131,14 +131,6 @@ newsticker-groups
 Example: (\"Topmost group\" \"feed1\" (\"subgroup1\" \"feed 2\")
 \"feed3\")")
 
-(defcustom newsticker-groups-filename
-  nil
-  "Name of the newsticker groups settings file."
-  :version "25.1"                       ; changed default value to nil
-  :type '(choice (const nil) string)
-  :group 'newsticker-treeview)
-(make-obsolete-variable 'newsticker-groups-filename 'newsticker-dir "23.1")
-
 ;; ======================================================================
 ;;; internal variables
 ;; ======================================================================
@@ -1265,29 +1257,9 @@ newsticker-treeview-save
 (defun newsticker--treeview-load ()
   "Load treeview settings."
   (let* ((coding-system-for-read 'utf-8)
-         (filename
-          (or (and newsticker-groups-filename
-                   (not (string=
-                         (expand-file-name newsticker-groups-filename)
-                         (expand-file-name (concat newsticker-dir "/groups"))))
-                   (file-exists-p newsticker-groups-filename)
-                   (y-or-n-p
-                    (format-message
-                     (concat "Obsolete variable `newsticker-groups-filename' "
-                             "points to existing file \"%s\".\n"
-                             "Read it? ")
-                     newsticker-groups-filename))
-                   newsticker-groups-filename)
-              (concat newsticker-dir "/groups")))
+         (filename (concat newsticker-dir "/groups"))
          (buf (and (file-exists-p filename)
                    (find-file-noselect filename))))
-    (and newsticker-groups-filename
-         (file-exists-p newsticker-groups-filename)
-	 (y-or-n-p (format-message
-                    (concat "Delete the file \"%s\",\nto which the obsolete "
-                            "variable `newsticker-groups-filename' points ? ")
-                    newsticker-groups-filename))
-	 (delete-file newsticker-groups-filename))
     (when buf
       (set-buffer buf)
       (goto-char (point-min))
diff --git a/lisp/obsolete/tpu-edt.el b/lisp/obsolete/tpu-edt.el
index d71f79c87b..0de7aa096d 100644
--- a/lisp/obsolete/tpu-edt.el
+++ b/lisp/obsolete/tpu-edt.el
@@ -287,14 +287,6 @@ tpu-version
 ;;;
 ;;;  User Configurable Variables
 ;;;
-(defcustom tpu-have-ispell t
-  "Non-nil means `tpu-spell-check' uses `ispell-region' for spell checking.
-Otherwise, use `spell-region'."
-  :type 'boolean
-  :group 'tpu)
-(make-obsolete-variable 'tpu-have-ispell "the `spell' package is obsolete."
-                        "23.1")
-
 (defcustom tpu-kill-buffers-silently nil
   "If non-nil, TPU-edt kills modified buffers without asking."
   :type 'boolean
@@ -315,7 +307,6 @@ tpu-pan-columns
 ;;;  Global Keymaps
 ;;;
 
-(define-obsolete-variable-alias 'GOLD-map 'tpu-gold-map "23.1")
 (defvar tpu-gold-map
   (let ((map (make-keymap)))
     ;; Previously we used escape sequences here.  We now instead presume
@@ -892,8 +883,7 @@ tpu-spell-check
 if no region is selected."
   (interactive)
   (let ((m (tpu-mark)))
-    (apply (if tpu-have-ispell 'ispell-region
-             'spell-region)
+    (apply 'ispell-region
            (if m
                (if (> m (point)) (list (point) m)
                  (list m (point)))
diff --git a/lisp/obsolete/vc-arch.el b/lisp/obsolete/vc-arch.el
index 93bd991eb3..6b34b31fcd 100644
--- a/lisp/obsolete/vc-arch.el
+++ b/lisp/obsolete/vc-arch.el
@@ -84,8 +84,6 @@ vc-arch-diff-switches
   :version "23.1"
   :group 'vc-arch)
 
-(define-obsolete-variable-alias 'vc-arch-command 'vc-arch-program "23.1")
-
 (defcustom vc-arch-program
   (let ((candidates '("tla" "baz")))
     (while (and candidates (not (executable-find (car candidates))))
diff --git a/lisp/password-cache.el b/lisp/password-cache.el
index 5e5f3240bc..5aed4e4049 100644
--- a/lisp/password-cache.el
+++ b/lisp/password-cache.el
@@ -93,22 +93,6 @@ password-read
   (or (password-read-from-cache key)
       (read-passwd prompt)))
 
-(defun password-read-and-add (prompt &optional key)
-  "Read password, for use with KEY, from user, or from cache if wanted.
-Then store the password in the cache.  Uses `password-read' and
-`password-cache-add'.  Custom variables `password-cache' and
-`password-cache-expiry' regulate cache behavior.
-
-Warning: the password is cached without checking that it is
-correct.  It is better to check the password before caching.  If
-you must use this function, take care to check passwords and
-remove incorrect ones from the cache."
-  (declare (obsolete password-read "23.1"))
-  (let ((password (password-read prompt key)))
-    (when (and password key)
-      (password-cache-add key password))
-    password))
-
 (defun password-cache-remove (key)
   "Remove password indexed by KEY from password cache.
 This is typically run by a timer setup from `password-cache-add',
diff --git a/lisp/progmodes/idlwave.el b/lisp/progmodes/idlwave.el
index 3092d4c45b..55acb77425 100644
--- a/lisp/progmodes/idlwave.el
+++ b/lisp/progmodes/idlwave.el
@@ -159,8 +159,6 @@
   (defalias 'line-beginning-position 'point-at-bol))
 (unless (fboundp 'line-end-position)
   (defalias 'line-end-position 'point-at-eol))
-(unless (fboundp 'char-valid-p)
-  (defalias 'char-valid-p 'characterp))
 (unless (fboundp 'match-string-no-properties)
   (defalias 'match-string-no-properties 'match-string))
 
diff --git a/lisp/shell.el b/lisp/shell.el
index 1e2679f723..3461ce71ef 100644
--- a/lisp/shell.el
+++ b/lisp/shell.el
@@ -988,9 +988,6 @@ shell-dirtrack-mode
       (add-hook 'comint-input-filter-functions #'shell-directory-tracker nil t)
     (remove-hook 'comint-input-filter-functions #'shell-directory-tracker t)))
 
-(define-obsolete-function-alias 'shell-dirtrack-toggle #'shell-dirtrack-mode
-  "23.1")
-
 (defun shell-cd (dir)
   "Do normal `cd' to DIR, and set `list-buffers-directory'."
   (cd dir)
diff --git a/lisp/subr.el b/lisp/subr.el
index 324c59f13f..b7ef32d5d5 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1578,11 +1578,6 @@ posn-object-width-height
 (make-obsolete 'string-as-multibyte "use `decode-coding-string'." "26.1")
 (make-obsolete 'string-make-multibyte "use `decode-coding-string'." "26.1")
 
-(defun forward-point (n)
-  "Return buffer position N characters after (before if N negative) point."
-  (declare (obsolete "use (+ (point) N) instead." "23.1"))
-  (+ (point) n))
-
 (defun log10 (x)
   "Return (log X 10), the log base 10 of X."
   (declare (obsolete log "24.4"))
@@ -1598,17 +1593,11 @@ log10
 \f
 ;;;; Obsolescence declarations for variables, and aliases.
 
-(make-obsolete-variable 'define-key-rebound-commands nil "23.2")
-(make-obsolete-variable 'redisplay-end-trigger-functions 'jit-lock-register "23.1")
 (make-obsolete-variable 'deferred-action-list 'post-command-hook "24.1")
 (make-obsolete-variable 'deferred-action-function 'post-command-hook "24.1")
 (make-obsolete-variable 'redisplay-dont-pause nil "24.5")
-(make-obsolete 'window-redisplay-end-trigger nil "23.1")
-(make-obsolete 'set-window-redisplay-end-trigger nil "23.1")
 
 (make-obsolete 'run-window-configuration-change-hook nil "27.1")
-(make-obsolete 'process-filter-multibyte-p nil "23.1")
-(make-obsolete 'set-process-filter-multibyte nil "23.1")
 
 (make-obsolete-variable 'command-debug-status
                         "expect it to be removed in a future version." "25.2")
diff --git a/lisp/t-mouse.el b/lisp/t-mouse.el
index a1af53d8c4..4feab71401 100644
--- a/lisp/t-mouse.el
+++ b/lisp/t-mouse.el
@@ -62,8 +62,6 @@ gpm-mouse-disable
     (gpm-mouse-stop))
   (set-terminal-parameter nil 'gpm-mouse-active nil))
 
-;;;###autoload
-(define-obsolete-function-alias 't-mouse-mode 'gpm-mouse-mode "23.1")
 ;;;###autoload
 (define-minor-mode gpm-mouse-mode
   "Toggle mouse support in GNU/Linux consoles (GPM Mouse mode).
diff --git a/lisp/term/w32-win.el b/lisp/term/w32-win.el
index 5901e0295e..e866fdc36c 100644
--- a/lisp/term/w32-win.el
+++ b/lisp/term/w32-win.el
@@ -78,12 +78,8 @@
 (require 'dnd)
 (require 'w32-vars)
 
-;; Keep an obsolete alias for w32-focus-frame and w32-select-font in case
-;; they are used by code outside Emacs.
-(define-obsolete-function-alias 'w32-focus-frame 'x-focus-frame "23.1")
 (declare-function x-select-font "w32font.c"
                   (&optional frame exclude-proportional))
-(define-obsolete-function-alias 'w32-select-font 'x-select-font "23.1")
 
 (defvar w32-color-map) ;; defined in w32fns.c
 (make-obsolete 'w32-default-color-map nil "24.1")
diff --git a/lisp/textmodes/ispell.el b/lisp/textmodes/ispell.el
index 65f61644b6..ffcdb9bd16 100644
--- a/lisp/textmodes/ispell.el
+++ b/lisp/textmodes/ispell.el
@@ -621,15 +621,6 @@ ispell-encoding8-command
 Earlier Aspell versions do not consistently support charset encoding.  Handling
 this would require some extra guessing in `ispell-aspell-find-dictionary'.")
 
-(defvar ispell-aspell-supports-utf8 nil
-  "Non-nil if Aspell has consistent command line UTF-8 support.  Obsolete.
-ispell.el and flyspell.el will use for this purpose the more generic
-variable `ispell-encoding8-command' for both Aspell and Hunspell.  Is left
-here just for backwards compatibility.")
-
-(make-obsolete-variable 'ispell-aspell-supports-utf8
-                        'ispell-encoding8-command "23.1")
-
 (defvar ispell-dicts-name2locale-equivs-alist
   '(("american"      "en_US")
     ("brasileiro"    "pt_BR")
diff --git a/lisp/textmodes/remember.el b/lisp/textmodes/remember.el
index 279dbb4450..7bc7dc1762 100644
--- a/lisp/textmodes/remember.el
+++ b/lisp/textmodes/remember.el
@@ -487,9 +487,6 @@ remember-finalize
   (interactive)
   (remember-region (point-min) (point-max)))
 
-;; Org needs this
-(define-obsolete-function-alias 'remember-buffer 'remember-finalize "23.1")
-
 (defun remember-destroy ()
   "Destroy the current *Remember* buffer."
   (interactive)
diff --git a/lisp/tooltip.el b/lisp/tooltip.el
index f35f6b9a03..5f5a4788b2 100644
--- a/lisp/tooltip.el
+++ b/lisp/tooltip.el
@@ -167,8 +167,6 @@ tooltip-resize-echo-area
 \f
 ;;; Variables that are not customizable.
 
-(define-obsolete-variable-alias 'tooltip-hook 'tooltip-functions "23.1")
-
 (defvar tooltip-functions nil
   "Functions to call to display tooltips.
 Each function is called with one argument EVENT which is a copy
diff --git a/lisp/url/url-util.el b/lisp/url/url-util.el
index 6dd7a9c2aa..0a7e7e205e 100644
--- a/lisp/url/url-util.el
+++ b/lisp/url/url-util.el
@@ -569,31 +569,6 @@ url-get-url-at-point
 	  (setq url nil))
       url)))
 
-(defun url-generate-unique-filename (&optional fmt)
-  "Generate a unique filename in `url-temporary-directory'."
-  (declare (obsolete make-temp-file "23.1"))
-  ;; This variable is obsolete, but so is this function.
-  (let ((tempdir (with-no-warnings url-temporary-directory)))
-    (if (not fmt)
-	(let ((base (format "url-tmp.%d" (user-real-uid)))
-	      (fname "")
-	      (x 0))
-	  (setq fname (format "%s%d" base x))
-	  (while (file-exists-p
-		  (expand-file-name fname tempdir))
-	    (setq x (1+ x)
-		  fname (concat base (int-to-string x))))
-	  (expand-file-name fname tempdir))
-      (let ((base (concat "url" (int-to-string (user-real-uid))))
-	    (fname "")
-	    (x 0))
-	(setq fname (format fmt (concat base (int-to-string x))))
-	(while (file-exists-p
-		(expand-file-name fname tempdir))
-	  (setq x (1+ x)
-		fname (format fmt (concat base (int-to-string x)))))
-	(expand-file-name fname tempdir)))))
-
 (defun url-extract-mime-headers ()
   "Set `url-current-mime-headers' in current buffer."
   (save-excursion
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index d9277cf6f4..e35823ab9a 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -312,13 +312,6 @@ url-max-password-attempts
   :type 'integer
   :group 'url)
 
-(defcustom url-temporary-directory (or (getenv "TMPDIR") "/tmp")
-  "Where temporary files go."
-  :type 'directory
-  :group 'url-file)
-(make-obsolete-variable 'url-temporary-directory
-			'temporary-file-directory "23.1")
-
 (defcustom url-show-status t
   "Whether to show a running total of bytes transferred.
 Can cause a large hit if using a remote X display over a slow link, or
diff --git a/lisp/vc/vc-hooks.el b/lisp/vc/vc-hooks.el
index 2ca9d3e620..20002c4f59 100644
--- a/lisp/vc/vc-hooks.el
+++ b/lisp/vc/vc-hooks.el
@@ -506,9 +506,8 @@ vc-working-revision
                             backend 'working-revision file))))))
 
 ;; Backward compatibility.
-(define-obsolete-function-alias
-  'vc-workfile-version 'vc-working-revision "23.1")
 (defun vc-default-working-revision (backend file)
+  ;; FIXME: Should this be declared obsolete?
   (message
    "`working-revision' not found: using the old `workfile-version' instead")
   (vc-call-backend backend 'workfile-version file))
diff --git a/lisp/vc/vc-mtn.el b/lisp/vc/vc-mtn.el
index 092d8b5396..3c26ffc0e5 100644
--- a/lisp/vc/vc-mtn.el
+++ b/lisp/vc/vc-mtn.el
@@ -60,7 +60,6 @@ vc-mtn-annotate-switches
   :version "25.1"
   :group 'vc-mtn)
 
-(define-obsolete-variable-alias 'vc-mtn-command 'vc-mtn-program "23.1")
 (defcustom vc-mtn-program "mtn"
   "Name of the monotone executable."
   :type 'string
diff --git a/lisp/vc/vc.el b/lisp/vc/vc.el
index c640ba0420..2128f0edc1 100644
--- a/lisp/vc/vc.el
+++ b/lisp/vc/vc.el
@@ -2700,9 +2700,6 @@ vc-revert
       (vc-revert-file file)
       (message "Reverting %s...done" (vc-delistify files)))))
 
-;;;###autoload
-(define-obsolete-function-alias 'vc-revert-buffer 'vc-revert "23.1")
-
 ;;;###autoload
 (defun vc-pull (&optional arg)
   "Update the current fileset or branch.
diff --git a/lisp/vcursor.el b/lisp/vcursor.el
index fa0cbb74b0..3601abcd6e 100644
--- a/lisp/vcursor.el
+++ b/lisp/vcursor.el
@@ -1132,9 +1132,6 @@ vcursor-copy-line
     (vcursor-copy (if (or (= count 0) arg) (1+ count) count)))
   )
 
-(define-obsolete-function-alias
-  'vcursor-toggle-vcursor-map 'vcursor-use-vcursor-map "23.1")
-
 (defun vcursor-post-command ()
   (and vcursor-auto-disable (not vcursor-last-command)
        vcursor-overlay
-- 
2.26.2

Re: Delete variables obsolete since Emacs 23

[Paul Eggert]

On 5/16/20 6:18 AM, Stefan Kangas wrote:
> Other than that, does the attached patch look okay for master?

Yes, it looks good to me, thanks. Though Glenn is the real expert here....

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

> 3. Note the added FIXME in `vc-default-working-revision', where I'm not
>    sure if it should be declared obsolete or not.  See Stefan M's commit
>    6e5d0e9e73.

This function is never called directly (the only way it could be called
was via (vc-call-backend .. 'working-revision) for backends that
defined `vc-<backend>-workfile-revision` instead of
`vc-<backend>-working-revision`.

So a `make-obsolete` would have been ineffective (since it's never
called directly, the byte-compiler would never emit an obsolescence
warning) and I replaced it with a more annoying "message".  Since noone
complained about that message, I think it's definitely safe to remove
the function.

> 4. The variable `translation-table-for-input' was declared obsolete with
>    in Emacs 23.1 and has the following comment.  I'm not sure what to do
>    about it, if anything.
>
>    ;; This was introduced in 21.4 for pre-unicode unification.  That
>    ;; usage was rendered obsolete in 23.1, which uses Unicode internally.
>    ;; Other uses are possible, so this variable is not _really_ obsolete,
>    ;; but Stefan insists to mark it so.

It's not obsolete in the sense that there's nothing to replace
it, indeed.  But it's been completely unused since Emacs-23.1 so it's
complexity that's not pulling its weight.


        Stefan

Re: Delete variables obsolete since Emacs 23

[Dmitry Gutov]

On 17.05.2020 05:52, Stefan Monnier wrote:
>> 3. Note the added FIXME in `vc-default-working-revision', where I'm not
>>     sure if it should be declared obsolete or not.  See Stefan M's commit
>>     6e5d0e9e73.
> This function is never called directly (the only way it could be called
> was via (vc-call-backend .. 'working-revision) for backends that
> defined `vc-<backend>-workfile-revision` instead of
> `vc-<backend>-working-revision`.
> 
> So a `make-obsolete` would have been ineffective (since it's never
> called directly, the byte-compiler would never emit an obsolescence
> warning) and I replaced it with a more annoying "message".  Since noone
> complained about that message, I think it's definitely safe to remove
> the function.
> 

Agree.

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

Stefan Kangas <stefankangas@gmail.com> writes:

> Other than that, does the attached patch look okay for master?

I have looked this all over again, and I'm attaching a (hopefully) final
patch here. I found some places where the documentation needed updating,
and one or two uses that I had missed previously. There have been
relatively minor changes since last time, but I want to give everyone a
chance to comment before pushing.

If someone wants to take a stab at it, we still have to remove the below
items. I left them alone for now simply because they were a bit tricky
to untangle:

 * lisp/net/newst-backend.el (newsticker-cache-filename)
 * lisp/subr.el (translation-table-for-input)
 * lisp/subr.el (define-key-rebound-commands)
 * lisp/subr.el (redisplay-end-trigger-functions)
 * lisp/subr.el (window-redisplay-end-trigger)
 * lisp/subr.el (set-window-redisplay-end-trigger)

(The last four above should be easier to remove when the fast-lock and
lazy-lock libraries are removed.)

Besides that, the next step here would be to remove variables obsoleted
in 23.{2,3,4}.

Best regards,
Stefan Kangas

[-- Attachment #2: 0001-Remove-many-items-obsolete-since-Emacs-23.1.patch --]
[-- Type: text/x-diff, Size: 54202 bytes --]

From e8ac1ab0495684bec64c5a18226d9f61299523b2 Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Sat, 16 May 2020 14:16:24 +0200
Subject: [PATCH] Remove many items obsolete since Emacs 23.1

Emacs 23.1 was five major releases and over a decade ago.
This list can be reviewed before to the next release, but for now
hopefully this motivates any needed external updates.
Ref: https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg02198.html

* lisp/abbrev.el (pre-abbrev-expand-hook):
* lisp/bookmark.el (bookmark-read-annotation-text-func)
(bookmark-jump-noselect):
* lisp/buff-menu.el (buffer-menu-mode-hook):
* lisp/cus-edit.el (custom-mode-hook, custom-mode):
* lisp/dirtrack.el (dirtrack-debug-toggle, dirtrack-debug):
* lisp/emacs-lisp/crm.el (crm-minibuffer-complete)
(crm-minibuffer-completion-help)
(crm-minibuffer-complete-and-exit):
* lisp/emacs-lisp/easymenu.el
(easy-menu-precalculate-equivalent-keybindings):
* lisp/emacs-lisp/lisp-mode.el (lisp-mode-auto-fill):
* lisp/epa.el (epa-display-verify-result):
* lisp/epg.el (epg-passphrase-callback-function):
* lisp/eshell/eshell.el (eshell-report-bug):
* lisp/ffap.el (ffap-bug, ffap-submit-bug):
* lisp/files.el (locate-file-completion):
* lisp/hi-lock.el (hi-lock-face-history, hi-lock-regexp-history):
* lisp/hilit-chg.el (highlight-changes-initial-state)
(highlight-changes-active-string)
(highlight-changes-passive-string, global-highlight-changes):
* lisp/international/mule-cmds.el (nonascii-insert-offset)
(nonascii-translation-table):
* lisp/international/mule-diag.el (non-iso-charset-alist):
* lisp/international/mule-util.el (detect-coding-with-priority):
* lisp/international/mule.el (charset-id, charset-bytes)
(charset-list, char-valid-p, generic-char-p)
(char-coding-system-table, make-coding-system)
(set-coding-priority)
* lisp/mail/rmail.el (rmail-message-filter):
* lisp/minibuffer.el (complete-in-turn, dynamic-completion-table)
(completion-common-substring)
(minibuffer-local-must-match-filename-map):
* lisp/mouse.el (mouse-major-mode-menu, mouse-popup-menubar)
(mouse-popup-menubar-stuff):
* lisp/net/newst-treeview.el (newsticker-groups-filename):
* lisp/obsolete/tpu-edt.el (tpu-have-ispell, GOLD-map):
* lisp/password-cache.el (password-read-and-add):
* lisp/shell.el (shell-dirtrack-toggle):
* lisp/subr.el (forward-point, redisplay-end-trigger-functions)
(process-filter-multibyte-p, set-process-filter-multibyte):
* lisp/t-mouse.el (t-mouse-mode):
* lisp/term/w32-win.el (w32-focus-frame, w32-select-font):
* lisp/textmodes/ispell.el (ispell-aspell-supports-utf8):
* lisp/textmodes/remember.el (remember-buffer):
* lisp/tooltip.el (tooltip-hook):
* lisp/url/url-util.el (url-generate-unique-filename):
* lisp/url/url-vars.el (url-temporary-directory):
* lisp/vc/vc-hooks.el (vc-workfile-version)
(vc-default-working-revision):
* lisp/vc/vc-mtn.el (vc-mtn-command):
* lisp/vc/vc.el (vc-revert-buffer):
* lisp/vcursor.el (vcursor-toggle-vcursor-map):
Remove items, obsolete since Emacs 23.1.
* lisp/abbrev.el (expand-abbrev):
* lisp/epg.el (epg-context): Change
'epg-passphrase-callback-function' call to 'epa-' alternative.
* lisp/eshell/em-rebind.el (eshell-cannot-leave-input-list): Don't
refer to removed function 'forward-point'.
* test/manual/etags/c-src/abbrev.c (Fexpand_abbrev):
(syms_of_abbrev): Don't run removed hook 'pre-abbrev-expand-hook'.
* lisp/international/mule.el (transform-make-coding-system-args):
Declare obsolete.
* lisp/progmodes/idlwave.el:
Update reference to removed function 'char-valid-p'.
* lisp/gnus/mml2015.el (epg-encrypt-string):
* lisp/gnus/mml1991.el (epg-make-context):
* lisp/gnus/mml-smime.el (autoload):
Remove autoload of removed 'epg-passphrase-callback-function'.
* lisp/minibuffer.el (completion-extra-properties):
Remove support for `completion-common-substring'.
* lisp/obsolete/tpu-edt.el (tpu-toggle-overwrite-mode)
Remove support for removed `spell' package.
* src/coding.c (syms_of_coding):
* doc/misc/efaq.texi:
* doc/emacs/frames.texi (Menu Mouse Clicks):
* doc/misc/url.texi (Customization): Doc fixes.
; * etc/NEWS: List removed items.
---
 doc/emacs/custom.texi            |   1 +
 doc/emacs/frames.texi            |   8 +-
 doc/misc/efaq.texi               |   2 +-
 doc/misc/url.texi                |   2 -
 etc/NEWS                         |  34 ++++++++
 lisp/abbrev.el                   |  10 ---
 lisp/bookmark.el                 |  13 ---
 lisp/buff-menu.el                |   3 -
 lisp/cus-edit.el                 |   4 -
 lisp/dirtrack.el                 |   3 -
 lisp/emacs-lisp/crm.el           |   6 --
 lisp/emacs-lisp/easymenu.el      |  10 ---
 lisp/emacs-lisp/lisp-mode.el     |   2 -
 lisp/epa.el                      |   4 -
 lisp/epg.el                      |  17 +---
 lisp/eshell/em-rebind.el         |   1 -
 lisp/eshell/eshell.el            |   9 ---
 lisp/ffap.el                     |   6 --
 lisp/files.el                    |   8 --
 lisp/gnus/mml-smime.el           |   1 -
 lisp/gnus/mml1991.el             |   1 -
 lisp/gnus/mml2015.el             |   1 -
 lisp/hi-lock.el                  |   6 --
 lisp/hilit-chg.el                |  16 ----
 lisp/international/mule-cmds.el  |   5 --
 lisp/international/mule-diag.el  |   4 -
 lisp/international/mule-util.el  |   9 ---
 lisp/international/mule.el       | 135 +------------------------------
 lisp/mail/rmail.el               |  19 -----
 lisp/minibuffer.el               |  18 +----
 lisp/mouse.el                    |  28 -------
 lisp/net/newst-treeview.el       |  30 +------
 lisp/obsolete/tpu-edt.el         |  12 +--
 lisp/password-cache.el           |  16 ----
 lisp/progmodes/idlwave.el        |   2 -
 lisp/shell.el                    |   3 -
 lisp/subr.el                     |   7 --
 lisp/t-mouse.el                  |   2 -
 lisp/term/w32-win.el             |   4 -
 lisp/textmodes/ispell.el         |   9 ---
 lisp/textmodes/remember.el       |   3 -
 lisp/tooltip.el                  |   2 -
 lisp/url/url-util.el             |  25 ------
 lisp/url/url-vars.el             |   7 --
 lisp/vc/vc-hooks.el              |   8 --
 lisp/vc/vc-mtn.el                |   1 -
 lisp/vc/vc.el                    |   3 -
 lisp/vcursor.el                  |   3 -
 src/coding.c                     |   3 +-
 test/manual/etags/c-src/abbrev.c |  14 ----
 50 files changed, 50 insertions(+), 490 deletions(-)

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index acd7fb13ae..a512fd14c8 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2605,6 +2605,7 @@ Init Examples
 (if (fboundp 'blink-cursor-mode)
     (blink-cursor-mode 0))
 
+@c FIXME: Find better example since `set-coding-priority' is removed.
 (if (boundp 'coding-category-utf-8)
     (set-coding-priority '(coding-category-utf-8)))
 @end example
diff --git a/doc/emacs/frames.texi b/doc/emacs/frames.texi
index b99d8ab145..b74887612b 100644
--- a/doc/emacs/frames.texi
+++ b/doc/emacs/frames.texi
@@ -366,9 +366,13 @@ Menu Mouse Clicks
 @kbd{mouse-3} by adding the following line to your init file
 (@pxref{Init Rebinding}):
 
-@c FIXME: `mouse-popup-menubar-stuff' is obsolete since 23.1.
 @smallexample
-(global-set-key [mouse-3] 'mouse-popup-menubar-stuff)
+(global-set-key [mouse-3]
+  '(menu-item "Menu Bar" ignore
+    :filter (lambda (_)
+              (if (zerop (or (frame-parameter nil 'menu-bar-lines) 0))
+                  (mouse-menu-bar-map)
+                (mouse-menu-major-mode-map)))))
 @end smallexample
 
 @node Mode Line Mouse
diff --git a/doc/misc/efaq.texi b/doc/misc/efaq.texi
index 82467048a0..62dcc0b753 100644
--- a/doc/misc/efaq.texi
+++ b/doc/misc/efaq.texi
@@ -4192,7 +4192,7 @@ SPC no longer completes file names
 (define-key minibuffer-local-filename-completion-map (kbd "SPC")
   'minibuffer-complete-word)
 
-(define-key minibuffer-local-must-match-filename-map (kbd "SPC")
+(define-key minibuffer-local-filename-must-match-map (kbd "SPC")
   'minibuffer-complete-word)
 @end lisp
 
diff --git a/doc/misc/url.texi b/doc/misc/url.texi
index 8d9b102407..0304ff4b9f 100644
--- a/doc/misc/url.texi
+++ b/doc/misc/url.texi
@@ -1312,8 +1312,6 @@ Customization
 @end defopt
 @defopt url-max-password-attempts
 @end defopt
-@defopt url-temporary-directory
-@end defopt
 @defopt url-show-status
 @end defopt
 @defopt url-confirmation-func
diff --git a/etc/NEWS b/etc/NEWS
index 850b166069..68fffef197 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -771,6 +771,40 @@ have now been removed.
 ** Some libraries obsolete since Emacs 23 have been removed:
 'ledit.el', 'lmenu.el', 'lucid.el and 'old-whitespace.el'.
 
+---
+** Some functions and variables obsolete since Emacs 23 have been removed:
+
+'GOLD-map', 'bookmark-jump-noselect',
+'bookmark-read-annotation-text-func', 'buffer-menu-mode-hook',
+'char-coding-system-table', 'char-valid-p', 'charset-bytes',
+'charset-id', 'charset-list' (function), 'complete-in-turn',
+'completion-common-substring', 'crm-minibuffer-complete',
+'crm-minibuffer-complete-and-exit', 'crm-minibuffer-completion-help',
+'custom-mode', 'custom-mode-hook', 'detect-coding-with-priority',
+'dirtrack-debug' (function), 'dirtrack-debug-toggle',
+'dynamic-completion-table',
+'easy-menu-precalculate-equivalent-keybindings',
+'epa-display-verify-result', 'epg-passphrase-callback-function',
+'eshell-report-bug', 'ffap-bug', 'ffap-submit-bug', 'forward-point',
+'generic-char-p', 'global-highlight-changes', 'hi-lock-face-history',
+'hi-lock-regexp-history', 'highlight-changes-active-string',
+'highlight-changes-initial-state', 'highlight-changes-passive-string',
+'ispell-aspell-supports-utf8', 'lisp-mode-auto-fill',
+'locate-file-completion', 'make-coding-system',
+'minibuffer-local-must-match-filename-map', 'mouse-major-mode-menu',
+'mouse-popup-menubar', 'mouse-popup-menubar-stuff',
+'newsticker-groups-filename', 'non-iso-charset-alist',
+'nonascii-insert-offset', 'nonascii-translation-table',
+'password-read-and-add', 'pre-abbrev-expand-hook',
+'process-filter-multibyte-p', 'remember-buffer' (function),
+'rmail-message-filter', 'set-coding-priority',
+'set-process-filter-multibyte', 'shell-dirtrack-toggle',
+'t-mouse-mode', 'tooltip-hook', 'tpu-have-ispell',
+'url-generate-unique-filename', 'url-temporary-directory',
+'vc-arch-command', 'vc-default-working-revision' (variable),
+'vc-mtn-command', 'vc-revert-buffer', 'vc-workfile-version',
+'vcursor-toggle-vcursor-map', 'w32-focus-frame', 'w32-select-font'.
+
 \f
 * Lisp Changes in Emacs 28.1
 
diff --git a/lisp/abbrev.el b/lisp/abbrev.el
index 2d61a96010..468b0d995b 100644
--- a/lisp/abbrev.el
+++ b/lisp/abbrev.el
@@ -517,14 +517,6 @@ last-abbrev-location
 ;;   "Local (mode-specific) abbrev table of current buffer.")
 ;; (make-variable-buffer-local 'local-abbrev-table)
 
-(defcustom pre-abbrev-expand-hook nil
-  "Function or functions to be called before abbrev expansion is done.
-This is the first thing that `expand-abbrev' does, and so this may change
-the current abbrev table before abbrev lookup happens."
-  :type 'hook
-  :group 'abbrev-mode)
-(make-obsolete-variable 'pre-abbrev-expand-hook 'abbrev-expand-function "23.1")
-
 (defun clear-abbrev-table (table)
   "Undefine all abbrevs in abbrev table TABLE, leaving it empty."
   (setq abbrevs-changed t)
@@ -836,12 +828,10 @@ abbrev-expand-function
 (defun expand-abbrev ()
   "Expand the abbrev before point, if there is an abbrev there.
 Effective when explicitly called even when `abbrev-mode' is nil.
-Before doing anything else, runs `pre-abbrev-expand-hook'.
 Calls the value of `abbrev-expand-function' with no argument to do
 the work, and returns whatever it does.  (That return value should
 be the abbrev symbol if expansion occurred, else nil.)"
   (interactive)
-  (run-hooks 'pre-abbrev-expand-hook)
   (funcall abbrev-expand-function))
 
 (defun abbrev--default-expand ()
diff --git a/lisp/bookmark.el b/lisp/bookmark.el
index de7d60f97e..12b8c145fe 100644
--- a/lisp/bookmark.el
+++ b/lisp/bookmark.el
@@ -922,8 +922,6 @@ bookmark-default-annotation-text
 	  "#  Date:    " (current-time-string) "\n"))
 
 
-(define-obsolete-variable-alias 'bookmark-read-annotation-text-func
-  'bookmark-edit-annotation-text-func "23.1")
 (defvar bookmark-edit-annotation-text-func 'bookmark-default-annotation-text
   "Function to return default text to use for a bookmark annotation.
 It takes one argument, the name of the bookmark, as a string.")
@@ -1142,17 +1140,6 @@ bookmark-jump-other-frame
   (let ((pop-up-frames t))
     (bookmark-jump-other-window bookmark)))
 
-(defun bookmark-jump-noselect (bookmark)
-  "Return the location pointed to by BOOKMARK (see `bookmark-jump').
-The return value has the form (BUFFER . POINT).
-
-Note: this function is deprecated and is present for Emacs 22
-compatibility only."
-  (declare (obsolete bookmark-handle-bookmark "23.1"))
-  (save-excursion
-    (bookmark-handle-bookmark bookmark)
-    (cons (current-buffer) (point))))
-
 (defun bookmark-handle-bookmark (bookmark-name-or-record)
   "Call BOOKMARK-NAME-OR-RECORD's handler or `bookmark-default-handler'
 if it has none.  This changes current buffer and point and returns nil,
diff --git a/lisp/buff-menu.el b/lisp/buff-menu.el
index 9fe0dbae38..324646dc3c 100644
--- a/lisp/buff-menu.el
+++ b/lisp/buff-menu.el
@@ -214,9 +214,6 @@ Buffer-menu-mode-map
     map)
   "Local keymap for `Buffer-menu-mode' buffers.")
 
-(define-obsolete-variable-alias 'buffer-menu-mode-hook
-  'Buffer-menu-mode-hook "23.1")
-
 (define-derived-mode Buffer-menu-mode tabulated-list-mode "Buffer Menu"
   "Major mode for Buffer Menu buffers.
 The Buffer Menu is invoked by the commands \\[list-buffers],
diff --git a/lisp/cus-edit.el b/lisp/cus-edit.el
index 16695967df..5ec5799f80 100644
--- a/lisp/cus-edit.el
+++ b/lisp/cus-edit.el
@@ -4868,8 +4868,6 @@ Custom-goto-parent
 	       (parent (downcase (widget-get  button :tag))))
 	  (customize-group parent)))))
 
-(define-obsolete-variable-alias 'custom-mode-hook 'Custom-mode-hook "23.1")
-
 (defcustom Custom-mode-hook nil
   "Hook called when entering Custom mode."
   :type 'hook
@@ -4940,8 +4938,6 @@ Custom-mode
 
 (put 'Custom-mode 'mode-class 'special)
 
-(define-obsolete-function-alias 'custom-mode 'Custom-mode "23.1")
-
 ;;; The End.
 
 (provide 'cus-edit)
diff --git a/lisp/dirtrack.el b/lisp/dirtrack.el
index 3a0bbd2c9c..ad0c18d1b3 100644
--- a/lisp/dirtrack.el
+++ b/lisp/dirtrack.el
@@ -196,9 +196,6 @@ dirtrack-mode
     (remove-hook 'comint-preoutput-filter-functions 'dirtrack t)))
 
 
-(define-obsolete-function-alias 'dirtrack-debug-toggle 'dirtrack-debug-mode
-  "23.1")
-(define-obsolete-variable-alias 'dirtrack-debug 'dirtrack-debug-mode "23.1")
 (define-minor-mode dirtrack-debug-mode
   "Toggle Dirtrack debugging."
   nil nil nil
diff --git a/lisp/emacs-lisp/crm.el b/lisp/emacs-lisp/crm.el
index 65483d0813..89d106ee48 100644
--- a/lisp/emacs-lisp/crm.el
+++ b/lisp/emacs-lisp/crm.el
@@ -270,12 +270,6 @@ completing-read-multiple
     (remove-hook 'choose-completion-string-functions
 		 'crm--choose-completion-string)))
 
-(define-obsolete-function-alias 'crm-minibuffer-complete 'crm-complete "23.1")
-(define-obsolete-function-alias
-  'crm-minibuffer-completion-help 'crm-completion-help "23.1")
-(define-obsolete-function-alias
-  'crm-minibuffer-complete-and-exit 'crm-complete-and-exit "23.1")
-
 ;; testing and debugging
 ;; (defun crm-init-test-environ ()
 ;;   "Set up some variables for testing."
diff --git a/lisp/emacs-lisp/easymenu.el b/lisp/emacs-lisp/easymenu.el
index 6ba8b997f8..73dabef3fa 100644
--- a/lisp/emacs-lisp/easymenu.el
+++ b/lisp/emacs-lisp/easymenu.el
@@ -29,16 +29,6 @@
 
 ;;; Code:
 
-(defvar easy-menu-precalculate-equivalent-keybindings nil
-  "Determine when equivalent key bindings are computed for easy-menu menus.
-It can take some time to calculate the equivalent key bindings that are shown
-in a menu.  If the variable is on, then this calculation gives a (maybe
-noticeable) delay when a mode is first entered.  If the variable is off, then
-this delay will come when a menu is displayed the first time.  If you never use
-menus, turn this variable off, otherwise it is probably better to keep it on.")
-(make-obsolete-variable
- 'easy-menu-precalculate-equivalent-keybindings nil "23.1")
-
 (defsubst easy-menu-intern (s)
   (if (stringp s) (intern s) s))
 
diff --git a/lisp/emacs-lisp/lisp-mode.el b/lisp/emacs-lisp/lisp-mode.el
index 1311d94cb0..d0c28ec5dc 100644
--- a/lisp/emacs-lisp/lisp-mode.el
+++ b/lisp/emacs-lisp/lisp-mode.el
@@ -785,8 +785,6 @@ lisp-comment-indent
             nil)))
       (comment-indent-default)))
 
-(define-obsolete-function-alias 'lisp-mode-auto-fill 'do-auto-fill "23.1")
-
 (defcustom lisp-indent-offset nil
   "If non-nil, indent second line of expressions that many more columns."
   :group 'lisp
diff --git a/lisp/epa.el b/lisp/epa.el
index 3c7dd8309a..beefc71ddf 100644
--- a/lisp/epa.el
+++ b/lisp/epa.el
@@ -644,10 +644,6 @@ epa-display-error
 	  (goto-char (point-min)))
 	(display-buffer buffer)))))
 
-(defun epa-display-verify-result (verify-result)
-  (declare (obsolete epa-display-info "23.1"))
-  (epa-display-info (epg-verify-result-to-string verify-result)))
-
 (defun epa-passphrase-callback-function (context key-id handback)
   (if (eq key-id 'SYM)
       (read-passwd
diff --git a/lisp/epg.el b/lisp/epg.el
index 5b90bc290a..dd19f73eb1 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -180,6 +180,8 @@ 'epg-error
   (file nil :read-only t)
   (string nil :read-only t))
 
+(declare-function epa-passphrase-callback-function "epa.el")
+
 (cl-defstruct (epg-context
                (:constructor nil)
                (:constructor epg-context--make
@@ -204,7 +206,7 @@ 'epg-error
   cipher-algorithm
   digest-algorithm
   compress-algorithm
-  (passphrase-callback (list #'epg-passphrase-callback-function))
+  (passphrase-callback (list #'epa-passphrase-callback-function))
   progress-callback
   edit-callback
   signers
@@ -1234,19 +1236,6 @@ epg--status-IMPORT_RES
 			     (epg-context-result-for context 'import-status)))
     (epg-context-set-result-for context 'import-status nil)))
 
-(defun epg-passphrase-callback-function (context key-id _handback)
-  (declare (obsolete epa-passphrase-callback-function "23.1"))
-  (if (eq key-id 'SYM)
-      (read-passwd "Passphrase for symmetric encryption: "
-		   (eq (epg-context-operation context) 'encrypt))
-    (read-passwd
-     (if (eq key-id 'PIN)
-	"Passphrase for PIN: "
-       (let ((entry (assoc key-id epg-user-id-alist)))
-	 (if entry
-	     (format "Passphrase for %s %s: " key-id (cdr entry))
-	   (format "Passphrase for %s: " key-id)))))))
-
 (defun epg--list-keys-1 (context name mode)
   (let ((args (append (if (epg-context-home-directory context)
 			  (list "--homedir"
diff --git a/lisp/eshell/em-rebind.el b/lisp/eshell/em-rebind.el
index bf5a4bf1af..7991c63177 100644
--- a/lisp/eshell/em-rebind.el
+++ b/lisp/eshell/em-rebind.el
@@ -114,7 +114,6 @@ eshell-cannot-leave-input-list
     backward-list
     forward-page
     backward-page
-    forward-point
     forward-paragraph
     backward-paragraph
     backward-prefix-chars
diff --git a/lisp/eshell/eshell.el b/lisp/eshell/eshell.el
index 5ffb159b57..6698ca45de 100644
--- a/lisp/eshell/eshell.el
+++ b/lisp/eshell/eshell.el
@@ -384,15 +384,6 @@ eshell-command-result
 	      (set status-var eshell-last-command-status))
 	  (cadr result))))))
 
-;;;_* Reporting bugs
-;;
-;; If you do encounter a bug, on any system, please report
-;; it -- in addition to any particular oddities in your configuration
-;; -- so that the problem may be corrected for the benefit of others.
-
-;;;###autoload
-(define-obsolete-function-alias 'eshell-report-bug 'report-emacs-bug "23.1")
-
 ;;; Code:
 
 (defun eshell-unload-all-modules ()
diff --git a/lisp/ffap.el b/lisp/ffap.el
index ceba9d2622..4a506207d5 100644
--- a/lisp/ffap.el
+++ b/lisp/ffap.el
@@ -1824,12 +1824,6 @@ ffap-literally
 
 (defalias 'find-file-literally-at-point 'ffap-literally)
 
-\f
-;;; Bug Reporter:
-
-(define-obsolete-function-alias 'ffap-bug 'report-emacs-bug "23.1")
-(define-obsolete-function-alias 'ffap-submit-bug 'report-emacs-bug "23.1")
-
 \f
 ;;; Hooks for Gnus, VM, Rmail:
 ;;
diff --git a/lisp/files.el b/lisp/files.el
index 1909669346..dd48c151e3 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -979,14 +979,6 @@ locate-file-completion-table
       (completion-table-with-context
        string-dir names string-file pred action)))))
 
-(defun locate-file-completion (string path-and-suffixes action)
-  "Do completion for file names passed to `locate-file'.
-PATH-AND-SUFFIXES is a pair of lists, (DIRECTORIES . SUFFIXES)."
-  (declare (obsolete locate-file-completion-table "23.1"))
-  (locate-file-completion-table (car path-and-suffixes)
-                                (cdr path-and-suffixes)
-                                string nil action))
-
 (defvar locate-dominating-stop-dir-regexp
   (purecopy "\\`\\(?:[\\/][\\/][^\\/]+[\\/]\\|/\\(?:net\\|afs\\|\\.\\.\\.\\)/\\)\\'")
   "Regexp of directory names that stop the search in `locate-dominating-file'.
diff --git a/lisp/gnus/mml-smime.el b/lisp/gnus/mml-smime.el
index 4754f37a2d..acddb30033 100644
--- a/lisp/gnus/mml-smime.el
+++ b/lisp/gnus/mml-smime.el
@@ -329,7 +329,6 @@ password-cache-expiry
   (autoload 'epg-verify-string "epg")
   (autoload 'epg-sign-string "epg")
   (autoload 'epg-encrypt-string "epg")
-  (autoload 'epg-passphrase-callback-function "epg")
   (autoload 'epg-context-set-passphrase-callback "epg")
   (autoload 'epg-sub-key-fingerprint "epg")
   (autoload 'epg-configuration "epg-config")
diff --git a/lisp/gnus/mml1991.el b/lisp/gnus/mml1991.el
index 8be1b84e52..88864ea357 100644
--- a/lisp/gnus/mml1991.el
+++ b/lisp/gnus/mml1991.el
@@ -242,7 +242,6 @@ mml1991-pgg-encrypt
 (defvar epg-user-id-alist)
 
 (autoload 'epg-make-context "epg")
-(autoload 'epg-passphrase-callback-function "epg")
 (autoload 'epa-select-keys "epa")
 (autoload 'epg-list-keys "epg")
 (autoload 'epg-context-set-armor "epg")
diff --git a/lisp/gnus/mml2015.el b/lisp/gnus/mml2015.el
index d1d150ad2e..45c9bbfe90 100644
--- a/lisp/gnus/mml2015.el
+++ b/lisp/gnus/mml2015.el
@@ -712,7 +712,6 @@ inhibit-redisplay
 (autoload 'epg-verify-string "epg")
 (autoload 'epg-sign-string "epg")
 (autoload 'epg-encrypt-string "epg")
-(autoload 'epg-passphrase-callback-function "epg")
 (autoload 'epg-context-set-passphrase-callback "epg")
 (autoload 'epg-key-sub-key-list "epg")
 (autoload 'epg-sub-key-capability "epg")
diff --git a/lisp/hi-lock.el b/lisp/hi-lock.el
index 33ca40f8de..0ffe77d276 100644
--- a/lisp/hi-lock.el
+++ b/lisp/hi-lock.el
@@ -237,17 +237,11 @@ hi-lock-interactive-lighters
   "Human-readable lighters for `hi-lock-interactive-patterns'.")
 (put 'hi-lock-interactive-lighters 'permanent-local t)
 
-(define-obsolete-variable-alias 'hi-lock-face-history
-                                'hi-lock-face-defaults "23.1")
 (defvar hi-lock-face-defaults
   '("hi-yellow" "hi-pink" "hi-green" "hi-blue" "hi-salmon" "hi-aquamarine"
     "hi-black-b" "hi-blue-b" "hi-red-b" "hi-green-b" "hi-black-hb")
   "Default faces for hi-lock interactive functions.")
 
-(define-obsolete-variable-alias 'hi-lock-regexp-history
-                                'regexp-history
-                                "23.1")
-
 (defvar hi-lock-file-patterns-prefix "Hi-lock"
   "String used to identify hi-lock patterns at the start of files.")
 
diff --git a/lisp/hilit-chg.el b/lisp/hilit-chg.el
index 04a5ccd8d5..ae97bb008a 100644
--- a/lisp/hilit-chg.el
+++ b/lisp/hilit-chg.el
@@ -224,9 +224,6 @@ highlight-changes-colors
 ;; When you invoke highlight-changes-mode, should highlight-changes-visible-mode
 ;; be on or off?
 
-(define-obsolete-variable-alias 'highlight-changes-initial-state
-  'highlight-changes-visibility-initial-state "23.1")
-
 (defcustom highlight-changes-visibility-initial-state t
   "Controls whether changes are initially visible in Highlight Changes mode.
 
@@ -236,13 +233,7 @@ highlight-changes-visibility-initial-state
   :type 'boolean
   :group 'highlight-changes)
 
-;; highlight-changes-global-initial-state has been removed
-
-
-
 ;; These are the strings displayed in the mode-line for the minor mode:
-(define-obsolete-variable-alias 'highlight-changes-active-string
-  'highlight-changes-visible-string "23.1")
 
 (defcustom highlight-changes-visible-string " +Chg"
   "The string used when in Highlight Changes mode and changes are visible.
@@ -252,9 +243,6 @@ highlight-changes-visible-string
 		 (const :tag "None"  nil))
   :group 'highlight-changes)
 
-(define-obsolete-variable-alias 'highlight-changes-passive-string
-  'highlight-changes-invisible-string "23.1")
-
 (defcustom highlight-changes-invisible-string " -Chg"
   "The string used when in Highlight Changes mode and changes are hidden.
 This should be set to nil if no indication is desired, or to
@@ -957,10 +945,6 @@ hilit-chg-get-diff-list-hk
 (define-globalized-minor-mode global-highlight-changes-mode
   highlight-changes-mode highlight-changes-mode-turn-on)
 
-(define-obsolete-function-alias
- 'global-highlight-changes
-  'global-highlight-changes-mode "23.1")
-
 (defun highlight-changes-mode-turn-on ()
   "See if Highlight Changes mode should be turned on for this buffer.
 This is called when `global-highlight-changes-mode' is turned on."
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 7714a778fc..5fe931dd9b 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -2968,11 +2968,6 @@ unify-8859-on-decoding-mode
 ;; Doc said "obsolete" in 23.1, this statement only added in 24.1.
 (make-obsolete 'unify-8859-on-decoding-mode "don't use it." "23.1")
 
-(defvar nonascii-insert-offset 0)
-(make-obsolete-variable 'nonascii-insert-offset "do not use it." "23.1")
-(defvar nonascii-translation-table nil)
-(make-obsolete-variable 'nonascii-translation-table "do not use it." "23.1")
-
 (defvar ucs-names nil
   "Hash table of cached CHAR-NAME keys to CHAR-CODE values.")
 
diff --git a/lisp/international/mule-diag.el b/lisp/international/mule-diag.el
index 80e78ef787..b13bde58ca 100644
--- a/lisp/international/mule-diag.el
+++ b/lisp/international/mule-diag.el
@@ -200,10 +200,6 @@ list-character-sets-2
 ;;;		   (charset-iso-graphic-plane charset)
 		   (charset-description charset)))))
 
-(defvar non-iso-charset-alist nil
-  "Obsolete.")
-(make-obsolete-variable 'non-iso-charset-alist "no longer relevant." "23.1")
-
 ;; A variable to hold charset input history.
 (defvar charset-history nil)
 
diff --git a/lisp/international/mule-util.el b/lisp/international/mule-util.el
index 5cc10b1315..660ac58e02 100644
--- a/lisp/international/mule-util.el
+++ b/lisp/international/mule-util.el
@@ -274,15 +274,6 @@ with-coding-priority
 ;;;###autoload(put 'with-coding-priority 'lisp-indent-function 1)
 (put 'with-coding-priority 'edebug-form-spec t)
 
-;;;###autoload
-(defmacro detect-coding-with-priority (from to priority-list)
-  "Detect a coding system of the text between FROM and TO with PRIORITY-LIST.
-PRIORITY-LIST is an alist of coding categories vs the corresponding
-coding systems ordered by priority."
-  (declare (obsolete with-coding-priority "23.1"))
-  `(with-coding-priority (mapcar #'cdr ,priority-list)
-     (detect-coding-region ,from ,to)))
-
 ;;;###autoload
 (defun detect-coding-with-language-environment (from to lang-env)
   "Detect a coding system for the text between FROM and TO with LANG-ENV.
diff --git a/lisp/international/mule.el b/lisp/international/mule.el
index df71205d51..092abc09b0 100644
--- a/lisp/international/mule.el
+++ b/lisp/international/mule.el
@@ -408,16 +408,6 @@ charset-info
 ;; because that makes a bootstrapping problem
 ;; if you need to recompile all the Lisp files using interpreted code.
 
-(defun charset-id (_charset)
-  "Always return 0.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  0)
-
-(defmacro charset-bytes (_charset)
-  "Always return 0.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  0)
-
 (defun get-charset-property (charset propname)
   "Return the value of CHARSET's PROPNAME property.
 This is the last value stored with
@@ -463,19 +453,8 @@ charset-long-name
   "Return long name of CHARSET."
   (plist-get (charset-plist charset) :long-name))
 
-(defun charset-list ()
-  "Return list of all charsets ever defined."
-  (declare (obsolete charset-list "23.1"))
-  charset-list)
-
 \f
 ;;; CHARACTER
-(define-obsolete-function-alias 'char-valid-p 'characterp "23.1")
-
-(defun generic-char-p (_char)
-  "Always return nil.  This is provided for backward compatibility."
-  (declare (obsolete nil "23.1"))
-  nil)
 
 (defun make-char-internal (charset-id &optional code1 code2)
   (let ((charset (aref emacs-mule-charset-table charset-id)))
@@ -1085,14 +1064,11 @@ coding-system-list
 	      (setq codings (cons alias codings))))))
     codings))
 
-(defconst char-coding-system-table nil
-  "It exists just for backward compatibility, and the value is always nil.")
-(make-obsolete-variable 'char-coding-system-table nil "23.1")
-
 (defun transform-make-coding-system-args (name type &optional doc-string props)
   "For internal use only.
 Transform XEmacs style args for `make-coding-system' to Emacs style.
 Value is a list of transformed arguments."
+  (declare (obsolete nil "28.1"))
   (let ((mnemonic (string-to-char (or (plist-get props 'mnemonic) "?")))
 	(eol-type (plist-get props 'eol-type))
 	properties tmp)
@@ -1170,106 +1146,6 @@ transform-make-coding-system-args
       (error "unsupported XEmacs style make-coding-style arguments: %S"
 	     `(,name ,type ,doc-string ,props))))))
 
-(defun make-coding-system (coding-system type mnemonic doc-string
-					 &optional
-					 flags
-					 properties
-					 eol-type)
-  "Define a new coding system CODING-SYSTEM (symbol).
-This function is provided for backward compatibility."
-  (declare (obsolete define-coding-system "23.1"))
-  ;; For compatibility with XEmacs, we check the type of TYPE.  If it
-  ;; is a symbol, perhaps, this function is called with XEmacs-style
-  ;; arguments.  Here, try to transform that kind of arguments to
-  ;; Emacs style.
-  (if (symbolp type)
-      (let ((args (transform-make-coding-system-args coding-system type
-						     mnemonic doc-string)))
-	(setq coding-system (car args)
-	      type (nth 1 args)
-	      mnemonic (nth 2 args)
-	      doc-string (nth 3 args)
-	      flags (nth 4 args)
-	      properties (nth 5 args)
-	      eol-type (nth 6 args))))
-
-  (setq type
-	(cond ((eq type 0) 'emacs-mule)
-	      ((eq type 1) 'shift-jis)
-	      ((eq type 2) 'iso2022)
-	      ((eq type 3) 'big5)
-	      ((eq type 4) 'ccl)
-	      ((eq type 5) 'raw-text)
-	      (t
-	       (error "Invalid coding system type: %s" type))))
-
-  (setq properties
-	(let ((plist nil) key)
-	  (dolist (elt properties)
-	    (setq key (car elt))
-	    (cond ((eq key 'post-read-conversion)
-		   (setq key :post-read-conversion))
-		  ((eq key 'pre-write-conversion)
-		   (setq key :pre-write-conversion))
-		  ((eq key 'translation-table-for-decode)
-		   (setq key :decode-translation-table))
-		  ((eq key 'translation-table-for-encode)
-		   (setq key :encode-translation-table))
-		  ((eq key 'safe-charsets)
-		   (setq key :charset-list))
-		  ((eq key 'mime-charset)
-		   (setq key :mime-charset))
-		  ((eq key 'valid-codes)
-		   (setq key :valids)))
-	    (setq plist (plist-put plist key (cdr elt))))
-	  plist))
-  (setq properties (plist-put properties :mnemonic mnemonic))
-  (plist-put properties :coding-type type)
-  (cond ((eq eol-type 0) (setq eol-type 'unix))
-	((eq eol-type 1) (setq eol-type 'dos))
-	((eq eol-type 2) (setq eol-type 'mac))
-	((vectorp eol-type) (setq eol-type nil)))
-  (plist-put properties :eol-type eol-type)
-
-  (cond
-   ((eq type 'iso2022)
-    (plist-put properties :flags
-	       (list (and (or (consp (nth 0 flags))
-			      (consp (nth 1 flags))
-			      (consp (nth 2 flags))
-			      (consp (nth 3 flags))) 'designation)
-		     (or (nth 4 flags) 'long-form)
-		     (and (nth 5 flags) 'ascii-at-eol)
-		     (and (nth 6 flags) 'ascii-at-cntl)
-		     (and (nth 7 flags) '7-bit)
-		     (and (nth 8 flags) 'locking-shift)
-		     (and (nth 9 flags) 'single-shift)
-		     (and (nth 10 flags) 'use-roman)
-		     (and (nth 11 flags) 'use-oldjis)
-		     (or (nth 12 flags) 'direction)
-		     (and (nth 13 flags) 'init-at-bol)
-		     (and (nth 14 flags) 'designate-at-bol)
-		     (and (nth 15 flags) 'safe)
-		     (and (nth 16 flags) 'latin-extra)))
-    (plist-put properties :designation
-	       (let ((vec (make-vector 4 nil)))
-		 (dotimes (i 4)
-		   (let ((spec (nth i flags)))
-		     (if (eq spec t)
-			 (aset vec i '(94 96))
-		     (if (consp spec)
-			 (progn
-			   (if (memq t spec)
-			       (setq spec (append (delq t spec) '(94 96))))
-			   (aset vec i spec))))))
-		 vec)))
-
-   ((eq type 'ccl)
-    (plist-put properties :ccl-decoder (car flags))
-    (plist-put properties :ccl-encoder (cdr flags))))
-
-  (apply 'define-coding-system coding-system doc-string properties))
-
 (defun merge-coding-systems (first second)
   "Fill in any unspecified aspects of coding system FIRST from SECOND.
 Return the resulting coding system."
@@ -1616,15 +1492,6 @@ set-next-selection-coding-system
 
   (setq next-selection-coding-system coding-system))
 
-(defun set-coding-priority (arg)
-  "Set priority of coding categories according to ARG.
-ARG is a list of coding categories ordered by priority.
-
-This function is provided for backward compatibility."
-  (declare (obsolete set-coding-system-priority "23.1"))
-  (apply 'set-coding-system-priority
-	 (mapcar #'(lambda (x) (symbol-value x)) arg)))
-
 ;;; X selections
 
 (defvar ctext-non-standard-encodings-alist
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index 44cde7cb5a..312baffb90 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -521,25 +521,6 @@ rmail-mmdf-delim1
 (defvar rmail-mmdf-delim2 "^\001\001\001\001\n"
   "Regexp marking the end of an mmdf message.")
 
-;; FIXME Post-mbox, this is now unused.
-;; In Emacs-22, this was called:
-;;  i) the very first time a message was shown.
-;; ii) when toggling the headers to the normal state, every time.
-;; It's not clear what it should do now, since there is nothing that
-;; records when a message is shown for the first time (unseen is not
-;; necessarily the same thing).
-;; See https://lists.gnu.org/r/emacs-devel/2009-03/msg00013.html
-(defcustom rmail-message-filter nil
-  "If non-nil, a filter function for new messages in RMAIL.
-Called with region narrowed to the message, including headers,
-before obeying `rmail-ignored-headers'."
-  :group 'rmail-headers
-  :type '(choice (const nil) function))
-
-(make-obsolete-variable 'rmail-message-filter
-			"it is not used (try `rmail-show-message-hook')."
-			"23.1")
-
 (defcustom rmail-automatic-folder-directives nil
   "List of directives specifying how to automatically file messages.
 Whenever Rmail shows a message in the folder that `rmail-file-name'
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index d2c3f9045e..0d99f4687c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -685,13 +685,6 @@ completion--twq-all
                completions)
        qboundary))))
 
-;; (defmacro complete-in-turn (a b) `(completion-table-in-turn ,a ,b))
-;; (defmacro dynamic-completion-table (fun) `(completion-table-dynamic ,fun))
-(define-obsolete-function-alias
-  'complete-in-turn #'completion-table-in-turn "23.1")
-(define-obsolete-function-alias
-  'dynamic-completion-table #'completion-table-dynamic "23.1")
-
 ;;; Minibuffer completion
 
 (defgroup minibuffer nil
@@ -1770,9 +1763,6 @@ completion--insert-strings
 			      ;; Round up to a whole number of columns.
 			      (* colwidth (ceiling length colwidth))))))))))))
 
-(defvar completion-common-substring nil)
-(make-obsolete-variable 'completion-common-substring nil "23.1")
-
 (defvar completion-setup-hook nil
   "Normal hook run at the end of setting up a completion list buffer.
 When this hook is run, the current buffer is the one in which the
@@ -1864,11 +1854,7 @@ display-completion-list
         (insert "Possible completions are:\n")
         (completion--insert-strings completions))))
 
-  ;; The hilit used to be applied via completion-setup-hook, so there
-  ;; may still be some code that uses completion-common-substring.
-  (with-no-warnings
-    (let ((completion-common-substring common-substring))
-      (run-hooks 'completion-setup-hook)))
+  (run-hooks 'completion-setup-hook)
   nil)
 
 (defvar completion-extra-properties nil
@@ -2374,8 +2360,6 @@ minibuffer-local-filename-completion-map
 Gets combined either with `minibuffer-local-completion-map' or
 with `minibuffer-local-must-match-map'.")
 
-(define-obsolete-variable-alias 'minibuffer-local-must-match-filename-map
-  'minibuffer-local-filename-must-match-map "23.1")
 (defvar minibuffer-local-filename-must-match-map (make-sparse-keymap))
 (make-obsolete-variable 'minibuffer-local-filename-must-match-map nil "24.1")
 
diff --git a/lisp/mouse.el b/lisp/mouse.el
index d369545f18..a06ca2a56c 100644
--- a/lisp/mouse.el
+++ b/lisp/mouse.el
@@ -274,34 +274,6 @@ mouse-menu-bar-map
            local-menu
            minor-mode-menus)))
 
-(defun mouse-major-mode-menu (event &optional prefix)
-  "Pop up a mode-specific menu of mouse commands.
-Default to the Edit menu if the major mode doesn't define a menu."
-  (declare (obsolete mouse-menu-major-mode-map "23.1"))
-  (interactive "@e\nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu (mouse-menu-major-mode-map) event prefix))
-
-(defun mouse-popup-menubar (event prefix)
-  "Pop up a menu equivalent to the menu bar for keyboard EVENT with PREFIX.
-The contents are the items that would be in the menu bar whether or
-not it is actually displayed."
-  (declare (obsolete mouse-menu-bar-map "23.1"))
-  (interactive "@e \nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu (mouse-menu-bar-map) (unless (integerp event) event) prefix))
-
-(defun mouse-popup-menubar-stuff (event prefix)
-  "Popup a menu like either `mouse-major-mode-menu' or `mouse-popup-menubar'.
-Use the former if the menu bar is showing, otherwise the latter."
-  (declare (obsolete nil "23.1"))
-  (interactive "@e\nP")
-  (run-hooks 'activate-menubar-hook 'menu-bar-update-hook)
-  (popup-menu
-   (if (zerop (or (frame-parameter nil 'menu-bar-lines) 0))
-       (mouse-menu-bar-map)
-     (mouse-menu-major-mode-map))
-   event prefix))
 \f
 ;; Commands that operate on windows.
 
diff --git a/lisp/net/newst-treeview.el b/lisp/net/newst-treeview.el
index 1bed61f3e7..ff8a447c7c 100644
--- a/lisp/net/newst-treeview.el
+++ b/lisp/net/newst-treeview.el
@@ -131,14 +131,6 @@ newsticker-groups
 Example: (\"Topmost group\" \"feed1\" (\"subgroup1\" \"feed 2\")
 \"feed3\")")
 
-(defcustom newsticker-groups-filename
-  nil
-  "Name of the newsticker groups settings file."
-  :version "25.1"                       ; changed default value to nil
-  :type '(choice (const nil) string)
-  :group 'newsticker-treeview)
-(make-obsolete-variable 'newsticker-groups-filename 'newsticker-dir "23.1")
-
 ;; ======================================================================
 ;;; internal variables
 ;; ======================================================================
@@ -1265,29 +1257,9 @@ newsticker-treeview-save
 (defun newsticker--treeview-load ()
   "Load treeview settings."
   (let* ((coding-system-for-read 'utf-8)
-         (filename
-          (or (and newsticker-groups-filename
-                   (not (string=
-                         (expand-file-name newsticker-groups-filename)
-                         (expand-file-name (concat newsticker-dir "/groups"))))
-                   (file-exists-p newsticker-groups-filename)
-                   (y-or-n-p
-                    (format-message
-                     (concat "Obsolete variable `newsticker-groups-filename' "
-                             "points to existing file \"%s\".\n"
-                             "Read it? ")
-                     newsticker-groups-filename))
-                   newsticker-groups-filename)
-              (concat newsticker-dir "/groups")))
+         (filename (concat newsticker-dir "/groups"))
          (buf (and (file-exists-p filename)
                    (find-file-noselect filename))))
-    (and newsticker-groups-filename
-         (file-exists-p newsticker-groups-filename)
-	 (y-or-n-p (format-message
-                    (concat "Delete the file \"%s\",\nto which the obsolete "
-                            "variable `newsticker-groups-filename' points ? ")
-                    newsticker-groups-filename))
-	 (delete-file newsticker-groups-filename))
     (when buf
       (set-buffer buf)
       (goto-char (point-min))
diff --git a/lisp/obsolete/tpu-edt.el b/lisp/obsolete/tpu-edt.el
index d71f79c87b..0de7aa096d 100644
--- a/lisp/obsolete/tpu-edt.el
+++ b/lisp/obsolete/tpu-edt.el
@@ -287,14 +287,6 @@ tpu-version
 ;;;
 ;;;  User Configurable Variables
 ;;;
-(defcustom tpu-have-ispell t
-  "Non-nil means `tpu-spell-check' uses `ispell-region' for spell checking.
-Otherwise, use `spell-region'."
-  :type 'boolean
-  :group 'tpu)
-(make-obsolete-variable 'tpu-have-ispell "the `spell' package is obsolete."
-                        "23.1")
-
 (defcustom tpu-kill-buffers-silently nil
   "If non-nil, TPU-edt kills modified buffers without asking."
   :type 'boolean
@@ -315,7 +307,6 @@ tpu-pan-columns
 ;;;  Global Keymaps
 ;;;
 
-(define-obsolete-variable-alias 'GOLD-map 'tpu-gold-map "23.1")
 (defvar tpu-gold-map
   (let ((map (make-keymap)))
     ;; Previously we used escape sequences here.  We now instead presume
@@ -892,8 +883,7 @@ tpu-spell-check
 if no region is selected."
   (interactive)
   (let ((m (tpu-mark)))
-    (apply (if tpu-have-ispell 'ispell-region
-             'spell-region)
+    (apply 'ispell-region
            (if m
                (if (> m (point)) (list (point) m)
                  (list m (point)))
diff --git a/lisp/password-cache.el b/lisp/password-cache.el
index f5007579a8..2443f374a8 100644
--- a/lisp/password-cache.el
+++ b/lisp/password-cache.el
@@ -94,22 +94,6 @@ password-read
   (or (password-read-from-cache key)
       (read-passwd prompt)))
 
-(defun password-read-and-add (prompt &optional key)
-  "Read password, for use with KEY, from user, or from cache if wanted.
-Then store the password in the cache.  Uses `password-read' and
-`password-cache-add'.  Custom variables `password-cache' and
-`password-cache-expiry' regulate cache behavior.
-
-Warning: the password is cached without checking that it is
-correct.  It is better to check the password before caching.  If
-you must use this function, take care to check passwords and
-remove incorrect ones from the cache."
-  (declare (obsolete password-read "23.1"))
-  (let ((password (password-read prompt key)))
-    (when (and password key)
-      (password-cache-add key password))
-    password))
-
 (defun password-cache-remove (key)
   "Remove password indexed by KEY from password cache.
 This is typically run by a timer setup from `password-cache-add',
diff --git a/lisp/progmodes/idlwave.el b/lisp/progmodes/idlwave.el
index 3092d4c45b..55acb77425 100644
--- a/lisp/progmodes/idlwave.el
+++ b/lisp/progmodes/idlwave.el
@@ -159,8 +159,6 @@
   (defalias 'line-beginning-position 'point-at-bol))
 (unless (fboundp 'line-end-position)
   (defalias 'line-end-position 'point-at-eol))
-(unless (fboundp 'char-valid-p)
-  (defalias 'char-valid-p 'characterp))
 (unless (fboundp 'match-string-no-properties)
   (defalias 'match-string-no-properties 'match-string))
 
diff --git a/lisp/shell.el b/lisp/shell.el
index dc528412a6..39f2b4ce0f 100644
--- a/lisp/shell.el
+++ b/lisp/shell.el
@@ -985,9 +985,6 @@ shell-dirtrack-mode
       (add-hook 'comint-input-filter-functions #'shell-directory-tracker nil t)
     (remove-hook 'comint-input-filter-functions #'shell-directory-tracker t)))
 
-(define-obsolete-function-alias 'shell-dirtrack-toggle #'shell-dirtrack-mode
-  "23.1")
-
 (defun shell-cd (dir)
   "Do normal `cd' to DIR, and set `list-buffers-directory'."
   (cd dir)
diff --git a/lisp/subr.el b/lisp/subr.el
index 6bd06a0b82..8eb4cf452d 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1583,11 +1583,6 @@ posn-object-width-height
 (make-obsolete 'string-as-multibyte "use `decode-coding-string'." "26.1")
 (make-obsolete 'string-make-multibyte "use `decode-coding-string'." "26.1")
 
-(defun forward-point (n)
-  "Return buffer position N characters after (before if N negative) point."
-  (declare (obsolete "use (+ (point) N) instead." "23.1"))
-  (+ (point) n))
-
 (defun log10 (x)
   "Return (log X 10), the log base 10 of X."
   (declare (obsolete log "24.4"))
@@ -1612,8 +1607,6 @@ log10
 (make-obsolete 'set-window-redisplay-end-trigger nil "23.1")
 
 (make-obsolete 'run-window-configuration-change-hook nil "27.1")
-(make-obsolete 'process-filter-multibyte-p nil "23.1")
-(make-obsolete 'set-process-filter-multibyte nil "23.1")
 
 (make-obsolete-variable 'command-debug-status
                         "expect it to be removed in a future version." "25.2")
diff --git a/lisp/t-mouse.el b/lisp/t-mouse.el
index a1af53d8c4..4feab71401 100644
--- a/lisp/t-mouse.el
+++ b/lisp/t-mouse.el
@@ -62,8 +62,6 @@ gpm-mouse-disable
     (gpm-mouse-stop))
   (set-terminal-parameter nil 'gpm-mouse-active nil))
 
-;;;###autoload
-(define-obsolete-function-alias 't-mouse-mode 'gpm-mouse-mode "23.1")
 ;;;###autoload
 (define-minor-mode gpm-mouse-mode
   "Toggle mouse support in GNU/Linux consoles (GPM Mouse mode).
diff --git a/lisp/term/w32-win.el b/lisp/term/w32-win.el
index 5901e0295e..e866fdc36c 100644
--- a/lisp/term/w32-win.el
+++ b/lisp/term/w32-win.el
@@ -78,12 +78,8 @@
 (require 'dnd)
 (require 'w32-vars)
 
-;; Keep an obsolete alias for w32-focus-frame and w32-select-font in case
-;; they are used by code outside Emacs.
-(define-obsolete-function-alias 'w32-focus-frame 'x-focus-frame "23.1")
 (declare-function x-select-font "w32font.c"
                   (&optional frame exclude-proportional))
-(define-obsolete-function-alias 'w32-select-font 'x-select-font "23.1")
 
 (defvar w32-color-map) ;; defined in w32fns.c
 (make-obsolete 'w32-default-color-map nil "24.1")
diff --git a/lisp/textmodes/ispell.el b/lisp/textmodes/ispell.el
index 65f61644b6..ffcdb9bd16 100644
--- a/lisp/textmodes/ispell.el
+++ b/lisp/textmodes/ispell.el
@@ -621,15 +621,6 @@ ispell-encoding8-command
 Earlier Aspell versions do not consistently support charset encoding.  Handling
 this would require some extra guessing in `ispell-aspell-find-dictionary'.")
 
-(defvar ispell-aspell-supports-utf8 nil
-  "Non-nil if Aspell has consistent command line UTF-8 support.  Obsolete.
-ispell.el and flyspell.el will use for this purpose the more generic
-variable `ispell-encoding8-command' for both Aspell and Hunspell.  Is left
-here just for backwards compatibility.")
-
-(make-obsolete-variable 'ispell-aspell-supports-utf8
-                        'ispell-encoding8-command "23.1")
-
 (defvar ispell-dicts-name2locale-equivs-alist
   '(("american"      "en_US")
     ("brasileiro"    "pt_BR")
diff --git a/lisp/textmodes/remember.el b/lisp/textmodes/remember.el
index 279dbb4450..7bc7dc1762 100644
--- a/lisp/textmodes/remember.el
+++ b/lisp/textmodes/remember.el
@@ -487,9 +487,6 @@ remember-finalize
   (interactive)
   (remember-region (point-min) (point-max)))
 
-;; Org needs this
-(define-obsolete-function-alias 'remember-buffer 'remember-finalize "23.1")
-
 (defun remember-destroy ()
   "Destroy the current *Remember* buffer."
   (interactive)
diff --git a/lisp/tooltip.el b/lisp/tooltip.el
index f35f6b9a03..5f5a4788b2 100644
--- a/lisp/tooltip.el
+++ b/lisp/tooltip.el
@@ -167,8 +167,6 @@ tooltip-resize-echo-area
 \f
 ;;; Variables that are not customizable.
 
-(define-obsolete-variable-alias 'tooltip-hook 'tooltip-functions "23.1")
-
 (defvar tooltip-functions nil
   "Functions to call to display tooltips.
 Each function is called with one argument EVENT which is a copy
diff --git a/lisp/url/url-util.el b/lisp/url/url-util.el
index 6dd7a9c2aa..0a7e7e205e 100644
--- a/lisp/url/url-util.el
+++ b/lisp/url/url-util.el
@@ -569,31 +569,6 @@ url-get-url-at-point
 	  (setq url nil))
       url)))
 
-(defun url-generate-unique-filename (&optional fmt)
-  "Generate a unique filename in `url-temporary-directory'."
-  (declare (obsolete make-temp-file "23.1"))
-  ;; This variable is obsolete, but so is this function.
-  (let ((tempdir (with-no-warnings url-temporary-directory)))
-    (if (not fmt)
-	(let ((base (format "url-tmp.%d" (user-real-uid)))
-	      (fname "")
-	      (x 0))
-	  (setq fname (format "%s%d" base x))
-	  (while (file-exists-p
-		  (expand-file-name fname tempdir))
-	    (setq x (1+ x)
-		  fname (concat base (int-to-string x))))
-	  (expand-file-name fname tempdir))
-      (let ((base (concat "url" (int-to-string (user-real-uid))))
-	    (fname "")
-	    (x 0))
-	(setq fname (format fmt (concat base (int-to-string x))))
-	(while (file-exists-p
-		(expand-file-name fname tempdir))
-	  (setq x (1+ x)
-		fname (format fmt (concat base (int-to-string x)))))
-	(expand-file-name fname tempdir)))))
-
 (defun url-extract-mime-headers ()
   "Set `url-current-mime-headers' in current buffer."
   (save-excursion
diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el
index d9277cf6f4..e35823ab9a 100644
--- a/lisp/url/url-vars.el
+++ b/lisp/url/url-vars.el
@@ -312,13 +312,6 @@ url-max-password-attempts
   :type 'integer
   :group 'url)
 
-(defcustom url-temporary-directory (or (getenv "TMPDIR") "/tmp")
-  "Where temporary files go."
-  :type 'directory
-  :group 'url-file)
-(make-obsolete-variable 'url-temporary-directory
-			'temporary-file-directory "23.1")
-
 (defcustom url-show-status t
   "Whether to show a running total of bytes transferred.
 Can cause a large hit if using a remote X display over a slow link, or
diff --git a/lisp/vc/vc-hooks.el b/lisp/vc/vc-hooks.el
index ce72a49b95..f09ceddcb3 100644
--- a/lisp/vc/vc-hooks.el
+++ b/lisp/vc/vc-hooks.el
@@ -505,14 +505,6 @@ vc-working-revision
                            (vc-call-backend
                             backend 'working-revision file))))))
 
-;; Backward compatibility.
-(define-obsolete-function-alias
-  'vc-workfile-version 'vc-working-revision "23.1")
-(defun vc-default-working-revision (backend file)
-  (message
-   "`working-revision' not found: using the old `workfile-version' instead")
-  (vc-call-backend backend 'workfile-version file))
-
 (defun vc-default-registered (backend file)
   "Check if FILE is registered in BACKEND using vc-BACKEND-master-templates."
   (let ((sym (vc-make-backend-sym backend 'master-templates)))
diff --git a/lisp/vc/vc-mtn.el b/lisp/vc/vc-mtn.el
index 092d8b5396..3c26ffc0e5 100644
--- a/lisp/vc/vc-mtn.el
+++ b/lisp/vc/vc-mtn.el
@@ -60,7 +60,6 @@ vc-mtn-annotate-switches
   :version "25.1"
   :group 'vc-mtn)
 
-(define-obsolete-variable-alias 'vc-mtn-command 'vc-mtn-program "23.1")
 (defcustom vc-mtn-program "mtn"
   "Name of the monotone executable."
   :type 'string
diff --git a/lisp/vc/vc.el b/lisp/vc/vc.el
index 65775f8e46..5561292d8c 100644
--- a/lisp/vc/vc.el
+++ b/lisp/vc/vc.el
@@ -2709,9 +2709,6 @@ vc-revert
       (vc-revert-file file)
       (message "Reverting %s...done" (vc-delistify files)))))
 
-;;;###autoload
-(define-obsolete-function-alias 'vc-revert-buffer 'vc-revert "23.1")
-
 ;;;###autoload
 (defun vc-pull (&optional arg)
   "Update the current fileset or branch.
diff --git a/lisp/vcursor.el b/lisp/vcursor.el
index fa0cbb74b0..3601abcd6e 100644
--- a/lisp/vcursor.el
+++ b/lisp/vcursor.el
@@ -1132,9 +1132,6 @@ vcursor-copy-line
     (vcursor-copy (if (or (= count 0) arg) (1+ count) count)))
   )
 
-(define-obsolete-function-alias
-  'vcursor-toggle-vcursor-map 'vcursor-use-vcursor-map "23.1")
-
 (defun vcursor-post-command ()
   (and vcursor-auto-disable (not vcursor-last-command)
        vcursor-overlay
diff --git a/src/coding.c b/src/coding.c
index 071124b4ef..1d79c703a3 100644
--- a/src/coding.c
+++ b/src/coding.c
@@ -11829,8 +11829,7 @@ syms_of_coding (void)
 This variable is given to `completing-read' as COLLECTION argument.
 
 Do not alter the value of this variable manually.  This variable should be
-updated by the functions `make-coding-system' and
-`define-coding-system-alias'.  */);
+updated by `define-coding-system-alias'.  */);
   Vcoding_system_alist = Qnil;
 
   DEFVAR_LISP ("coding-category-list", Vcoding_category_list,
diff --git a/test/manual/etags/c-src/abbrev.c b/test/manual/etags/c-src/abbrev.c
index 03b9f0e65b..44563d6046 100644
--- a/test/manual/etags/c-src/abbrev.c
+++ b/test/manual/etags/c-src/abbrev.c
@@ -78,9 +78,6 @@
 
 int last_abbrev_point;
 
-/* Hook to run before expanding any abbrev.  */
-
-Lisp_Object Vpre_abbrev_expand_hook, Qpre_abbrev_expand_hook;
 \f
 DEFUN ("make-abbrev-table", Fmake_abbrev_table, Smake_abbrev_table, 0, 0, 0,
   "Create a new, empty abbrev table object.")
@@ -232,9 +229,6 @@ DEFUN ("expand-abbrev", Fexpand_abbrev, Sexpand_abbrev, 0, 0, "",
 
   value = Qnil;
 
-  if (!NILP (Vrun_hooks))
-    call1 (Vrun_hooks, Qpre_abbrev_expand_hook);
-
   wordstart = 0;
   if (!(BUFFERP (Vabbrev_start_location_buffer)
 	&& XBUFFER (Vabbrev_start_location_buffer) == current_buffer))
@@ -595,14 +589,6 @@ syms_of_abbrev ()
     "*Set non-nil means expand multi-word abbrevs all caps if abbrev was so.");
   abbrev_all_caps = 0;
 
-  DEFVAR_LISP ("pre-abbrev-expand-hook", &Vpre_abbrev_expand_hook,
-    "Function or functions to be called before abbrev expansion is done.\n\
-This is the first thing that `expand-abbrev' does, and so this may change\n\
-the current abbrev table before abbrev lookup happens.");
-  Vpre_abbrev_expand_hook = Qnil;
-  Qpre_abbrev_expand_hook = intern ("pre-abbrev-expand-hook");
-  staticpro (&Qpre_abbrev_expand_hook);
-
   defsubr (&Smake_abbrev_table);
   defsubr (&Sclear_abbrev_table);
   defsubr (&Sdefine_abbrev);
-- 
2.27.0

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

Stefan Kangas <stefankangas@gmail.com> writes:

> I have looked this all over again, and I'm attaching a (hopefully) final
> patch here. I found some places where the documentation needed updating,
> and one or two uses that I had missed previously. There have been
> relatively minor changes since last time, but I want to give everyone a
> chance to comment before pushing.

No further comments within a week.  Pushed to master as commit
874ba85363e.

Best regards,
Stefan Kangas

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

Stefan Kangas <stefankangas@gmail.com> writes:

> Besides that, the next step here would be to remove variables obsoleted
> in 23.{2,3,4}.

I'm attaching a patch to remove most things declared obsolete in 23.2
and 23.3.  (Nothing was declared obsolete in 23.4 AFAICT.)

There are still some items obsoleted in 23.x left to delete in Semantic.
I intend to do that in a separate patch.

Comments welcome.

Best regards,
Stefan Kangas

[-- Attachment #2: 0001-Remove-many-items-obsolete-since-Emacs-23.2-and-23.3.patch --]
[-- Type: text/x-diff, Size: 36276 bytes --]

From 7f3f615b3547ee902461f752ba420ffc705c20d1 Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Sat, 8 Aug 2020 05:32:37 +0200
Subject: [PATCH] Remove many items obsolete since Emacs 23.2 and 23.3

* lisp/allout.el (allout-init):
* lisp/emacs-lisp/shadow.el (shadows-compare-text-p):
* lisp/ffap.el (ffap-version):
* lisp/filecache.el (file-cache-choose-completion):
* lisp/help.el (print-help-return-message):
* lisp/image-mode.el (image-mode-maybe):
* lisp/imenu.el (imenu-example--name-and-position):
* lisp/international/mule-cmds.el (princ-list):
* lisp/mail/rmail.el (rmail-highlight-face):
* lisp/minibuffer.el (read-file-name-predicate):
* lisp/mouse.el (mouse-choose-completion):
* lisp/progmodes/cc-cmds.el (c-forward-into-nomenclature):
* lisp/progmodes/xscheme.el
(advertised-xscheme-send-previous-expression):
* lisp/simple.el (completion-base-size)
(choose-completion-delete-max-match, exchange-dot-and-mark):
* lisp/subr.el (eval-next-after-load, interactive-p):
* lisp/term.el (term-dynamic-simple-complete):
Remove items, obsolete since Emacs 23.2 and 23.3.
* doc/misc/cc-mode.texi (Movement Commands): Doc fix.
* doc/lispref/help.texi (Accessing Documentation):
* lisp/emacs-lisp/edebug.el (edebug-wrap-def-body):
* lisp/cedet/data-debug.el:
* lisp/simple.el (append-next-kill):
* test/manual/cedet/cedet-utests.el (cedet-utest, pulse-test):
* test/manual/cedet/semantic-tests.el
(semantic-symref-test-count-hits-in-tag)
(semantic-lex-spp-write-utest): Use 'called-interactively-p'
instead of 'interactive-p'.
* lisp/comint.el (comint-dynamic-list-completions):
* lisp/progmodes/idlwave.el (idlwave-make-modified-completion-map-xemacs)
(idlwave-make-modified-completion-map-emacs):
* lisp/progmodes/vhdl-mode.el:
* lisp/term.el (term-dynamic-list-completions):
Remove references to 'mouse-choose-completion'.
* lisp/image-mode.el (image-mode-to-text):
Remove reference to 'image-mode-maybe'.
* lisp/mail/rmail.el (rmail-highlight-headers):
Use 'rmail-highlight' face instead of 'rmail-highlight-face'.
* lisp/progmodes/antlr-mode.el (antlr-mode-map, antlr-mode-menu):
Remove reference to 'c-forward-into-nomenclature'.
* lisp/simple.el (choose-completion, choose-completion-string)
(completion-list-mode, completion-setup-function): Don't use
'completion-base-size'.
; * etc/NEWS: List removed items.
---
 doc/lispref/help.texi               |  3 +-
 doc/misc/cc-mode.texi               |  2 --
 etc/NEWS                            | 49 ++++++++++++++++-------------
 lisp/allout.el                      | 15 +--------
 lisp/cedet/data-debug.el            |  2 +-
 lisp/comint.el                      |  2 +-
 lisp/emacs-lisp/edebug.el           |  2 +-
 lisp/emacs-lisp/shadow.el           |  3 --
 lisp/ffap.el                        |  2 --
 lisp/filecache.el                   |  3 --
 lisp/help.el                        |  1 -
 lisp/image-mode.el                  |  4 +--
 lisp/imenu.el                       | 22 -------------
 lisp/international/mule-cmds.el     |  6 ----
 lisp/mail/rmail.el                  | 22 ++-----------
 lisp/minibuffer.el                  |  5 ---
 lisp/mouse.el                       |  3 --
 lisp/progmodes/antlr-mode.el        |  4 +--
 lisp/progmodes/cc-cmds.el           | 13 --------
 lisp/progmodes/idlwave.el           | 11 ++-----
 lisp/progmodes/vhdl-mode.el         |  4 ---
 lisp/progmodes/xscheme.el           |  2 --
 lisp/simple.el                      | 43 +++----------------------
 lisp/subr.el                        | 30 ------------------
 lisp/term.el                        | 49 +----------------------------
 test/manual/cedet/cedet-utests.el   | 12 +++----
 test/manual/cedet/semantic-tests.el |  4 +--
 27 files changed, 55 insertions(+), 263 deletions(-)

diff --git a/doc/lispref/help.texi b/doc/lispref/help.texi
index 9b3c4fcb23..d4505d5c3f 100644
--- a/doc/lispref/help.texi
+++ b/doc/lispref/help.texi
@@ -220,7 +220,8 @@ Accessing Documentation
 
 @group
     ;; @r{Display the data.}
-    (help-setup-xref (list 'describe-symbols pattern) (interactive-p))
+    (help-setup-xref (list 'describe-symbols pattern)
+                 (called-interactively-p 'interactive))
     (with-help-window (help-buffer)
       (mapcar describe-func (sort sym-list 'string<)))))
 @end group
diff --git a/doc/misc/cc-mode.texi b/doc/misc/cc-mode.texi
index 10bbf8ff09..adc233d99d 100644
--- a/doc/misc/cc-mode.texi
+++ b/doc/misc/cc-mode.texi
@@ -1024,9 +1024,7 @@ Movement Commands
 preprocessor statements.
 
 @item @kbd{M-x c-backward-into-nomenclature}
-@itemx @kbd{M-x c-forward-into-nomenclature}
 @findex c-backward-into-nomenclature
-@findex c-forward-into-nomenclature
 @findex backward-into-nomenclature @r{(c-)}
 @findex forward-into-nomenclature @r{(c-)}
 A popular programming style, especially for object-oriented languages
diff --git a/etc/NEWS b/etc/NEWS
index 91add027e4..ef9a11c521 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -910,35 +910,42 @@ have now been removed.
 
 ---
 ** Some functions and variables obsolete since Emacs 23 have been removed:
-
-'GOLD-map', 'bookmark-jump-noselect',
+'GOLD-map', 'advertised-xscheme-send-previous-expression',
+'allout-init', 'bookmark-jump-noselect',
 'bookmark-read-annotation-text-func', 'buffer-menu-mode-hook',
-'char-coding-system-table', 'char-valid-p', 'charset-bytes',
-'charset-id', 'charset-list' (function), 'complete-in-turn',
-'completion-common-substring', 'crm-minibuffer-complete',
-'crm-minibuffer-complete-and-exit', 'crm-minibuffer-completion-help',
-'custom-mode', 'custom-mode-hook', 'detect-coding-with-priority',
-'dirtrack-debug' (function), 'dirtrack-debug-toggle',
-'dynamic-completion-table',
+'c-forward-into-nomenclature', 'char-coding-system-table',
+'char-valid-p', 'charset-bytes', 'charset-id', 'charset-list'
+(function), 'choose-completion-delete-max-match', 'complete-in-turn',
+'completion-base-size', 'completion-common-substring',
+'crm-minibuffer-complete', 'crm-minibuffer-complete-and-exit',
+'crm-minibuffer-completion-help', 'custom-mode', 'custom-mode-hook',
+'detect-coding-with-priority', 'dirtrack-debug' (function),
+'dirtrack-debug-toggle', 'dynamic-completion-table',
 'easy-menu-precalculate-equivalent-keybindings',
 'epa-display-verify-result', 'epg-passphrase-callback-function',
-'eshell-report-bug', 'ffap-bug', 'ffap-submit-bug', 'forward-point',
-'generic-char-p', 'global-highlight-changes', 'hi-lock-face-history',
+'eshell-report-bug', 'eval-next-after-load', 'exchange-dot-and-mark',
+'ffap-bug', 'ffap-submit-bug', 'ffap-version',
+'file-cache-choose-completion', 'forward-point', 'generic-char-p',
+'global-highlight-changes', 'hi-lock-face-history',
 'hi-lock-regexp-history', 'highlight-changes-active-string',
 'highlight-changes-initial-state', 'highlight-changes-passive-string',
-'ispell-aspell-supports-utf8', 'lisp-mode-auto-fill',
+'image-mode-maybe', 'imenu-example--name-and-position',
+'interactive-p', 'ispell-aspell-supports-utf8', 'lisp-mode-auto-fill',
 'locate-file-completion', 'make-coding-system',
-'minibuffer-local-must-match-filename-map', 'mouse-major-mode-menu',
-'mouse-popup-menubar', 'mouse-popup-menubar-stuff',
-'newsticker-groups-filename', 'non-iso-charset-alist',
-'nonascii-insert-offset', 'nonascii-translation-table',
-'password-read-and-add', 'pre-abbrev-expand-hook',
-'process-filter-multibyte-p', 'remember-buffer' (function),
+'minibuffer-local-must-match-filename-map', 'mouse-choose-completion',
+'mouse-major-mode-menu', 'mouse-popup-menubar',
+'mouse-popup-menubar-stuff', 'newsticker-groups-filename',
+'non-iso-charset-alist', 'nonascii-insert-offset',
+'nonascii-translation-table', 'password-read-and-add',
+'pre-abbrev-expand-hook', 'princ-list', 'print-help-return-message',
+'process-filter-multibyte-p', 'read-file-name-predicate',
+'remember-buffer' (function), 'rmail-highlight-face',
 'rmail-message-filter', 'set-coding-priority',
-'set-process-filter-multibyte', 'shell-dirtrack-toggle',
-'t-mouse-mode', 'tooltip-hook', 'tpu-have-ispell',
+'set-process-filter-multibyte', 'shadows-compare-text-p',
+'shell-dirtrack-toggle', 't-mouse-mode',
+'term-dynamic-simple-complete', 'tooltip-hook', 'tpu-have-ispell',
 'url-generate-unique-filename', 'url-temporary-directory',
-'vc-arch-command', 'vc-default-working-revision' (variable),
+'vc-arch-command', 'vc-default-working-revision (variable)',
 'vc-mtn-command', 'vc-revert-buffer', 'vc-workfile-version',
 'vcursor-toggle-vcursor-map', 'w32-focus-frame', 'w32-select-font'.
 
diff --git a/lisp/allout.el b/lisp/allout.el
index 05d9153a31..955b7000cb 100644
--- a/lisp/allout.el
+++ b/lisp/allout.el
@@ -62,8 +62,7 @@
 ;; The outline menubar additions provide quick reference to many of the
 ;; features.  See the docstring of the variables `allout-layout' and
 ;; `allout-auto-activation' for details on automatic activation of
-;; `allout-mode' as a minor mode.  (`allout-init' is deprecated in favor of
-;; a purely customization-based method.)
+;; `allout-mode' as a minor mode.
 ;;
 ;; Note -- the lines beginning with `;;;_' are outline topic headers.
 ;;        Customize `allout-auto-activation' to enable, then revisit this
@@ -1627,18 +1626,6 @@ allout-explicitly-deactivated
   "If t, `allout-mode's last deactivation was deliberate.
 So `allout-post-command-business' should not reactivate it...")
 (make-variable-buffer-local 'allout-explicitly-deactivated)
-;;;_  > allout-init (mode)
-(defun allout-init (mode)
-  "DEPRECATED - configure allout activation by customizing
-`allout-auto-activation'.  This function remains around, limited
-from what it did before, for backwards compatibility.
-
-MODE is the activation mode - see `allout-auto-activation' for
-valid values."
-  (declare (obsolete allout-auto-activation "23.3"))
-  (customize-set-variable 'allout-auto-activation (format "%s" mode))
-  (format "%s" mode))
-
 ;;;_  > allout-setup-menubar ()
 (defun allout-setup-menubar ()
   "Populate the current buffer's menubar with `allout-mode' stuff."
diff --git a/lisp/cedet/data-debug.el b/lisp/cedet/data-debug.el
index 604fc40926..44cce389cb 100644
--- a/lisp/cedet/data-debug.el
+++ b/lisp/cedet/data-debug.el
@@ -38,7 +38,7 @@
 ;;   "Calculate something complicated at point, and return it."
 ;;   (interactive) ;; function not normally interactive
 ;;   (let ((stuff (do-stuff)))
-;;     (when (interactive-p)
+;;     (when (called-interactively-p 'interactive)
 ;;       (data-debug-show-stuff stuff "myStuff"))
 ;;     stuff))
 
diff --git a/lisp/comint.el b/lisp/comint.el
index df4937a7d6..9460009bf4 100644
--- a/lisp/comint.el
+++ b/lisp/comint.el
@@ -3442,7 +3442,7 @@ comint-dynamic-list-completions
 		 (eq (window-buffer (posn-window (event-start first)))
 		     (get-buffer "*Completions*"))
 		 (memq (key-binding key)
-                       '(mouse-choose-completion choose-completion))))
+                       '(choose-completion))))
 	  ;; If the user does choose-completion with the mouse,
 	  ;; execute the command, then delete the completion window.
 	  (progn
diff --git a/lisp/emacs-lisp/edebug.el b/lisp/emacs-lisp/edebug.el
index d9bbf6129c..7ff6d68c3e 100644
--- a/lisp/emacs-lisp/edebug.el
+++ b/lisp/emacs-lisp/edebug.el
@@ -1229,7 +1229,7 @@ edebug-wrap-def-body
   "Wrap the FORMS of a definition body."
   (if edebug-def-interactive
       `(let ((,(edebug-interactive-p-name)
-	      (interactive-p)))
+	      (called-interactively-p 'interactive)))
 	 ,(edebug-make-enter-wrapper forms))
     (edebug-make-enter-wrapper forms)))
 
diff --git a/lisp/emacs-lisp/shadow.el b/lisp/emacs-lisp/shadow.el
index 4ff129e367..dd614dd792 100644
--- a/lisp/emacs-lisp/shadow.el
+++ b/lisp/emacs-lisp/shadow.el
@@ -55,9 +55,6 @@ lisp-shadow
   :prefix "load-path-shadows-"
   :group 'lisp)
 
-(define-obsolete-variable-alias 'shadows-compare-text-p
-  'load-path-shadows-compare-text "23.3")
-
 (defcustom load-path-shadows-compare-text nil
   "If non-nil, then shadowing files are reported only if their text differs.
 This is slower, but filters out some innocuous shadowing."
diff --git a/lisp/ffap.el b/lisp/ffap.el
index 4a506207d5..97e55df65c 100644
--- a/lisp/ffap.el
+++ b/lisp/ffap.el
@@ -110,8 +110,6 @@
 (require 'url-parse)
 (require 'thingatpt)
 
-(define-obsolete-variable-alias 'ffap-version 'emacs-version "23.2")
-
 (defgroup ffap nil
   "Find file or URL at point."
   :group 'matching
diff --git a/lisp/filecache.el b/lisp/filecache.el
index 3c07a49420..113d28cf75 100644
--- a/lisp/filecache.el
+++ b/lisp/filecache.el
@@ -614,9 +614,6 @@ file-cache-choose-completion
     (select-window (active-minibuffer-window))
     (file-cache-minibuffer-complete nil)))
 
-(define-obsolete-function-alias 'file-cache-mouse-choose-completion
-  #'file-cache-choose-completion "23.2")
-
 (defun file-cache-complete  ()
   "Complete the word at point, using the filecache."
   (interactive)
diff --git a/lisp/help.el b/lisp/help.el
index b7d867eb70..1b0149616f 100644
--- a/lisp/help.el
+++ b/lisp/help.el
@@ -131,7 +131,6 @@ help-return-method
  (WINDOW . quit-window)    do quit-window, then select WINDOW.
  (WINDOW BUF START POINT)  display BUF at START, POINT, then select WINDOW.")
 
-(define-obsolete-function-alias 'print-help-return-message 'help-print-return-message "23.2")
 (defun help-print-return-message (&optional function)
   "Display or return message saying how to restore windows after help command.
 This function assumes that `standard-output' is the help buffer.
diff --git a/lisp/image-mode.el b/lisp/image-mode.el
index 948e62e10d..534609932c 100644
--- a/lisp/image-mode.el
+++ b/lisp/image-mode.el
@@ -709,7 +709,7 @@ image-mode-to-text
 displays an image file as text."
   ;; image-mode-as-text = normal-mode + image-minor-mode
   (let ((previous-image-type image-type)) ; preserve `image-type'
-    (major-mode-restore '(image-mode image-mode-maybe image-mode-as-text))
+    (major-mode-restore '(image-mode image-mode-as-text))
     ;; Restore `image-type' after `kill-all-local-variables' in `normal-mode'.
     (setq image-type previous-image-type)
     ;; Enable image minor mode with `C-c C-c'.
@@ -759,8 +759,6 @@ image-mode-as-text
                  (if (image-get-display-property)
                      "text" "an image or hex") ".")))
 
-(define-obsolete-function-alias 'image-mode-maybe 'image-mode "23.2")
-
 (defun image-toggle-display-text ()
   "Show the image file as text.
 Remove text properties that display the image."
diff --git a/lisp/imenu.el b/lisp/imenu.el
index 1949f2f48f..3a16dcb9ac 100644
--- a/lisp/imenu.el
+++ b/lisp/imenu.el
@@ -316,28 +316,6 @@ imenu-progress-message
 )
 
 
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;;;
-;;;; Some examples of functions utilizing the framework of this
-;;;; package.
-;;;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-;; FIXME: This was the only imenu-example-* definition actually used,
-;; by cperl-mode.el.  Now cperl-mode has its own copy, so these can
-;; all be removed.
-(defun imenu-example--name-and-position ()
-  "Return the current/previous sexp and its (beginning) location.
-Don't move point."
-  (declare (obsolete "use your own function instead." "23.2"))
-  (save-excursion
-    (forward-sexp -1)
-    ;; [ydi] modified for imenu-use-markers
-    (let ((beg (if imenu-use-markers (point-marker) (point)))
-	  (end (progn (forward-sexp) (point))))
-      (cons (buffer-substring beg end)
-	    beg))))
-
 ;;;
 ;;; Lisp
 ;;;
diff --git a/lisp/international/mule-cmds.el b/lisp/international/mule-cmds.el
index 5fe931dd9b..02dacaf0a2 100644
--- a/lisp/international/mule-cmds.el
+++ b/lisp/international/mule-cmds.el
@@ -2070,12 +2070,6 @@ set-language-environment-unibyte
   "Do various unibyte-mode setups for language environment LANGUAGE-NAME."
   (set-display-table-and-terminal-coding-system language-name))
 
-(defun princ-list (&rest args)
-  "Print all arguments with `princ', then print \"\\n\"."
-  (declare (obsolete "use mapc and princ instead." "23.3"))
-  (mapc #'princ args)
-  (princ "\n"))
-
 (put 'describe-specified-language-support 'apropos-inhibit t)
 
 ;; Print language-specific information such as input methods,
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index 312baffb90..f14025a93a 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -417,20 +417,6 @@ rmail-highlight
   :group 'rmail-headers
   :version "22.1")
 
-;; This was removed in Emacs 23.1 with no notification, an unnecessary
-;; incompatible change.
-(defcustom rmail-highlight-face 'rmail-highlight
-  "Face used by Rmail for highlighting headers."
-  ;; Note that nil doesn't actually mean use the default face, it
-  ;; means use either bold or highlight. It's not worth fixing this
-  ;; now that this is obsolete.
-  :type '(choice (const :tag "Default" nil)
-		 face)
-  :group 'rmail-headers)
-(make-obsolete-variable 'rmail-highlight-face
-			"customize the face `rmail-highlight' instead."
-			"23.2")
-
 (defface rmail-header-name
   '((t (:inherit font-lock-function-name-face)))
   "Face to use for highlighting the header names.
@@ -3012,7 +2998,7 @@ rmail-redecode-body
 
 (defun rmail-highlight-headers ()
   "Highlight the headers specified by `rmail-highlighted-headers'.
-Uses the face specified by `rmail-highlight-face'."
+Uses the face `rmail-highlight'."
   (if rmail-highlighted-headers
       (save-excursion
 	(search-forward "\n\n" nil 'move)
@@ -3020,11 +3006,7 @@ rmail-highlight-headers
 	  (narrow-to-region (point-min) (point))
 	  (let ((case-fold-search t)
 		(inhibit-read-only t)
-		;; When rmail-highlight-face is removed, just
-		;; use 'rmail-highlight here.
-		(face (or rmail-highlight-face
-			  (if (face-differs-from-default-p 'bold)
-			      'bold 'highlight)))
+		(face 'rmail-highlight)
 		;; List of overlays to reuse.
 		(overlays rmail-overlay-list))
 	    (goto-char (point-min))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 641a2e5315..1f2dcc4755 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -2539,11 +2539,6 @@ completion-file-name-table
               all))))))
     (file-error nil)))               ;PCM often calls with invalid directories.
 
-(defvar read-file-name-predicate nil
-  "Current predicate used by `read-file-name-internal'.")
-(make-obsolete-variable 'read-file-name-predicate
-                        "use the regular PRED argument" "23.2")
-
 (defun completion--sifn-requote (upos qstr)
   ;; We're looking for `qpos' such that:
   ;; (equal (substring (substitute-in-file-name qstr) 0 upos)
diff --git a/lisp/mouse.el b/lisp/mouse.el
index a06ca2a56c..06fdca12b9 100644
--- a/lisp/mouse.el
+++ b/lisp/mouse.el
@@ -2303,9 +2303,6 @@ mouse-buffer-menu-split
     ;; Few buffers--put them all in one pane.
     (list (cons title alist))))
 \f
-(define-obsolete-function-alias
-  'mouse-choose-completion 'choose-completion "23.2")
-
 ;; Font selection.
 
 (defun font-menu-add-default ()
diff --git a/lisp/progmodes/antlr-mode.el b/lisp/progmodes/antlr-mode.el
index bf56a7ee49..24e1f8831a 100644
--- a/lisp/progmodes/antlr-mode.el
+++ b/lisp/progmodes/antlr-mode.el
@@ -695,7 +695,7 @@ antlr-mode-map
     (define-key map "\e\C-e" 'antlr-end-of-rule)
     (define-key map "\C-c\C-a" 'antlr-beginning-of-body)
     (define-key map "\C-c\C-e" 'antlr-end-of-body)
-    (define-key map "\C-c\C-f" 'c-forward-into-nomenclature)
+    (define-key map "\C-c\C-f" 'subword-forward)
     (define-key map "\C-c\C-b" 'c-backward-into-nomenclature)
     (define-key map "\C-c\C-c" 'comment-region)
     (define-key map "\C-c\C-v" 'antlr-hide-actions)
@@ -745,7 +745,7 @@ antlr-mode-menu
      ["Backward Statement" c-beginning-of-statement t]
      ["Forward Statement" c-end-of-statement t]
      ["Backward Into Nomencl." c-backward-into-nomenclature t]
-     ["Forward Into Nomencl." c-forward-into-nomenclature t])
+     ["Forward Into Nomencl." subword-forward t])
     ["Indent Region" indent-region
      :active (and (not buffer-read-only) (c-region-is-active-p))]
     ["Comment Out Region" comment-region
diff --git a/lisp/progmodes/cc-cmds.el b/lisp/progmodes/cc-cmds.el
index 1b557c41a5..4425e275ac 100644
--- a/lisp/progmodes/cc-cmds.el
+++ b/lisp/progmodes/cc-cmds.el
@@ -1554,19 +1554,6 @@ c-toggle-cpp-indent-to-body
 (declare-function c-backward-subword "ext:cc-subword" (&optional arg))
 
 ;; "nomenclature" functions + c-scope-operator.
-(defun c-forward-into-nomenclature (&optional arg)
-  "Compatibility alias for `c-forward-subword'."
-  (interactive "p")
-  (if (fboundp 'subword-mode)
-      (progn
-        (require 'subword)
-        (subword-forward arg))
-    (require 'cc-subword)
-    (c-forward-subword arg)))
-(make-obsolete 'c-forward-into-nomenclature
-               (if (fboundp 'subword-mode) 'subword-forward 'c-forward-subword)
-               "23.2")
-
 (defun c-backward-into-nomenclature (&optional arg)
   "Compatibility alias for `c-backward-subword'."
   (interactive "p")
diff --git a/lisp/progmodes/idlwave.el b/lisp/progmodes/idlwave.el
index f7e53ec02d..2d591dd91d 100644
--- a/lisp/progmodes/idlwave.el
+++ b/lisp/progmodes/idlwave.el
@@ -7118,7 +7118,7 @@ idlwave-default-choose-completion
   (apply 'idlwave-choose 'default-choose-completion args))
 
 (defun idlwave-make-modified-completion-map-xemacs (old-map)
-  "Replace `choose-completion' and `mouse-choose-completion' in OLD-MAP."
+  "Replace `choose-completion' in OLD-MAP."
   (let ((new-map (copy-keymap old-map)))
     (define-key new-map [button3up] 'idlwave-mouse-completion-help)
     (define-key new-map [button3] (lambda ()
@@ -7141,12 +7141,10 @@ idlwave-display-completion-list-emacs
 		(current-local-map)))))))
 
 (defun idlwave-make-modified-completion-map-emacs (old-map)
-  "Replace `choose-completion' and `mouse-choose-completion' in OLD-MAP."
+  "Replace `choose-completion' in OLD-MAP."
   (let ((new-map (copy-keymap old-map)))
     (substitute-key-definition
      'choose-completion 'idlwave-choose-completion new-map)
-    (substitute-key-definition
-     'mouse-choose-completion 'idlwave-mouse-choose-completion new-map)
     (define-key new-map [mouse-3] 'idlwave-mouse-completion-help)
     new-map))
 
@@ -7155,11 +7153,6 @@ idlwave-choose-completion
   (interactive (list last-nonmenu-event))
   (apply 'idlwave-choose 'choose-completion args))
 
-(defun idlwave-mouse-choose-completion (&rest args)
-  "Click on an alternative in the `*Completions*' buffer to choose it."
-  (interactive "e")
-  (apply 'idlwave-choose 'mouse-choose-completion args))
-
 ;;----------------------------------------------------------------------
 ;;----------------------------------------------------------------------
 
diff --git a/lisp/progmodes/vhdl-mode.el b/lisp/progmodes/vhdl-mode.el
index 9cd84cf713..3d66483b83 100644
--- a/lisp/progmodes/vhdl-mode.el
+++ b/lisp/progmodes/vhdl-mode.el
@@ -2304,10 +2304,6 @@ vhdl-last-input-event
     (defvaralias 'vhdl-last-input-event 'last-input-char)
   (defvaralias 'vhdl-last-input-event 'last-input-event))
 
-;; `help-print-return-message' changed to `print-help-return-message' in Emacs
-;;;(unless (fboundp 'help-print-return-message)
-;;;  (defalias 'help-print-return-message 'print-help-return-message))
-
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Compatibility with older VHDL Mode versions
 
diff --git a/lisp/progmodes/xscheme.el b/lisp/progmodes/xscheme.el
index 8dfb3a40dd..c6997862f7 100644
--- a/lisp/progmodes/xscheme.el
+++ b/lisp/progmodes/xscheme.el
@@ -446,8 +446,6 @@ xscheme-enter-interaction-mode
 	    (scheme-interaction-mode-initialize)
 	    (scheme-interaction-mode t)))))
 
-(define-obsolete-function-alias 'advertised-xscheme-send-previous-expression
-  'xscheme-send-previous-expression "23.2")
 \f
 ;;;; Debugger Mode
 
diff --git a/lisp/simple.el b/lisp/simple.el
index 1cb93c5722..5a8816c8d6 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -5052,7 +5052,8 @@ append-next-kill
 
 The argument is used for internal purposes; do not supply one."
   (interactive "p")
-  ;; We don't use (interactive-p), since that breaks kbd macros.
+  ;; We don't use (called-interactively-p 'interactive), since that
+  ;; breaks kbd macros.
   (if interactive
       (progn
 	(setq this-command 'kill-region)
@@ -6114,8 +6115,6 @@ pop-mark
     (pop mark-ring))
   (deactivate-mark))
 
-(define-obsolete-function-alias
-  'exchange-dot-and-mark 'exchange-point-and-mark "23.3")
 (defun exchange-point-and-mark (&optional arg)
   "Put the mark where point is now, and point where the mark is now.
 This command works even when the mark is not active,
@@ -8400,18 +8399,6 @@ completion-list-insert-choice-function
 between BEG and END with TEXT.  Expected to be set buffer-locally
 in the *Completions* buffer.")
 
-(defvar completion-base-size nil
-  "Number of chars before point not involved in completion.
-This is a local variable in the completion list buffer.
-It refers to the chars in the minibuffer if completing in the
-minibuffer, or in `completion-reference-buffer' otherwise.
-Only characters in the field at point are included.
-
-If nil, Emacs determines which part of the tail end of the
-buffer's text is involved in completion by comparing the text
-directly.")
-(make-obsolete-variable 'completion-base-size 'completion-base-position "23.2")
-
 (defun delete-completion-window ()
   "Delete the completion list window.
 Go to the window from which completion was requested."
@@ -8465,7 +8452,6 @@ choose-completion
   (run-hooks 'mouse-leave-buffer-hook)
   (with-current-buffer (window-buffer (posn-window (event-start event)))
     (let ((buffer completion-reference-buffer)
-          (base-size completion-base-size)
           (base-position completion-base-position)
           (insert-function completion-list-insert-choice-function)
           (choice
@@ -8492,10 +8478,6 @@ choose-completion
         (choose-completion-string
          choice buffer
          (or base-position
-             (when base-size
-               ;; Someone's using old completion code that doesn't know
-               ;; about base-position yet.
-               (list (+ base-size (field-beginning))))
              ;; If all else fails, just guess.
              (list (choose-completion-guess-base-position choice)))
          insert-function)))))
@@ -8523,10 +8505,6 @@ choose-completion-guess-base-position
         (forward-char 1))
       (point))))
 
-(defun choose-completion-delete-max-match (string)
-  (declare (obsolete choose-completion-guess-base-position "23.2"))
-  (delete-region (choose-completion-guess-base-position string) (point)))
-
 (defvar choose-completion-string-functions nil
   "Functions that may override the normal insertion of a completion choice.
 These functions are called in order with three arguments:
@@ -8555,13 +8533,6 @@ choose-completion-string
   ;; unless it is reading a file name and CHOICE is a directory,
   ;; or completion-no-auto-exit is non-nil.
 
-  ;; Some older code may call us passing `base-size' instead of
-  ;; `base-position'.  It's difficult to make any use of `base-size',
-  ;; so we just ignore it.
-  (unless (consp base-position)
-    (message "Obsolete `base-size' passed to choose-completion-string")
-    (setq base-position nil))
-
   (let* ((buffer (or buffer completion-reference-buffer))
 	 (mini-p (minibufferp buffer)))
     ;; If BUFFER is a minibuffer, barf unless it's the currently
@@ -8617,8 +8588,7 @@ completion-list-mode
  to select the completion near point.
 Or click to select one with the mouse.
 
-\\{completion-list-mode-map}"
-  (set (make-local-variable 'completion-base-size) nil))
+\\{completion-list-mode-map}")
 
 (defun completion-list-mode-finish ()
   "Finish setup of the completions buffer.
@@ -8655,14 +8625,11 @@ completion-setup-function
           (if minibuffer-completing-file-name
               (file-name-as-directory
                (expand-file-name
-                (buffer-substring (minibuffer-prompt-end)
-                                  (- (point) (or completion-base-size 0))))))))
+                (buffer-substring (minibuffer-prompt-end) (point)))))))
     (with-current-buffer standard-output
-      (let ((base-size completion-base-size) ;Read before killing localvars.
-            (base-position completion-base-position)
+      (let ((base-position completion-base-position)
             (insert-fun completion-list-insert-choice-function))
         (completion-list-mode)
-        (set (make-local-variable 'completion-base-size) base-size)
         (set (make-local-variable 'completion-base-position) base-position)
         (set (make-local-variable 'completion-list-insert-choice-function)
 	     insert-fun))
diff --git a/lisp/subr.el b/lisp/subr.el
index 8eb4cf452d..f86d0e00f4 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -4669,13 +4669,6 @@ do-after-load-evaluation
   ;; Finally, run any other hook.
   (run-hook-with-args 'after-load-functions abs-file))
 
-(defun eval-next-after-load (file)
-  "Read the following input sexp, and run it whenever FILE is loaded.
-This makes or adds to an entry on `after-load-alist'.
-FILE should be the name of a library, with no directory name."
-  (declare (obsolete eval-after-load "23.2"))
-  (eval-after-load file (read)))
-
 \f
 (defun display-delayed-warnings ()
   "Display delayed warnings from `delayed-warnings-list'.
@@ -5135,29 +5128,6 @@ called-interactively-p
                    . ,_))
          t)))))
 
-(defun interactive-p ()
-  "Return t if the containing function was run directly by user input.
-This means that the function was called with `call-interactively'
-\(which includes being called as the binding of a key)
-and input is currently coming from the keyboard (not a keyboard macro),
-and Emacs is not running in batch mode (`noninteractive' is nil).
-
-The only known proper use of `interactive-p' is in deciding whether to
-display a helpful message, or how to display it.  If you're thinking
-of using it for any other purpose, it is quite likely that you're
-making a mistake.  Think: what do you want to do when the command is
-called from a keyboard macro or in batch mode?
-
-To test whether your function was called with `call-interactively',
-either (i) add an extra optional argument and give it an `interactive'
-spec that specifies non-nil unconditionally (such as \"p\"); or (ii)
-use `called-interactively-p'.
-
-To test whether a function can be called interactively, use
-`commandp'."
-  (declare (obsolete called-interactively-p "23.2"))
-  (called-interactively-p 'interactive))
-
 (defun internal-push-keymap (keymap symbol)
   (let ((map (symbol-value symbol)))
     (unless (memq keymap map)
diff --git a/lisp/term.el b/lisp/term.el
index 149405fa41..40c8481fdf 100644
--- a/lisp/term.el
+++ b/lisp/term.el
@@ -4102,53 +4102,6 @@ term-replace-by-expanded-filename
   (term-dynamic-complete-filename))
 
 
-(defun term-dynamic-simple-complete (stub candidates)
-  "Dynamically complete STUB from CANDIDATES list.
-This function inserts completion characters at point by completing STUB from
-the strings in CANDIDATES.  A completions listing may be shown in a help buffer
-if completion is ambiguous.
-
-Returns nil if no completion was inserted.
-Returns `sole' if completed with the only completion match.
-Returns `shortest' if completed with the shortest of the completion matches.
-Returns `partial' if completed as far as possible with the completion matches.
-Returns `listed' if a completion listing was shown.
-
-See also `term-dynamic-complete-filename'."
-  (declare (obsolete completion-in-region "23.2"))
-  (let* ((completion-ignore-case nil)
-	 (completions (all-completions stub candidates)))
-    (cond ((null completions)
- 	   (message "No completions of %s" stub)
-	   nil)
- 	  ((= 1 (length completions))	; Gotcha!
- 	   (let ((completion (car completions)))
- 	     (if (string-equal completion stub)
- 		 (message "Sole completion")
- 	       (insert (substring completion (length stub)))
- 	       (message "Completed"))
-	     (when term-completion-addsuffix (insert " "))
-	     'sole))
- 	  (t				; There's no unique completion.
- 	   (let ((completion (try-completion stub candidates)))
- 	     ;; Insert the longest substring.
- 	     (insert (substring completion (length stub)))
- 	     (cond ((and term-completion-recexact term-completion-addsuffix
- 			 (string-equal stub completion)
- 			 (member completion completions))
- 		    ;; It's not unique, but user wants shortest match.
- 		    (insert " ")
- 		    (message "Completed shortest")
-		    'shortest)
- 		   ((or term-completion-autolist
- 			(string-equal stub completion))
- 		    ;; It's not unique, list possible completions.
- 		    (term-dynamic-list-completions completions)
-		    'listed)
- 		   (t
-		    (message "Partially completed")
-		    'partial)))))))
-
 (defun term-dynamic-list-filename-completions ()
   "List in help buffer possible completions of the filename at point."
   (interactive)
@@ -4178,7 +4131,7 @@ term-dynamic-list-completions
 		 (eq (window-buffer (posn-window (event-start first)))
 		     (get-buffer "*Completions*"))
 		 (memq (key-binding key)
-                       '(mouse-choose-completion choose-completion))))
+                       '(choose-completion))))
 	  ;; If the user does choose-completion with the mouse,
 	  ;; execute the command, then delete the completion window.
 	  (progn
diff --git a/test/manual/cedet/cedet-utests.el b/test/manual/cedet/cedet-utests.el
index 124b49907d..ee6be438dd 100644
--- a/test/manual/cedet/cedet-utests.el
+++ b/test/manual/cedet/cedet-utests.el
@@ -150,7 +150,7 @@ cedet-utest
       ;; Cleanup stray input and events that are in the way.
       ;; Not doing this causes sit-for to not refresh the screen.
       ;; Doing this causes the user to need to press keys more frequently.
-      (when (and (interactive-p) (input-pending-p))
+      (when (and (called-interactively-p 'interactive) (input-pending-p))
 	(if (fboundp 'read-event)
 	    (read-event)
 	  (read-char)))
@@ -497,11 +497,11 @@ pulse-test
 	(error (concat "Pulse test only works on versions of Emacs"
 		       " that support pulsing")))
     ;; Run the tests
-    (when (interactive-p)
+    (when (called-interactively-p 'interactive)
       (message "<Press a key> Pulse one line.")
       (read-char))
     (pulse-momentary-highlight-one-line (point))
-    (when (interactive-p)
+    (when (called-interactively-p 'interactive)
       (message "<Press a key> Pulse a region.")
       (read-char))
     (pulse-momentary-highlight-region (point)
@@ -510,11 +510,11 @@ pulse-test
 					    (forward-char 30)
 					  (error nil))
 					(point)))
-    (when (interactive-p)
+    (when (called-interactively-p 'interactive)
       (message "<Press a key> Pulse line a specific color.")
       (read-char))
     (pulse-momentary-highlight-one-line (point) 'mode-line)
-    (when (interactive-p)
+    (when (called-interactively-p 'interactive)
       (message "<Press a key> Pulse a pre-existing overlay.")
       (read-char))
     (let* ((start (point-at-bol))
@@ -530,7 +530,7 @@ pulse-test
 	  (delete-overlay o)
 	(error "Non-temporary overlay was deleted!"))
       )
-    (when (interactive-p)
+    (when (called-interactively-p 'interactive)
       (message "Done!"))))
 
 (provide 'cedet-utests)
diff --git a/test/manual/cedet/semantic-tests.el b/test/manual/cedet/semantic-tests.el
index 53552be06b..a0899cb932 100644
--- a/test/manual/cedet/semantic-tests.el
+++ b/test/manual/cedet/semantic-tests.el
@@ -235,7 +235,7 @@ semantic-lex-spp-write-utest
       (set-buffer buff)
       (semantic-lex-spp-write-test)
       (kill-buffer buff)
-      (when (not (interactive-p))
+      (when (not (called-interactively-p 'interactive))
         (kill-buffer "*SPP Write Test*"))
       )))
 
@@ -276,7 +276,7 @@ semantic-symref-test-count-hits-in-tag
        target (lambda (start end prefix) (setq Lcount (1+ Lcount)))
        (semantic-tag-start tag)
        (semantic-tag-end tag))
-      (when (interactive-p)
+      (when (called-interactively-p 'interactive)
 	(message "Found %d occurrences of %s in %.2f seconds"
 		 Lcount (semantic-tag-name target)
 		 (semantic-elapsed-time start nil)))
-- 
2.28.0

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> I'm attaching a patch to remove most things declared obsolete in 23.2
> and 23.3.  (Nothing was declared obsolete in 23.4 AFAICT.)
> 
> There are still some items obsoleted in 23.x left to delete in Semantic.
> I intend to do that in a separate patch.
> 
> Comments welcome.


PLEASE do not remove `interactive-p'.

I have over 2500 lines that use `interactive-p'.

Is there some problem with keeping its simple
definition as (called-interactively 'interactive)?

Re: Delete variables obsolete since Emacs 23

[Lars Ingebrigtsen]

Drew Adams <drew.adams@oracle.com> writes:

> PLEASE do not remove `interactive-p'.
>
> I have over 2500 lines that use `interactive-p'.
>
> Is there some problem with keeping its simple
> definition as (called-interactively 'interactive)?

Is there some problem with you adding your own interactive-p if you want
to use that name instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> > Is there some problem with keeping its simple
> > definition as (called-interactively 'interactive)?
> 
> Is there some problem with you adding your own interactive-p
> if you want to use that name instead?

Is there some problem with Emacs keeping the existing
simple definition of `interactive-p'?

(defun interactive-p () (called-interactively-p 'interactive))

What's the problem with that?

With that longstanding definition _in Emacs_, no one
needs to define it in a million files or create a
file for it and require that file everywhere.

Re: Delete variables obsolete since Emacs 23

[Ulrich Mueller]

>>>>> On Fri, 14 Aug 2020, Drew Adams wrote:

>> I'm attaching a patch to remove most things declared obsolete in 23.2
>> and 23.3.  (Nothing was declared obsolete in 23.4 AFAICT.)
>> 
>> There are still some items obsoleted in 23.x left to delete in Semantic.
>> I intend to do that in a separate patch.
>> 
>> Comments welcome.

> PLEASE do not remove `interactive-p'.

> I have over 2500 lines that use `interactive-p'.

> Is there some problem with keeping its simple
> definition as (called-interactively 'interactive)?

I second this.

Removal of interactive-p would unnecessarily break many third-party
elisp libraries, while keeping the backwards compatible definition in
Emacs requires only a single line.

RE: Delete variables obsolete since Emacs 23

[Stefan Kangas]

> PLEASE do not remove `interactive-p'.

This has been declared obsolete for a decade now.

Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Thu Oct 1 17:47:38 2009 +0000

    * eval.c (Fcalled_interactively_p): Add `kind' argument.
    * subr.el (interactive-p): Mark obsolete.

It has been discussed before:
https://lists.gnu.org/archive/html/emacs-devel/2015-07/msg00326.html

And here:
https://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01740.html

(I'm sure there are other threads that I couldn't find, if so please
link them.)

> I have over 2500 lines that use `interactive-p'.

Yet once we obsolete something, there surely has to come a point when we
can finally delete it.

Emacs is already extremely conservative when it comes to dropping
backwards support.  AFAICT, we generally don't delete anything until
five major releases after it was first obsoleted.

Best regards,
Stefan Kangas

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

There's no obligation, or any good reason, to
remove it.  Just leave it "obsolete", please.
It's not hurting anyone.  You're not really
helping anyone by removing it.

To repeat the (unanswered) question:

> > > Is there some problem with keeping its simple
> > > definition as (called-interactively 'interactive)?
> > 
> > Is there some problem with you adding your own
> > interactive-p if you want to use that name instead?
>
> Is there some problem with Emacs keeping the
> existing simple definition of `interactive-p'?
>
> (defun interactive-p () (called-interactively-p 'interactive))
>
> What's the problem with that?

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

> There's no obligation, or any good reason, to remove it.

Actually, `interactive-p` is harmful, so you should remove as many uses
of it as you can.  `called-interactive-p` is not a good replacement
for it.


        Stefan

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> > There's no obligation, or any good reason, to remove it.
> 
> Actually, `interactive-p` is harmful, so you should remove as many uses
> of it as you can.  `called-interactive-p` is not a good replacement
> for it.

It's not harming me.  Removing it will.

Emacs allows (called-interactively-p 'interactive).
It should also allow (interactive-p) which does the
same thing.

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

ISTR that we declared interactive-p obsolete because it is an not a
correct solution to the problem it is meant to solve.
The correct solution is to give the function an optional argument
and an interactive spec which provides a non-nil value for that argument
in interactive calls.

There are cases where that gives the right answer,
while interactive-p gives the wrong answer.

The point is, you really should change each of those functions
so that it will not misbehave in those special situations.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> ISTR that we declared interactive-p obsolete because it is an not a
> correct solution to the problem it is meant to solve.

Then neither is (called-interactively-p 'interactive).
Goose ... gander.

> The correct solution is to give the function an optional argument
> and an interactive spec which provides a non-nil value for that argument
> in interactive calls.

I already do that where it makes sense (e.g. to
show a message).  And that lesson is given in the
doc string of `called-interactively-p' (which is
_not_ obsolete, let alone being removed).

> There are cases where that gives the right answer,
> while interactive-p gives the wrong answer.

`called-interactively-p' is just as brittle as
`interactive-p' - see its doc string.

The main difference is for keyboard-macro use -
(called-interactively-p 'any) versus
(called-interactively-p 'interactive).

> The point is, you really should change each of those functions
> so that it will not misbehave in those special situations.

My code is backward-compatible, and there's too
much of it to want to change to

(if (fboundp 'called-interactively-p)
    (called-interactively-p 'interactive)
  (interactive-p))

I have no objection to Emacs having declared
`interactive-p' obsolete (well, I kinda do,
but not much, and it's not important at all).

I object to Emacs _removing_ it.  I see no reason
for that.  And no one has given a reason, so far.

Everything said against `interactive-p' so far is
true also of (called-interactively-p 'interactive).
Emacs allows the latter - it's not obsolete.  Emacs
should continue to allow (interactive-p), which is
_identical_.

As far as I'm concerned, Emacs can feel free to
continue to call `interactive-p' names.  Shout
from the rooftops that it's awful, evil, stupid,
obsolete, whatever.  Doesn't bother me.  What
bothers me is removing it.  Sticks & stones...

I didn't complain about the other removals being
made, BTW.

I didn't complain that `buffer-menu-mode-hook' was
deprecated in favor of the same with capital `B',
or that the former is now being removed.  Likewise,
for deprecating (and now removing)
`print-help-return-message' in favor of
`help-print-return-message'.  Lots of obsolete
things are being removed now, with no complaint
from me, although I had to make code adjustments.

`interactive-p' is used much more than the others.

(Mea culpa: I said I had 2500 occurrences of
`interactive-p'.  I actually have 406 occurrences.
A first grep included extraneous test files etc.)

Nothing is gained by removing `interactive-p'.
No one has been able to point out any gain.  Only
pain, no gain.

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

>> The correct solution is to give the function an optional argument
>> and an interactive spec which provides a non-nil value for that argument
>> in interactive calls.
> I already do that where it makes sense (e.g. to show a message).

I think you misunderstand: it makes sense everywhere where it
technically *can* be used.

There are still circumstances where `called-interactively-p` is
necessary (typically in cases where the arglist cannot be extended with
a new argument (mostly because it already uses `&rest`), or in advice),
but it's rare.

> I have no objection to Emacs having declared `interactive-p' obsolete
> (well, I kinda do, but not much, and it's not important at all).
>
> I object to Emacs _removing_ it.  I see no reason for that.  And no
> one has given a reason, so far.

You know the reason: because it's obsolete and deprecated.


        Stefan

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think we are miscommunicating.  You seem to be talking about
called-interactively-p.  I am talking about making your function take
an optional argument.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> I think we are miscommunicating.  You seem to be talking about
> called-interactively-p.  I am talking about making your function take
> an optional argument.

What "your function" are you talking about?
This is about the standard Emacs function
`interactive-p'.

What optional argument would you have it take,
and why?  And how would that solve the problem
of existing calls to `(interactive-p)'?

But yes, it does sound like we've perhaps
miscommunicated.

`interactive-p' has existed since nearly Day
One.  `called-interactively-p' was introduced
to replace/"fix" it - presumably instead of
fixing it directly by adding an optional arg
and leaving the default behavior (no arg) as is.

And `interactive-p' was declared obsolete.
I have no problem with that declaration, even
if I'd have preferred to see `interactive-p'
fixed directly (previous paragraph).

My only request is to not remove `interactive-p'.
My request is to please leave it "obsolete".

RE: Delete variables obsolete since Emacs 23

[Gregory Heytings via Emacs development discussions.]

Drew Adams:
>
> My only request is to not remove `interactive-p'.
> My request is to please leave it "obsolete".
>

Perhaps I'm misunderstanding something, but I don't even understand why it 
is marked "obsolete".  If (called-interactively-p 'interactive) is a 
common idiom (and it seems to be, the docstring of called-interactively-p 
indicates that "the only known proper use of 'interactive' for KIND is in 
deciding whether to display a helpful message, or how to display it."), it 
would make perfect sense to have, in subr.el:

(defun interactive-p ()
   "Shorthand for (called-interactively-p 'interactive).
See `called-interactively-p'."
   (called-interactively-p 'interactive))

Unless of course there is a long-term goal to replace "interactive-p" with 
some other definition.

Gregory

Re: Delete variables obsolete since Emacs 23

[Jeff Norden]

Drew Adams wrote:
> My only request is to not remove `interactive-p'.
> My request is to please leave it "obsolete".

The elisp manual says:

    You can mark a named function as "obsolete", meaning that it may be
    removed at some point in the future.  This causes Emacs to warn that the
    function is obsolete whenever it byte-compiles code containing that
    function, and whenever it displays the documentation for that function.
    In all other respects, an obsolete function behaves like any other
    function.

The phrase "may be removed" seems a bit vague.  Would "will be removed" or
"will probably be removed" be more accurate?

If so, perhaps it would help for obsolete functions and variables to also
include something like a 'target-expiration-date', which could be a release
number rather than an actual date.  A target-expiration of "31.1" would mean
that the current intent is that the function/variable will no longer exist
when version 31.1 is released.  The chromium project does something similar
for its experimental "flags" that can be used to customize the user interface.

From where I sit, it seems plausible that someone might view interactive-p
as just an obsolete way of writing (called-interactively-p 'interactive),
which exists for backward compatibility, and would be available long term.
It seems that the last part is a mis-interpretation of the intent of marking
it as obsolete.

The fact that emacs maintains backward compatibility so well is one of the
features that I appreciate the most.  But, in this case, the fact that
interactive-p has been obsolete for so long may give the impression that it
would continue to be available.  I can see arguments on both sides of the coin
for why interactive-p should or should not continue to exist.  If the
underlying fact is that virtually all functions marked as obsolete will
eventually be removed, then this should be made clear (or more clear
than it currently seems to be).  Then the issue of whether or not
something should be marked as obsolete would have a better focus.

Also, it might be worthwhile for the elisp manual to mention one other
difference that obsolete functions have.  When an interactive command is
obsolete, it no longer appears as a possible completion when you press
[tab] while entering a command-name after M-x.

Regards,
-Jeff

Re: Delete variables obsolete since Emacs 23

[Eli Zaretskii]

> From: Jeff Norden <jnorden@tntech.edu>
> Date: Mon, 17 Aug 2020 20:55:56 -0500
> Cc: ghe@sdf.org, stefankangas@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> The elisp manual says:
> 
>     You can mark a named function as "obsolete", meaning that it may be
>     removed at some point in the future.  This causes Emacs to warn that the
>     function is obsolete whenever it byte-compiles code containing that
>     function, and whenever it displays the documentation for that function.
>     In all other respects, an obsolete function behaves like any other
>     function.
> 
> The phrase "may be removed" seems a bit vague.  Would "will be removed" or
> "will probably be removed" be more accurate?

No, it won't.  Primarily because we don't really know whether we will
remove it, let alone when.  It depends on too many factors that we
cannot predict.

> The fact that emacs maintains backward compatibility so well is one of the
> features that I appreciate the most.  But, in this case, the fact that
> interactive-p has been obsolete for so long may give the impression that it
> would continue to be available.

I think the gentle annoyance we have now strikes the right balance
between not letting people forget the fact of obsolescence and not
annoying them too much.  We are talking to adults, not to children, so
we can rely on them doing TRT.

Re: Delete variables obsolete since Emacs 23

[Gregory Heytings via Emacs development discussions.]

>>> The elisp manual says:
>>>
>>> You can mark a named function as "obsolete", meaning that it may be 
>>> removed at some point in the future.  This causes Emacs to warn that 
>>> the function is obsolete whenever it byte-compiles code containing 
>>> that function, and whenever it displays the documentation for that 
>>> function. In all other respects, an obsolete function behaves like any 
>>> other function.
>>
>> The phrase "may be removed" seems a bit vague.  Would "will be removed" 
>> or "will probably be removed" be more accurate?
>
> No, it won't.  Primarily because we don't really know whether we will 
> remove it, let alone when.  It depends on too many factors that we 
> cannot predict.
>

In that case, would a two-step process not be better?  First declaring the 
function "obsolete", and when it is known that it will be removed declare 
it "pending-removal" with a target major version.

Gregory

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> >> The phrase "may be removed" seems a bit vague.  Would "will be removed"
> >> or "will probably be removed" be more accurate?
> >
> > No, it won't.  Primarily because we don't really know whether we will
> > remove it, let alone when.  It depends on too many factors that we
> > cannot predict.
> >
> 
> In that case, would a two-step process not be better?  First declaring the
> function "obsolete", and when it is known that it will be removed declare
> it "pending-removal" with a target major version.

In my experience, organizations typically avoid a
lot of commitments to future changes.  Instead, it's
about _no longer_ committing to something that's
currently committed to.  It's about removal of
commitments, not imposition of new commitments.

(Sometimes there's informal/unofficial mention to
some users that it's likely XYZ will be removed
at a specific point.  But even that is generally
avoided.)

Typically there are two possible stages:

1. Deprecation (sometimes calling the thing "obsolete").
   The thing _remains supported_.  There's no longer a
   _commitment_ to active development - that's all.

   Users are told that the thing _might_ be desupported
   at some time in the future.

   Because of the removal of a _commitment_ to actively
   develop AND the removal of a _commitment_ that the
   thing will remain forever, users are advised to
   start moving away from using the thing.

2. Desupport.  This is _optional_.  A deprecated thing
   need never be desupported.  And typically there is
   some (sometimes announced, sometimes not) minimal
   time period or number of intermediate releases,
   needed between deprecation and desupport.

   Something that's desupported gets no support.
   Trying to use it typically results in a no-op or
   raising an error.
___

That's my experience.  YMMV.  And of course nothing
requires GNU Emacs to follow any particular outside
convention or typical behavior.

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

> In that case, would a two-step process not be better?  First declaring the
> function "obsolete", and when it is known that it will be removed declare 
> it "pending-removal" with a target major version.

I'm not sure it would make much difference to the speed at which people
move away from using the obsolete feature (which is really all that
matters, in the end).

Another approach I toyed with is to change the `make-obsolete*`
functions so that they actually *remove* the obsolete feature when used
in development builds (i.e. builds with version XXX.0.50).
Just like that it's a bit blunt, so it'd have to come with some knobs so
the user can ask to disable this for some particular
funs/vars, and also so it's only applied when the fun/var has been
obsolete for at least 1 (or maybe 2) versions.


        Stefan

Re: Delete variables obsolete since Emacs 23

[Gregory Heytings via Emacs development discussions.]

>
>> In that case, would a two-step process not be better?  First declaring 
>> the function "obsolete", and when it is known that it will be removed 
>> declare it "pending-removal" with a target major version.
>
> I'm not sure it would make much difference to the speed at which people 
> move away from using the obsolete feature (which is really all that 
> matters, in the end).
>

FWIW, I think it would.  With the current situation there is no concrete 
reason to move away from such a feature.  As Eli wrote "we don't really 
know whether we will remove [an absolete feature], let alone when", so it 
could very wall stay there forever, and I can understand Drew's viewpoint 
who did not make the effort to update the 2500 lines of his codebase in 
which 'interactive-p' is used.  If he had been warned, when Emacs 25 was 
released, with:

(declare (pending-removal called-interactively-p "27"))

he would have known that it would be removed in a medium term, and would 
have had a reason to update his codebase.

That being said, about this particular feature, I still don't understand 
why it is obsolete.  It is just a way to express a common pattern in a 
shorter way, like say (1+ x) and (+ x 1).

Gregory

Re: Delete variables obsolete since Emacs 23

[Eli Zaretskii]

> Date: Wed, 19 Aug 2020 13:54:50 +0000
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> > I'm not sure it would make much difference to the speed at which people 
> > move away from using the obsolete feature (which is really all that 
> > matters, in the end).
> >
> 
> FWIW, I think it would.  With the current situation there is no concrete 
> reason to move away from such a feature.  As Eli wrote "we don't really 
> know whether we will remove [an absolete feature], let alone when", so it 
> could very wall stay there forever, and I can understand Drew's viewpoint 
> who did not make the effort to update the 2500 lines of his codebase in 
> which 'interactive-p' is used.  If he had been warned, when Emacs 25 was 
> released, with:
> 
> (declare (pending-removal called-interactively-p "27"))
> 
> he would have known that it would be removed in a medium term, and would 
> have had a reason to update his codebase.

You seem to be under the impression that we have difficulty removing
obsolete features when we decide it's time for that.  But the reality
is we do remove such obsolete features.  It's the decision to do so
that's difficult, not the execution of the removal once we do decide.

P.S.  Could you please refrain from using Reply-To in your messages?
It causes emacs-devel to appear twice in the addresses of the
responses, so I have to manually delete one of them.  TIA.

Re: Delete variables obsolete since Emacs 23

[Jeff Norden]

Stefan Monnier wrote:
> IMNSHO it doesn't: if we don't intend to remove it ever, then we don't
> declare it obsolete.
>
> Of course, declaring it obsolete doesn't guarantee that we will remove
> it: it's just a declaration of intention.

Eli Zaretskii wrote:
> You seem to be under the impression that we have difficulty removing
> obsolete features when we decide it's time for that.  But the reality
> is we do remove such obsolete features.  It's the decision to do so
> that's difficult, not the execution of the removal once we do decide.

This makes it clear, I think.  Perhaps the elisp manual could say something
like:

  When the Emacs developers mark a function or variable as obsolete, this
  signals the intent to remove it at some point in the future.  You should
  refrain from using it in new code, and should work towards removing it
  from existing code that you plan to maintain for the long term.

But maybe that is not really necessary.

---
Lars Ingebrigtsen wrote:
> ...this is a pretty entertaining read:
> https://medium.com/@steve.yegge/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc

Thanks for this link, which I really enjoyed.  In addition to characterizing
Emacs as "a sort of hybrid between Windows Notepad, a monolithic-kernel
operating system, and the International Space Station," Yegge says:

  I’m still using software that I wrote for Emacs back in 1995. ...
  Every once in a while it might require a minor tweak, but it’s really
  quite rare. I’m not aware of anything I’ve ever written for Emacs (and
  I’ve written a lot) that was ever forced into re-architecture.

  ... when they make an API obsolete, they are basically saying: "You really
  shouldn't use this approach, because even though it works, it suffers from
  various deficiencies which we enumerate here. But in the end it’s your
  call."

  Whereas in the Google world, deprecation means: "We are breaking our
  commitments to you." It really does.

---
As far as interactive-p is concerned, I grepped thru my personal stuff and
found exactly one instance of interactive-p, which dates to April 1993.
It is used to make a function behave differently on the rare times that I
invoke it via M-x.  I changed the interactive spec and introduced an optional
arg.  I agree this is a better solution - but I would also argue that this use
of interactive-p was harmless, since I'm the only person who uses this snippet
of code, and interactive-p did a perfectly good job for my use case.

I can see an argument for removing interactive-p as a way to encourage people
to carefully examine where it was used, and replace it with something better
when possible.  But the docstrings and elisp manual already do this, and it
isn't clear to me that removing the function would accomplish much more.
There is nothing gained by replacing (interactive-p) with
(called-interactively 'interactive) where it is really necessary.  And nothing
can prevent someone from just blindly doing a search-and-replace if they
choose to.

On the other hand, if I hadn't been reading this list this summer, it would
have been a "gentle annoyance" when my old snippet of code failed to run.
Or, as Yegge put it, one of the rare times that it required a minor tweak.
Maybe a stronger docstring could be used instead of deletion. Perhaps:

(defun interactive-p ()
  "Equivalent to (called-interactively-p 'interactive), for backward
compatibility.  See the Info node `(elisp)Distinguish Interactive'
for the limited situations in which this is appropriate, and
suggestions for more robust alternatives."
  (called-interactively-p 'interactive))

But, the design of Emacs makes this a very minor issue, especially compared
to other software I use.  (No matter how many times I set it, tex-live seems
find a new way to change my default paper size back to A4.)

---
Thinking more broadly, it seems to me that the issue is more than just
backward compatibility.  It's about having respect for the people who will be
using your software.  I think of Arethra Franklin singing R-E-S-P-E-C-T in her
rendition of the classic Otis Redding song.  More than any other software that
I know of, Emacs embodies the concept of "user-respect."  One thing that's
clear to me from reading this list recently is that *every* decision involves
a careful consideration of the effects it will have on the people who use
Emacs.

That a lack of user-respect is otherwise ubiquitous today is evidenced simply
by the number of times each day that you mutter "WTF, why did that just
happen?" as you browse the web, interact with software, etc, etc.
https://commadot.com/wtf-per-minute/

In commercial settings, this is likely due to the (perhaps misguided)
perception that decisions are in their financial interest, and the people who
use their products are willing to just "live with it."  In other circles, I
think that the ego and hubris of developers is, in large part, to blame.  If I
had to guess, I'd say that the respect embodied by Emacs is due, in no small
part, to Richard Stallman: his apparent humility and his absolutely unwavering
commitment to principles rather than self promotion.

Thanks,
-Jeff

Re: Delete variables obsolete since Emacs 23

[Eli Zaretskii]

> Date: Wed, 19 Aug 2020 08:31:55 +0000
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> >> The phrase "may be removed" seems a bit vague.  Would "will be removed" 
> >> or "will probably be removed" be more accurate?
> >
> > No, it won't.  Primarily because we don't really know whether we will 
> > remove it, let alone when.  It depends on too many factors that we 
> > cannot predict.
> 
> In that case, would a two-step process not be better?  First declaring the 
> function "obsolete", and when it is known that it will be removed declare 
> it "pending-removal" with a target major version.

How would it help?  It will definitely complicate our procedures, but
I see no gains.  Do you?

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> > My only request is to not remove `interactive-p'.
> > My request is to please leave it "obsolete".
> 
> The elisp manual says:
> 
>     You can mark a named function as "obsolete", meaning that it may be
>     removed at some point in the future.  This causes Emacs to warn that the
>     function is obsolete whenever it byte-compiles code containing that
>     function, and whenever it displays the documentation for that function.
>     In all other respects, an obsolete function behaves like any other
>     function.
> 
> The phrase "may be removed" seems a bit vague.  Would "will be removed" or
> "will probably be removed" be more accurate?

Not in my opinion.  I mean that shouldn't be what
we mean by "obsolete" or "deprecated".  Normally
it means just what the text says: there's no
guarantee that it won't be removed.

In general (e.g. outside Emacs), there's no
obligation to ever remove something that's been
deprecated (is obsolete).

In general, something that's deprecated is still
supported.  But it is not under active development,
and there may not be a lot of energy spent on
fixing its bugs.

> From where I sit, it seems plausible that someone might view interactive-p
> as just an obsolete way of writing (called-interactively-p 'interactive),
> which exists for backward compatibility, and would be available long term.
> It seems that the last part is a mis-interpretation of the intent of marking
> it as obsolete.

That's exactly the interpretation I'm looking
for.  I don't care that it's deprecated, as long
as it's still there.

> The fact that emacs maintains backward compatibility so well is one of the
> features that I appreciate the most.

Indeed.

https://medium.com/@steve.yegge/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc

> But, in this case, the fact that interactive-p
> has been obsolete for so long may give the
> impression that it would continue to be available.

I don't care about either possible impression.

I'm saying it makes sense to leave it alone.
There's no reason to remove it.  (And no reason
has yet been given.)  "It's obsolete" is not a
reason to remove it.

We're not removing
(called-interactively-p 'interactive).
Why not?

> If the underlying fact is that virtually
> all functions marked as obsolete will
> eventually be removed

Not a fact.  Not in my book, anyway.  And
certainly not in the world outside Emacs.

(Unless by "eventually" you include the
disappearance of the planet or sun. ;-))
 
> Also, it might be worthwhile for the elisp manual to mention one other
> difference that obsolete functions have.  When an interactive command is
> obsolete, it no longer appears as a possible completion when you press
> [tab] while entering a command-name after M-x.

That's not true either.  Perhaps you're
suggesting that such behavior should be
implemented?  (FWIW, my opinion is no.)

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think we are miscommunicating.  You seem to be talking about
  > > called-interactively-p.  I am talking about making your function take
  > > an optional argument.

  > What "your function" are you talking about?

The function which calls `interactive-p'.
It's in your code, right?  So I refer to it as yours.

You should change that function to have an additional optional
argument and use `interactive' to silently supply a non-nil value for
it when it gets called interactively.  Then use that argument
instead of calling `interactive-p'.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> You should change that function to have an additional optional
> argument and use `interactive' to silently supply a non-nil value for
> it when it gets called interactively.  Then use that argument
> instead of calling `interactive-p'.

I already addressed that "should".

Emacs "should" just leave `interactive-p' alone.
It's not hurting anyone.  It can't hurt any more
than (called-interactively 'interactive) hurts,
which is the exact same thing.

Re: Delete variables obsolete since Emacs 23

[Lars Ingebrigtsen]

Drew Adams <drew.adams@oracle.com> writes:

>> I'm attaching a patch to remove most things declared obsolete in 23.2
>> and 23.3.  (Nothing was declared obsolete in 23.4 AFAICT.)
>> 
>> There are still some items obsoleted in 23.x left to delete in Semantic.
>> I intend to do that in a separate patch.
>> 
>> Comments welcome.
>
> PLEASE do not remove `interactive-p'.

Emacs does have to move on and eventually remove stuff that we've
deprecated, and that's fine for most things.

But thinking about this a bit more, I agree with Drew here --
interactive-p is used too much in the wild for us to ever delete it.  It
should stay marked as obsolete, but with a comment saying "never delete"
or something, because removing it will break too much code out there.

For a perspective on deprecation (which mentions Emacs :-)), this is a
pretty entertaining read:

https://medium.com/@steve.yegge/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But thinking about this a bit more, I agree with Drew here --

Given the controversy here, I don't feel comfortable deleting
interactive-p.  So I left that part of my patch out and pushed the rest
as commit 326fdb9ec0.

The next step here is the obsoletions in semantic.

> interactive-p is used too much in the wild for us to ever delete it.  It
> should stay marked as obsolete, but with a comment saying "never delete"
> or something, because removing it will break too much code out there.

FWIW, I also think there's merit to the argument advanced by Drew and
Lars in the case of interactive-p.  (But my preference would be to say:
"Delete after year or version N".)

Best regards,
Stefan Kangas

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > interactive-p is used too much in the wild for us to ever delete it.  It
  > > should stay marked as obsolete, but with a comment saying "never delete"
  > > or something, because removing it will break too much code out there.

I agree we should never delete it, for that good reason.
But we made it obsolete for a good reason, too: any program which
uses will misbehave in some unusual cases.  The only way to get
fully correct behavior is to eliminate the calls to interactive-p,
and instead use an optional argument which gets a non-nil value
in an interactive call.

All the calls in the Emacs code have been replaced.
Do any remain in ELPA?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

>   > > interactive-p is used too much in the wild for us to ever delete it.
>   > > It should stay marked as obsolete, but with a comment saying "never
>   > > delete" or something, because removing it will break too much code
>   > > out there.
> 
> I agree we should never delete it, for that good reason.

Good.  That was the (only) point in question.

> But we made it obsolete for a good reason, too: any program which
> uses will misbehave in some unusual cases.  The only way to get
> fully correct behavior is to eliminate the calls to interactive-p,
> and instead use an optional argument which gets a non-nil value
> in an interactive call.
> 
> All the calls in the Emacs code have been replaced.

Uh, no, I don't think so.

(interactive-p) == (called-interactively-p 'interactive)

Grepping tells me there are 258 occurrences of the
latter in the Lisp code distributed with Emacs.

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Uh, no, I don't think so.

  > (interactive-p) == (called-interactively-p 'interactive)

I do not follow you -- your words are very terse.

Are you saying that people replaced calls to interactive-p with calls
to called-interactively-p, rather than converting to use an optional
argument?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Delete variables obsolete since Emacs 23

[Gregory Heytings via Emacs development discussions.]

>
>> Uh, no, I don't think so.
>>
>> (interactive-p) == (called-interactively-p 'interactive)
>
> I do not follow you -- your words are very terse.
>
> Are you saying that people replaced calls to interactive-p with calls to 
> called-interactively-p, rather than converting to use an optional 
> argument?
>

Yes.

And that's also what Stefan Kangas did in his patch for Emacs named 
0001-Remove-many-items-obsolete-since-Emacs-23.2-and-23.3.patch and sent 
on this list two weeks ago (on Fri, 14 Aug 2020 08:42:38 -0700).

Gregory

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

Gregory Heytings <ghe@sdf.org> writes:

>> Are you saying that people replaced calls to interactive-p with calls to
>> called-interactively-p, rather than converting to use an optional
>> argument?
>
> Yes.
>
> And that's also what Stefan Kangas did in his patch for Emacs named
> 0001-Remove-many-items-obsolete-since-Emacs-23.2-and-23.3.patch and sent
> on this list two weeks ago (on Fri, 14 Aug 2020 08:42:38 -0700).

My commit bc5da2c3fb does basically the following:

-    (help-setup-xref (list 'describe-symbols pattern) (interactive-p))
+    (help-setup-xref (list 'describe-symbols pattern)
+                 (called-interactively-p 'interactive))

Re: Delete variables obsolete since Emacs 23

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > My commit bc5da2c3fb does basically the following:

  > -    (help-setup-xref (list 'describe-symbols pattern) (interactive-p))
  > +    (help-setup-xref (list 'describe-symbols pattern)
  > +                 (called-interactively-p 'interactive))

That is not the recommend way to fix these calls.
called-interactively-p does not give the correct results in all
situations.  The right way is to make the optional argument.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

>   > My commit bc5da2c3fb does basically the following:
>
>   > -    (help-setup-xref (list 'describe-symbols pattern) (interactive-p))
>   > +    (help-setup-xref (list 'describe-symbols pattern)
>   > +                 (called-interactively-p 'interactive))
>
> That is not the recommend way to fix these calls.
> called-interactively-p does not give the correct results in all
> situations.  The right way is to make the optional argument.

This was simply to not use the old deprecated name, which AFAIU is being
considered for removal.

Perhaps we could also expand the section in the Manual that explains why
using an optional argument is better, and make the wording stronger.
Here's what I read in `(info "(elisp) Distinguish Interactive")':

    The above method with the additional argument is usually best,
    because it allows callers to say “treat this call as interactive”.  But
    you can also do the job by testing ‘called-interactively-p’.

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

>   > Uh, no, I don't think so.
>   > (interactive-p) == (called-interactively-p 'interactive)
> 
> I do not follow you -- your words are very terse.
> 
> Are you saying that people replaced calls to interactive-p with calls
> to called-interactively-p, rather than converting to use an optional
> argument?

Yes, apparently so.  You're correct that there are
now no occurrences of `interactive-p' in the source
code.  But there are hundreds of occurrences of 
what amounts to the same thing.

I've said it several times in this thread already,
which is why I didn't belabor it in my too-terse
reply: this is the _definition_ of `interactive-p'
(in `subr.el'):

(defun interactive-p ()
  "..."
  (declare (obsolete called-interactively-p "23.2"))
  (called-interactively-p 'interactive))

And the Emacs Lisp source code contains lots of
occurrences of `(called-interactively-p 'interactive)'.

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

> I've said it several times in this thread already,
> which is why I didn't belabor it in my too-terse
> reply: this is the _definition_ of `interactive-p'
> (in `subr.el'):
>
> (defun interactive-p ()
>   "..."
>   (declare (obsolete called-interactively-p "23.2"))
>   (called-interactively-p 'interactive))

For completeness, I have to warn that there is a bit more than meets the
eye, here: `called-interactively-p` is a big-ugly-hack and in order for
the above `interactive-p` to work, it has to know that `interactive-p`
is also one of those special not-really-functions.  More specifically,
this manifests itself in the presence of the symbol `interactive-p`
inside `called-interactively-p`.

After all, if taken literally, the above code would always return nil
since `interactive-p` can't be called interactively.

> And the Emacs Lisp source code contains lots of
> occurrences of `(called-interactively-p 'interactive)'.

Yes, and that still sucks, indeed.


        Stefan

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 217 bytes --]

Stefan Kangas <stefankangas@gmail.com> writes:

> The next step here is the obsoletions in semantic.

Please find attached a patch to remove obsolete items also from cedet.

Any comments?

Best regards,
Stefan Kangas

[-- Attachment #2: 0001-Remove-cedet-items-obsolete-since-23.2.patch --]
[-- Type: text/x-diff, Size: 45771 bytes --]

From a89738531bf9c37293d4d2fdfcf52fa660abd19c Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefankangas@gmail.com>
Date: Fri, 14 Aug 2020 13:40:29 +0200
Subject: [PATCH] Remove cedet items obsolete since 23.2

* lisp/cedet/semantic.el (semantic-toplevel-bovine-table)
(semantic-toplevel-bovine-cache)
(semantic-before-toplevel-bovination-hook)
(semantic-after-toplevel-bovinate-hook, semantic-init-hooks)
(semantic-init-mode-hooks, semantic-init-db-hooks)
(semantic-bovination-working-type, semantic-bovinate-toplevel)
(semantic-bovinate-region-until-error)
(semantic-bovinate-from-nonterminal-full):
* lisp/cedet/semantic/db-mode.el (semanticdb-mode-hooks):
* lisp/cedet/semantic/decorate/mode.el
(semantic-decorate-pending-decoration-hooks):
* lisp/cedet/semantic/edit.el
(semantic-edits-incremental-reparse-failed-hooks):
* lisp/cedet/semantic/fw.el (define-mode-overload-implementation):
* lisp/cedet/semantic/idle.el
(semantic-before-idle-scheduler-reparse-hooks)
(semantic-after-idle-scheduler-reparse-hooks):
(semantic-eldoc-current-symbol-info)
* lisp/cedet/semantic/imenu.el (semantic-imenu-expand-type-parts)
(semantic-imenu-bucketize-type-parts)
(semantic-imenu-expandable-token):
* lisp/cedet/semantic/java.el
(semantic-java-prototype-nonterminal):
* lisp/cedet/semantic/lex.el (semantic-flex-token-start)
(semantic-flex-token-end, semantic-flex-token-text)
(semantic-flex-make-keyword-table, semantic-flex-keyword-p)
(semantic-flex-keyword-put, semantic-flex-keyword-get)
(semantic-flex-map-keywords, semantic-flex-keywords)
(semantic-flex-buffer, semantic-flex-list, semantic-flex):
* lisp/cedet/semantic/tag-file.el (semantic-find-nonterminal)
(semantic-find-dependency):
* lisp/cedet/semantic/tag-ls.el (semantic-nonterminal-full-name)
(semantic-nonterminal-protection, semantic-nonterminal-abstract)
(semantic-nonterminal-leaf):
* lisp/cedet/semantic/tag.el (semantic-token-type-parent)
(semantic-tag-make-assoc-list, semantic-expand-nonterminal):
* lisp/cedet/semantic/util.el (semantic-file-token-stream)
(semantic-something-to-stream):
* lisp/cedet/semantic/wisent.el (wisent-lex-make-token-table):
Delete many items obsolete since Emacs 23.2.
* lisp/cedet/semantic.el (semantic--set-buffer-cache)
(semantic-fetch-tags): Don't run removed hooks
'semantic-after-toplevel-bovinate-hook' and
'semantic-before-toplevel-bovination-hook'.
* lisp/cedet/semantic/bovine/el.el: Remove reference to obsolete
variable 'define-mode-overload-implementation'.
* lisp/cedet/semantic/doc.el (semantic-doc-snarf-comment-for-tag):
Don't bind removed variable 'semantic-ignore-comments'.
* lisp/cedet/semantic/fw.el (semantic-overload-symbol-from-function)
(semantic-alias-obsolete, semantic-varalias-obsolete): Declare
obsolete in favor of standard Emacs 'define-obsolete-*-alias'.
* lisp/cedet/semantic/grammar.el (semantic-grammar-ASSOC): Don't
use obsolete names.
* lisp/cedet/semantic/tag-ls.el (semantic-tag-full-package)
(semantic-tag-full-name): Doc fixes.
* lisp/cedet/semantic/util.el (semantic-describe-buffer): Don't bind
removed variable 'semantic-after-toplevel-bovinate-hook'.
* lisp/cedet/semantic/lex.el (semantic-flex-tokens)
(semantic-flex-unterminated-syntax-end-function)
(semantic-flex-extensions, semantic-flex-syntax-modifications)
(semantic-ignore-comments, semantic-flex-enable-newlines)
(semantic-flex-enable-whitespace, semantic-flex-enable-bol)
(semantic-number-expression, semantic-flex-depth): Make unused
variables obsolete.
; * etc/NEWS: List removed items.
---
 etc/NEWS                             |  33 +++-
 lisp/cedet/semantic.el               |  79 +--------
 lisp/cedet/semantic/bovine/el.el     |   1 -
 lisp/cedet/semantic/db-mode.el       |   4 -
 lisp/cedet/semantic/decorate/mode.el |   3 -
 lisp/cedet/semantic/doc.el           |   3 +-
 lisp/cedet/semantic/edit.el          |   3 -
 lisp/cedet/semantic/fw.el            |  13 +-
 lisp/cedet/semantic/grammar.el       |   2 +-
 lisp/cedet/semantic/idle.el          |   9 --
 lisp/cedet/semantic/imenu.el         |   6 -
 lisp/cedet/semantic/java.el          |   3 -
 lisp/cedet/semantic/lex.el           | 231 ++-------------------------
 lisp/cedet/semantic/tag-file.el      |  13 --
 lisp/cedet/semantic/tag-ls.el        |  16 +-
 lisp/cedet/semantic/tag.el           |  20 ---
 lisp/cedet/semantic/util.el          |   7 -
 lisp/cedet/semantic/wisent.el        |   5 -
 18 files changed, 53 insertions(+), 398 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index e0ea8f53cc..13e145cec1 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1146,8 +1146,8 @@ ledit.el, lmenu.el, lucid.el and old-whitespace.el.
 'completion-base-size', 'completion-common-substring',
 'crm-minibuffer-complete', 'crm-minibuffer-complete-and-exit',
 'crm-minibuffer-completion-help', 'custom-mode', 'custom-mode-hook',
-'detect-coding-with-priority', 'dirtrack-debug',
-'dirtrack-debug-toggle', 'dynamic-completion-table',
+'define-mode-overload-implementation', 'detect-coding-with-priority',
+'dirtrack-debug', 'dirtrack-debug-toggle', 'dynamic-completion-table',
 'easy-menu-precalculate-equivalent-keybindings',
 'epa-display-verify-result', 'epg-passphrase-callback-function',
 'eshell-report-bug', 'eval-next-after-load', 'exchange-dot-and-mark',
@@ -1167,13 +1167,40 @@ ledit.el, lmenu.el, lucid.el and old-whitespace.el.
 'pre-abbrev-expand-hook', 'princ-list', 'print-help-return-message',
 'process-filter-multibyte-p', 'read-file-name-predicate',
 'remember-buffer', 'rmail-highlight-face', 'rmail-message-filter',
+'semantic-after-idle-scheduler-reparse-hooks',
+'semantic-after-toplevel-bovinate-hook',
+'semantic-before-idle-scheduler-reparse-hooks',
+'semantic-before-toplevel-bovination-hook',
+'semantic-bovinate-from-nonterminal-full',
+'semantic-bovinate-region-until-error', 'semantic-bovinate-toplevel',
+'semantic-bovination-working-type',
+'semantic-decorate-pending-decoration-hooks',
+'semantic-edits-incremental-reparse-failed-hooks',
+'semantic-eldoc-current-symbol-info', 'semantic-expand-nonterminal',
+'semantic-file-token-stream', 'semantic-find-dependency',
+'semantic-find-nonterminal', 'semantic-flex', 'semantic-flex-buffer',
+'semantic-flex-keyword-get', 'semantic-flex-keyword-p',
+'semantic-flex-keyword-put', 'semantic-flex-keywords',
+'semantic-flex-list', 'semantic-flex-make-keyword-table',
+'semantic-flex-map-keywords', 'semantic-flex-token-end',
+'semantic-flex-token-start', 'semantic-flex-token-text',
+'semantic-imenu-bucketize-type-parts',
+'semantic-imenu-expand-type-parts', 'semantic-imenu-expandable-token',
+'semantic-init-db-hooks)', 'semantic-init-hooks',
+'semantic-init-mode-hooks', 'semantic-java-prototype-nonterminal',
+'semantic-nonterminal-abstract', 'semantic-nonterminal-full-name',
+'semantic-nonterminal-leaf', 'semantic-nonterminal-protection',
+'semantic-something-to-stream', 'semantic-tag-make-assoc-list',
+'semantic-token-type-parent', 'semantic-toplevel-bovine-cache',
+'semantic-toplevel-bovine-table', 'semanticdb-mode-hooks',
 'set-coding-priority', 'set-process-filter-multibyte',
 'shadows-compare-text-p', 'shell-dirtrack-toggle', 't-mouse-mode',
 'term-dynamic-simple-complete', 'tooltip-hook', 'tpu-have-ispell',
 'url-generate-unique-filename', 'url-temporary-directory',
 'vc-arch-command', 'vc-default-working-revision' (variable),
 'vc-mtn-command', 'vc-revert-buffer', 'vc-workfile-version',
-'vcursor-toggle-vcursor-map', 'w32-focus-frame', 'w32-select-font'.
+'vcursor-toggle-vcursor-map', 'w32-focus-frame', 'w32-select-font',
+'wisent-lex-make-token-table'.
 
 \f
 * Lisp Changes in Emacs 28.1
diff --git a/lisp/cedet/semantic.el b/lisp/cedet/semantic.el
index 58a35d7d8a..71321e12da 100644
--- a/lisp/cedet/semantic.el
+++ b/lisp/cedet/semantic.el
@@ -82,8 +82,6 @@ semantic--parse-table
 This variable is for internal use only, and its content depends on the
 external parser used.")
 (make-variable-buffer-local 'semantic--parse-table)
-(semantic-varalias-obsolete 'semantic-toplevel-bovine-table
-			    'semantic--parse-table "23.2")
 
 (defvar semantic-symbol->name-assoc-list
   '((type     . "Types")
@@ -112,17 +110,6 @@ semantic-case-fold
   "Value for `case-fold-search' when parsing.")
 (make-variable-buffer-local 'semantic-case-fold)
 
-(defvar semantic-expand-nonterminal nil
-  "Function to call for each nonterminal production.
-Return a list of non-terminals derived from the first argument, or nil
-if it does not need to be expanded.
-Languages with compound definitions should use this function to expand
-from one compound symbol into several.  For example, in C the definition
-  int a, b;
-is easily parsed into one tag.  This function should take this
-compound tag and turn it into two tags, one for A, and the other for B.")
-(make-variable-buffer-local 'semantic-expand-nonterminal)
-
 (defvar semantic--buffer-cache nil
   "A cache of the fully parsed buffer.
 If no significant changes have been made (based on the state) then
@@ -134,8 +121,6 @@ semantic--buffer-cache
 cached values for some reason, chances are you can add a hook to
 `semantic-after-toplevel-cache-change-hook'.")
 (make-variable-buffer-local 'semantic--buffer-cache)
-(semantic-varalias-obsolete 'semantic-toplevel-bovine-cache
-			    'semantic--buffer-cache "23.2")
 
 (defvar semantic-unmatched-syntax-cache nil
   "A cached copy of unmatched syntax tokens.")
@@ -171,18 +156,6 @@ semantic--before-fetch-tags-hook
 `semantic-fetch-tags' by an application.
 If any hook returns a nil value, the cached value is returned
 immediately, even if it is empty.")
-(semantic-varalias-obsolete 'semantic-before-toplevel-bovination-hook
-			    'semantic--before-fetch-tags-hook "23.2")
-
-(defvar semantic-after-toplevel-bovinate-hook nil
-  "Hooks run after a toplevel parse.
-It is not run if the toplevel parse command is called, and buffer does
-not need to be fully reparsed.
-For language specific hooks, make sure you define this as a local hook.
-
-This hook should not be used any more.
-Use `semantic-after-toplevel-cache-change-hook' instead.")
-(make-obsolete-variable 'semantic-after-toplevel-bovinate-hook nil "23.2")
 
 (defvar semantic-after-toplevel-cache-change-hook nil
   "Hooks run after the buffer tag list has changed.
@@ -305,13 +278,6 @@ semantic-init-db-hook
 This guarantees that the DB will go before other modes that require
 a parse of the buffer.")
 
-(semantic-varalias-obsolete 'semantic-init-hooks
-			    'semantic-init-hook "23.2")
-(semantic-varalias-obsolete 'semantic-init-mode-hooks
-			    'semantic-init-mode-hook "23.2")
-(semantic-varalias-obsolete 'semantic-init-db-hooks
-			    'semantic-init-db-hook "23.2")
-
 (defsubst semantic-error-if-unparsed ()
   "Raise an error if current buffer was not parsed by Semantic."
   (unless semantic-new-buffer-fcn-was-run
@@ -516,8 +482,6 @@ semantic-clear-toplevel-cache
   (semantic-parse-tree-set-needs-rebuild)
   ;; Remove this hook which tracks if a buffer is up to date or not.
   (remove-hook 'after-change-functions 'semantic-change-function t)
-  ;; Old model.  Delete someday.
-  ;;(run-hooks 'semantic-after-toplevel-bovinate-hook)
 
   (run-hook-with-args 'semantic-after-toplevel-cache-change-hook
 		      semantic--buffer-cache)
@@ -540,17 +504,12 @@ semantic--set-buffer-cache
   (setq semantic--completion-cache nil)
   ;; Refresh the display of unmatched syntax tokens if enabled
   (run-hook-with-args 'semantic-unmatched-syntax-hook
-                      semantic-unmatched-syntax-cache)
-  ;; Old Semantic 1.3 hook API.  Maybe useful forever?
-  (run-hooks 'semantic-after-toplevel-bovinate-hook)
-  )
+                      semantic-unmatched-syntax-cache))
 
 (defvar semantic-working-type 'percent
   "The type of working message to use when parsing.
 'percent means we are doing a linear parse through the buffer.
 'dynamic means we are reparsing specific tags.")
-(semantic-varalias-obsolete 'semantic-bovination-working-type
-			    'semantic-working-type "23.2")
 
 (defvar semantic-minimum-working-buffer-size (* 1024 5)
   "The minimum size of a buffer before working messages are displayed.
@@ -585,8 +544,6 @@ semantic-fetch-tags
    ;; Is this a semantic enabled buffer?
    (semantic-active-p)
    ;; Application hooks say the buffer is safe for parsing
-   (run-hook-with-args-until-failure
-    'semantic-before-toplevel-bovination-hook)
    (run-hook-with-args-until-failure
     'semantic--before-fetch-tags-hook)
    ;; If the buffer was previously marked unparseable,
@@ -690,11 +647,6 @@ semantic-refresh-tags-safe
       ;; Return if we are lexically safe
       lexically-safe))))
 
-(defun semantic-bovinate-toplevel (&optional ignored)
-  "Backward compatibility function."
-  (semantic-fetch-tags))
-(make-obsolete 'semantic-bovinate-toplevel 'semantic-fetch-tags "23.2")
-
 ;; Another approach is to let Emacs call the parser on idle time, when
 ;; needed, use `semantic-fetch-available-tags' to only retrieve
 ;; available tags, and setup the `semantic-after-*-hook' hooks to
@@ -812,20 +764,6 @@ semantic-dump-parser-warnings
 ;; Please move away from these functions, and try using semantic 2.x
 ;; interfaces instead.
 ;;
-(defsubst semantic-bovinate-region-until-error
-  (start end nonterm &optional depth)
-  "NOTE: Use `semantic-parse-region' instead.
-
-Bovinate between START and END starting with NONTERM.
-Optional DEPTH specifies how many levels of parenthesis to enter.
-This command will parse until an error is encountered, and return
-the list of everything found until that moment.
-This is meant for finding variable definitions at the beginning of
-code blocks in methods.  If `bovine-inner-scope' can also support
-commands, use `semantic-bovinate-from-nonterminal-full'."
-  (semantic-parse-region start end nonterm depth t))
-(make-obsolete 'semantic-bovinate-region-until-error
-               'semantic-parse-region "23.2")
 
 (defsubst semantic-bovinate-from-nonterminal
   (start end nonterm &optional depth length)
@@ -840,21 +778,6 @@ semantic-bovinate-from-nonterminal
 		  (semantic-lex start end (or depth 1) length)
 		  nonterm))))
 
-(defsubst semantic-bovinate-from-nonterminal-full
-  (start end nonterm &optional depth)
-  "NOTE: Use `semantic-parse-region' instead.
-
-Bovinate from within a nonterminal lambda from START to END.
-Iterates until all the space between START and END is exhausted.
-Argument NONTERM is the nonterminal symbol to start with.
-If NONTERM is nil, use `bovine-block-toplevel'.
-Optional argument DEPTH is the depth of lists to dive into.
-When used in a `lambda' of a MATCH-LIST, there is no need to include
-a START and END part."
-  (semantic-parse-region start end nonterm (or depth 1)))
-(make-obsolete 'semantic-bovinate-from-nonterminal-full
-               'semantic-parse-region "23.2")
-
 ;;; User interface
 
 (defun semantic-force-refresh ()
diff --git a/lisp/cedet/semantic/bovine/el.el b/lisp/cedet/semantic/bovine/el.el
index 656c63b7ee..822ec176a3 100644
--- a/lisp/cedet/semantic/bovine/el.el
+++ b/lisp/cedet/semantic/bovine/el.el
@@ -420,7 +420,6 @@ semantic-elisp-use-read
 	 :parent (symbol-name (nth 2 form))
 	 :documentation (semantic-elisp-do-doc (nth 4 form))
 	 )))
-  define-mode-overload-implementation ;; obsoleted
   define-mode-local-override
   )
 
diff --git a/lisp/cedet/semantic/db-mode.el b/lisp/cedet/semantic/db-mode.el
index 0ab03ef49e..16a30b6cfb 100644
--- a/lisp/cedet/semantic/db-mode.el
+++ b/lisp/cedet/semantic/db-mode.el
@@ -69,10 +69,6 @@ global-semanticdb-minor-mode
     (dolist (elt semanticdb-hooks)
       (remove-hook (cadr elt) (car elt)))))
 
-(semantic-varalias-obsolete 'semanticdb-mode-hooks
-			    'global-semanticdb-minor-mode-hook "23.2")
-
-
 (defun semanticdb-toggle-global-mode ()
   "Toggle use of the Semantic Database feature.
 Update the environment of Semantic enabled buffers accordingly."
diff --git a/lisp/cedet/semantic/decorate/mode.el b/lisp/cedet/semantic/decorate/mode.el
index 8eb6a3bbd5..293692000d 100644
--- a/lisp/cedet/semantic/decorate/mode.el
+++ b/lisp/cedet/semantic/decorate/mode.el
@@ -204,9 +204,6 @@ semantic-decorate-add-decorations
 (defvar semantic-decorate-pending-decoration-hook nil
   "Normal hook run to perform pending decoration changes.")
 
-(semantic-varalias-obsolete 'semantic-decorate-pending-decoration-hooks
-			    'semantic-decorate-pending-decoration-hook "23.2")
-
 (defun semantic-decorate-add-pending-decoration (fcn &optional buffer)
   "Add a pending decoration change represented by FCN.
 Applies only to the current BUFFER.
diff --git a/lisp/cedet/semantic/doc.el b/lisp/cedet/semantic/doc.el
index 8b39e77578..896bc3bb42 100644
--- a/lisp/cedet/semantic/doc.el
+++ b/lisp/cedet/semantic/doc.el
@@ -93,8 +93,7 @@ semantic-doc-snarf-comment-for-tag
 Attempt to strip out comment syntactic sugar.
 Argument NOSNARF means don't modify the found text.
 If NOSNARF is `lex', then return the lex token."
-  (let* ((semantic-ignore-comments nil)
-	 (semantic-lex-analyzer #'semantic-comment-lexer))
+  (let* ((semantic-lex-analyzer #'semantic-comment-lexer))
     (if (memq nosnarf '(lex flex)) ;; keep `flex' for compatibility
 	(car (semantic-lex (point) (1+ (point))))
       (let ((ct (semantic-lex-token-text
diff --git a/lisp/cedet/semantic/edit.el b/lisp/cedet/semantic/edit.el
index a1225dfeee..e4319c7d1b 100644
--- a/lisp/cedet/semantic/edit.el
+++ b/lisp/cedet/semantic/edit.el
@@ -121,9 +121,6 @@ semantic-edits-incremental-reparse-failed-hook
   "Hook run after the incremental parser fails.
 When this happens, the buffer is marked as needing a full reparse.")
 
-(semantic-varalias-obsolete 'semantic-edits-incremental-reparse-failed-hooks
-			    'semantic-edits-incremental-reparse-failed-hook "23.2")
-
 (defcustom semantic-edits-verbose-flag nil
   "Non-nil means the incremental parser is verbose.
 If nil, errors are still displayed, but informative messages are not."
diff --git a/lisp/cedet/semantic/fw.el b/lisp/cedet/semantic/fw.el
index e347c99f19..c86cd3abf3 100644
--- a/lisp/cedet/semantic/fw.el
+++ b/lisp/cedet/semantic/fw.el
@@ -173,6 +173,7 @@ semantic-test-data-cache
 ;;
 (defun semantic-overload-symbol-from-function (name)
   "Return the symbol for overload used by NAME, the defined symbol."
+  (declare (obsolete define-obsolete-function-alias "28.1"))
   (let ((sym-name (symbol-name name)))
     (if (string-match "^semantic-" sym-name)
 	(intern (substring sym-name (match-end 0)))
@@ -182,6 +183,7 @@ semantic-alias-obsolete
   "Make OLDFNALIAS an alias for NEWFN.
 Mark OLDFNALIAS as obsolete, such that the byte compiler
 will throw a warning when it encounters this symbol."
+  (declare (obsolete define-obsolete-function-alias "28.1"))
   (defalias oldfnalias newfn)
   (make-obsolete oldfnalias newfn when)
   (when (and (mode-local--function-overload-p newfn)
@@ -196,13 +198,14 @@ semantic-alias-obsolete
      "%s: `%s' obsoletes overload `%s'"
      byte-compile-current-file
      newfn
-     (semantic-overload-symbol-from-function oldfnalias))
-    ))
+     (with-suppressed-warnings ((obsolete semantic-overload-symbol-from-function))
+       (semantic-overload-symbol-from-function oldfnalias)))))
 
 (defun semantic-varalias-obsolete (oldvaralias newvar when)
   "Make OLDVARALIAS an alias for variable NEWVAR.
 Mark OLDVARALIAS as obsolete, such that the byte compiler
 will throw a warning when it encounters this symbol."
+  (declare (obsolete define-obsolete-variable-alias "28.1"))
   (make-obsolete-variable oldvaralias newvar when)
   (condition-case nil
       (defvaralias oldvaralias newvar)
@@ -256,9 +259,6 @@ semantic-map-buffers
 
 (defalias 'semantic-map-mode-buffers 'mode-local-map-mode-buffers)
 
-(semantic-alias-obsolete 'define-mode-overload-implementation
-                         'define-mode-local-override "23.2")
-
 (defun semantic-install-function-overrides (overrides &optional transient)
   "Install the function OVERRIDES in the specified environment.
 OVERRIDES must be an alist ((OVERLOAD .  FUNCTION) ...) where OVERLOAD
@@ -396,13 +396,10 @@ semanticdb-without-unloaded-file-searches
 ;; 		 "define-lex-regex-type-analyzer"
 ;; 		 "define-lex-string-type-analyzer"
 ;; 		 "define-lex-block-type-analyzer"
-;; 		 ;;"define-mode-overload-implementation"
 ;; 		 ;;"define-semantic-child-mode"
 ;; 		 "define-semantic-idle-service"
 ;; 		 "define-semantic-decoration-style"
 ;; 		 "define-wisent-lexer"
-;; 		 "semantic-alias-obsolete"
-;; 		 "semantic-varalias-obsolete"
 ;; 		 "semantic-make-obsolete-overload"
 ;; 		 "defcustom-mode-local-semantic-dependency-system-include-path"
 ;; 		 ))
diff --git a/lisp/cedet/semantic/grammar.el b/lisp/cedet/semantic/grammar.el
index 6cd4832165..f71ac6c413 100644
--- a/lisp/cedet/semantic/grammar.el
+++ b/lisp/cedet/semantic/grammar.el
@@ -142,7 +142,7 @@ semantic-grammar-ASSOC
   "Return expansion of built-in ASSOC expression.
 ARGS are ASSOC's key value list."
   (let ((key t))
-    `(semantic-tag-make-assoc-list
+    `(semantic-tag-make-plist
       ,@(mapcar #'(lambda (i)
                     (prog1
                         (if key
diff --git a/lisp/cedet/semantic/idle.el b/lisp/cedet/semantic/idle.el
index 76218249c5..8301b19530 100644
--- a/lisp/cedet/semantic/idle.el
+++ b/lisp/cedet/semantic/idle.el
@@ -472,11 +472,6 @@ semantic-after-idle-scheduler-reparse-hook
 If any hook function throws an error, this variable is reset to nil.
 This hook is not protected from lexical errors.")
 
-(semantic-varalias-obsolete 'semantic-before-idle-scheduler-reparse-hooks
-			    'semantic-before-idle-scheduler-reparse-hook "23.2")
-(semantic-varalias-obsolete 'semantic-after-idle-scheduler-reparse-hooks
-			    'semantic-after-idle-scheduler-reparse-hook "23.2")
-
 (defun semantic-idle-scheduler-refresh-tags ()
   "Refreshes the current buffer's tags.
 This is called by `semantic-idle-scheduler-function' to update the
@@ -734,10 +729,6 @@ semantic-idle-summary-useful-context-p
 (define-overloadable-function semantic-idle-summary-current-symbol-info ()
   "Return a string message describing the current context.")
 
-(make-obsolete-overload 'semantic-eldoc-current-symbol-info
-                        'semantic-idle-summary-current-symbol-info
-                        "23.2")
-
 (defcustom semantic-idle-summary-mode-hook nil
   "Hook run at the end of `semantic-idle-summary'."
   :group 'semantic
diff --git a/lisp/cedet/semantic/imenu.el b/lisp/cedet/semantic/imenu.el
index cdf0a23fa0..25f7fdb842 100644
--- a/lisp/cedet/semantic/imenu.el
+++ b/lisp/cedet/semantic/imenu.el
@@ -88,8 +88,6 @@ semantic-imenu-expand-type-members
   :group 'semantic-imenu
   :type 'boolean)
 (make-variable-buffer-local 'semantic-imenu-expand-type-members)
-(semantic-varalias-obsolete 'semantic-imenu-expand-type-parts
-                            'semantic-imenu-expand-type-members "23.2")
 
 (defcustom semantic-imenu-bucketize-type-members t
   "Non-nil if members of a type should be grouped into buckets.
@@ -98,8 +96,6 @@ semantic-imenu-bucketize-type-members
   :group 'semantic-imenu
   :type 'boolean)
 (make-variable-buffer-local 'semantic-imenu-bucketize-type-members)
-(semantic-varalias-obsolete 'semantic-imenu-bucketize-type-parts
-                            'semantic-imenu-bucketize-type-members "23.2")
 
 (defcustom semantic-imenu-sort-bucket-function nil
   "Function to use when sorting tags in the buckets of functions.
@@ -145,8 +141,6 @@ semantic-imenu-expandable-tag-classes
 By default, a `type' has interesting children.  In Texinfo, however, a
 `section' has interesting children.")
 (make-variable-buffer-local 'semantic-imenu-expandable-tag-classes)
-(semantic-varalias-obsolete 'semantic-imenu-expandable-token
-                            'semantic-imenu-expandable-tag-classes "23.2")
 
 ;;; Code:
 (defun semantic-imenu-tag-overlay (tag)
diff --git a/lisp/cedet/semantic/java.el b/lisp/cedet/semantic/java.el
index 80d03dc629..2aa0ab0e3f 100644
--- a/lisp/cedet/semantic/java.el
+++ b/lisp/cedet/semantic/java.el
@@ -253,9 +253,6 @@ semantic-format-tag-prototype
                'semantic-format-tag-prototype-default)
              tag parent color)))
 
-(semantic-alias-obsolete 'semantic-java-prototype-nonterminal
-                         'semantic-format-tag-prototype-java-mode "23.2")
-
 ;; Include Tag Name
 ;;
 
diff --git a/lisp/cedet/semantic/lex.el b/lisp/cedet/semantic/lex.el
index 500a09d492..cb17f9fd93 100644
--- a/lisp/cedet/semantic/lex.el
+++ b/lisp/cedet/semantic/lex.el
@@ -1754,29 +1754,11 @@ semantic-lex-catch-errors
 ;;
 ;; NOTE: DELETE THIS SOMEDAY SOON
 
-(semantic-alias-obsolete 'semantic-flex-start 'semantic-lex-token-start "23.2")
-(semantic-alias-obsolete 'semantic-flex-end 'semantic-lex-token-end "23.2")
-(semantic-alias-obsolete 'semantic-flex-text 'semantic-lex-token-text "23.2")
-(semantic-alias-obsolete 'semantic-flex-make-keyword-table 'semantic-lex-make-keyword-table "23.2")
-(semantic-alias-obsolete 'semantic-flex-keyword-p 'semantic-lex-keyword-p "23.2")
-(semantic-alias-obsolete 'semantic-flex-keyword-put 'semantic-lex-keyword-put "23.2")
-(semantic-alias-obsolete 'semantic-flex-keyword-get 'semantic-lex-keyword-get "23.2")
-(semantic-alias-obsolete 'semantic-flex-map-keywords 'semantic-lex-map-keywords "23.2")
-(semantic-alias-obsolete 'semantic-flex-keywords 'semantic-lex-keywords "23.2")
-(semantic-alias-obsolete 'semantic-flex-buffer 'semantic-lex-buffer "23.2")
-(semantic-alias-obsolete 'semantic-flex-list 'semantic-lex-list "23.2")
-
-;; This simple scanner uses the syntax table to generate a stream of
-;; simple tokens of the form:
-;;
-;;  (SYMBOL START . END)
-;;
-;; Where symbol is the type of thing it is.  START and END mark that
-;; objects boundary.
-
 (defvar semantic-flex-tokens semantic-lex-tokens
   "An alist of semantic token types.
 See variable `semantic-lex-tokens'.")
+(make-obsolete-variable 'semantic-flex-tokens
+                        'semantic-lex-tokens "28.1")
 
 (defvar semantic-flex-unterminated-syntax-end-function
   (lambda (_syntax _syntax-start flex-end) flex-end)
@@ -1788,6 +1770,8 @@ semantic-flex-unterminated-syntax-end-function
 This function can be used for languages that can intelligently fix up
 broken syntax, or the exit lexical analysis via `throw' or `signal'
 when finding unterminated syntax.")
+(make-obsolete-variable 'semantic-flex-unterminated-syntax-end-function
+                        nil "28.1")
 
 (defvar semantic-flex-extensions nil
   "Buffer local extensions to the lexical analyzer.
@@ -1799,6 +1783,7 @@ semantic-flex-extensions
 TYPE can be any type of symbol, as long as it doesn't occur as a
 nonterminal in the language definition.")
 (make-variable-buffer-local 'semantic-flex-extensions)
+(make-obsolete-variable 'semantic-flex-extensions nil "28.1")
 
 (defvar semantic-flex-syntax-modifications nil
   "Changes to the syntax table for this buffer.
@@ -1809,237 +1794,47 @@ semantic-flex-syntax-modifications
 and CLASS is the string also passed to `modify-syntax-entry' to define
 what syntax class CHAR has.")
 (make-variable-buffer-local 'semantic-flex-syntax-modifications)
+(make-obsolete-variable 'semantic-flex-syntax-modifications nil "28.1")
 
 (defvar semantic-ignore-comments t
   "Default comment handling.
 The value t means to strip comments when flexing; nil means
 to keep comments as part of the token stream.")
 (make-variable-buffer-local 'semantic-ignore-comments)
+(make-obsolete-variable 'semantic-ignore-comments nil "28.1")
 
 (defvar semantic-flex-enable-newlines nil
   "When flexing, report newlines as syntactic elements.
 Useful for languages where the newline is a special case terminator.
 Only set this on a per mode basis, not globally.")
 (make-variable-buffer-local 'semantic-flex-enable-newlines)
+(make-obsolete-variable 'semantic-flex-enable-newlines nil "28.1")
 
 (defvar semantic-flex-enable-whitespace nil
   "When flexing, report whitespace as syntactic elements.
 Useful for languages where the syntax is whitespace dependent.
 Only set this on a per mode basis, not globally.")
 (make-variable-buffer-local 'semantic-flex-enable-whitespace)
+(make-obsolete-variable 'semantic-flex-enable-whitespace nil "28.1")
 
 (defvar semantic-flex-enable-bol nil
   "When flexing, report beginning of lines as syntactic elements.
 Useful for languages like python which are indentation sensitive.
 Only set this on a per mode basis, not globally.")
 (make-variable-buffer-local 'semantic-flex-enable-bol)
+(make-obsolete-variable 'semantic-flex-enable-bol nil "28.1")
 
 (defvar semantic-number-expression semantic-lex-number-expression
   "See variable `semantic-lex-number-expression'.")
 (make-variable-buffer-local 'semantic-number-expression)
+(make-obsolete-variable 'semantic-number-expression
+                        'semantic-lex-number-expression "28.1")
 
 (defvar semantic-flex-depth 0
   "Default flexing depth.
 This specifies how many lists to create tokens in.")
 (make-variable-buffer-local 'semantic-flex-depth)
-
-(defun semantic-flex (start end &optional depth length)
-  "Using the syntax table, do something roughly equivalent to flex.
-Semantically check between START and END.  Optional argument DEPTH
-indicates at what level to scan over entire lists.
-The return value is a token stream.  Each element is a list, such of
-the form (symbol start-expression .  end-expression) where SYMBOL
-denotes the token type.
-See `semantic-flex-tokens' variable for details on token types.
-END does not mark the end of the text scanned, only the end of the
-beginning of text scanned.  Thus, if a string extends past END, the
-end of the return token will be larger than END.  To truly restrict
-scanning, use `narrow-to-region'.
-The last argument, LENGTH specifies that `semantic-flex' should only
-return LENGTH tokens."
-  (declare (obsolete define-lex "23.2"))
-  (if (not semantic-flex-keywords-obarray)
-      (setq semantic-flex-keywords-obarray [ nil ]))
-  (let ((ts nil)
-        (pos (point))
-        (ep nil)
-        (curdepth 0)
-        (cs (if comment-start-skip
-                (concat "\\(\\s<\\|" comment-start-skip "\\)")
-              (concat "\\(\\s<\\)")))
-        (newsyntax (copy-syntax-table (syntax-table)))
-        (mods semantic-flex-syntax-modifications)
-        ;; Use the default depth if it is not specified.
-        (depth (or depth semantic-flex-depth)))
-    ;; Update the syntax table
-    (while mods
-      (modify-syntax-entry (car (car mods)) (car (cdr (car mods))) newsyntax)
-      (setq mods (cdr mods)))
-    (with-syntax-table newsyntax
-      (goto-char start)
-      (while (and (< (point) end) (or (not length) (<= (length ts) length)))
-        (cond
-         ;; catch beginning of lines when needed.
-         ;; Must be done before catching any other tokens!
-         ((and semantic-flex-enable-bol
-               (bolp)
-               ;; Just insert a (bol N . N) token in the token stream,
-               ;; without moving the point.  N is the point at the
-               ;; beginning of line.
-               (setq ts (cons (cons 'bol (cons (point) (point))) ts))
-               nil)) ;; CONTINUE
-         ;; special extensions, includes whitespace, nl, etc.
-         ((and semantic-flex-extensions
-               (let ((fe semantic-flex-extensions)
-                     (r nil))
-                 (while fe
-                   (if (looking-at (car (car fe)))
-                       (setq ts (cons (funcall (cdr (car fe))) ts)
-                             r t
-                             fe nil
-                             ep (point)))
-                   (setq fe (cdr fe)))
-                 (if (and r (not (car ts))) (setq ts (cdr ts)))
-                 r)))
-         ;; catch newlines when needed
-         ((looking-at "\\s-*\\(\n\\|\\s>\\)")
-          (if semantic-flex-enable-newlines
-              (setq ep (match-end 1)
-                    ts (cons (cons 'newline
-                                   (cons (match-beginning 1) ep))
-                             ts))))
-         ;; catch whitespace when needed
-         ((looking-at "\\s-+")
-          (if semantic-flex-enable-whitespace
-              ;; Language wants whitespaces, link them together.
-              (if (eq (car (car ts)) 'whitespace)
-                  (setcdr (cdr (car ts)) (match-end 0))
-                (setq ts (cons (cons 'whitespace
-                                     (cons (match-beginning 0)
-                                           (match-end 0)))
-                               ts)))))
-         ;; numbers
-         ((and semantic-number-expression
-               (looking-at semantic-number-expression))
-          (setq ts (cons (cons 'number
-                               (cons (match-beginning 0)
-                                     (match-end 0)))
-                         ts)))
-         ;; symbols
-         ((looking-at "\\(\\sw\\|\\s_\\)+")
-          (setq ts (cons (cons
-                          ;; Get info on if this is a keyword or not
-                          (or (semantic-lex-keyword-p (match-string 0))
-                              'symbol)
-                          (cons (match-beginning 0) (match-end 0)))
-                         ts)))
-         ;; Character quoting characters (ie, \n as newline)
-         ((looking-at "\\s\\+")
-          (setq ts (cons (cons 'charquote
-                               (cons (match-beginning 0) (match-end 0)))
-                         ts)))
-         ;; Open parens, or semantic-lists.
-         ((looking-at "\\s(")
-          (if (or (not depth) (< curdepth depth))
-              (progn
-                (setq curdepth (1+ curdepth))
-                (setq ts (cons (cons 'open-paren
-                                     (cons (match-beginning 0) (match-end 0)))
-                               ts)))
-            (setq ts (cons
-                      (cons 'semantic-list
-                            (cons (match-beginning 0)
-                                  (save-excursion
-                                    (condition-case nil
-                                        (forward-list 1)
-                                      ;; This case makes flex robust
-                                      ;; to broken lists.
-                                      (error
-                                       (goto-char
-                                        (funcall
-                                         semantic-flex-unterminated-syntax-end-function
-                                         'semantic-list
-                                         start end))))
-                                    (setq ep (point)))))
-                      ts))))
-         ;; Close parens
-         ((looking-at "\\s)")
-          (setq ts (cons (cons 'close-paren
-                               (cons (match-beginning 0) (match-end 0)))
-                         ts))
-          (setq curdepth (1- curdepth)))
-         ;; String initiators
-         ((looking-at "\\s\"")
-          ;; Zing to the end of this string.
-          (setq ts (cons (cons 'string
-                               (cons (match-beginning 0)
-                                     (save-excursion
-                                       (condition-case nil
-                                           (forward-sexp 1)
-                                         ;; This case makes flex
-                                         ;; robust to broken strings.
-                                         (error
-                                          (goto-char
-                                           (funcall
-                                            semantic-flex-unterminated-syntax-end-function
-                                            'string
-                                            start end))))
-                                       (setq ep (point)))))
-                         ts)))
-         ;; comments
-         ((looking-at cs)
-          (if (and semantic-ignore-comments
-                   (not semantic-flex-enable-whitespace))
-              ;; If the language doesn't deal with comments nor
-              ;; whitespaces, ignore them here.
-              (let ((comment-start-point (point)))
-                (forward-comment 1)
-                (if (eq (point) comment-start-point)
-                    ;; In this case our start-skip string failed
-                    ;; to work properly.  Lets try and move over
-                    ;; whatever white space we matched to begin
-                    ;; with.
-                    (skip-syntax-forward "-.'" (point-at-eol))
-                  ;;(forward-comment 1)
-                  ;; Generate newline token if enabled
-                  (if (and semantic-flex-enable-newlines
-                           (bolp))
-                      (backward-char 1)))
-                (if (eq (point) comment-start-point)
-                    (error "Strange comment syntax prevents lexical analysis"))
-                (setq ep (point)))
-            (let ((tk (if semantic-ignore-comments 'whitespace 'comment)))
-              (save-excursion
-                (forward-comment 1)
-                ;; Generate newline token if enabled
-                (if (and semantic-flex-enable-newlines
-                         (bolp))
-                    (backward-char 1))
-                (setq ep (point)))
-              ;; Language wants comments or want them as whitespaces,
-              ;; link them together.
-              (if (eq (car (car ts)) tk)
-                  (setcdr (cdr (car ts)) ep)
-                (setq ts (cons (cons tk (cons (match-beginning 0) ep))
-                               ts))))))
-         ;; punctuation
-         ((looking-at "\\(\\s.\\|\\s$\\|\\s'\\)")
-          (setq ts (cons (cons 'punctuation
-                               (cons (match-beginning 0) (match-end 0)))
-                         ts)))
-         ;; unknown token
-         (t
-          (error "What is that?")))
-        (goto-char (or ep (match-end 0)))
-        (setq ep nil)))
-    ;; maybe catch the last beginning of line when needed
-    (and semantic-flex-enable-bol
-         (= (point) end)
-         (bolp)
-         (setq ts (cons (cons 'bol (cons (point) (point))) ts)))
-    (goto-char pos)
-    ;;(message "Flexing muscles...done")
-    (nreverse ts)))
+(make-obsolete-variable 'semantic-flex-depth nil "28.1")
 
 (provide 'semantic/lex)
 
diff --git a/lisp/cedet/semantic/tag-file.el b/lisp/cedet/semantic/tag-file.el
index 50d43fe934..23f4b29cbd 100644
--- a/lisp/cedet/semantic/tag-file.el
+++ b/lisp/cedet/semantic/tag-file.el
@@ -101,9 +101,6 @@ semantic-go-to-tag
    )
   )
 
-(make-obsolete-overload 'semantic-find-nonterminal
-                        'semantic-go-to-tag "23.2")
-
 ;;; Dependencies
 ;;
 ;; A tag which is of type 'include specifies a dependency.
@@ -175,9 +172,6 @@ semantic-dependency-tag-file
 	nil)
       )))
 
-(make-obsolete-overload 'semantic-find-dependency
-                        'semantic-dependency-tag-file "23.2")
-
 ;;; PROTOTYPE FILE
 ;;
 ;; In C, a function in the .c file often has a representation in a
@@ -199,13 +193,6 @@ semantic-prototype-file
        (if (re-search-forward "::Header:: \\([a-zA-Z0-9.]+\\)" nil t)
            (match-string 1))))))
 
-(semantic-alias-obsolete 'semantic-find-nonterminal
-                         'semantic-go-to-tag "23.2")
-
-(semantic-alias-obsolete 'semantic-find-dependency
-                         'semantic-dependency-tag-file "23.2")
-
-
 (provide 'semantic/tag-file)
 
 ;; Local variables:
diff --git a/lisp/cedet/semantic/tag-ls.el b/lisp/cedet/semantic/tag-ls.el
index 16179a53cd..3ee11df7d8 100644
--- a/lisp/cedet/semantic/tag-ls.el
+++ b/lisp/cedet/semantic/tag-ls.el
@@ -190,7 +190,7 @@ semantic-tag-similar-p-default
 ;; will contain the info needed to determine the full name.
 (define-overloadable-function semantic-tag-full-package (tag &optional stream-or-buffer)
   "Return the fully qualified package name of TAG in a package hierarchy.
-STREAM-OR-BUFFER can be anything convertible by `semantic-something-to-stream',
+STREAM-OR-BUFFER can be anything convertible by `semantic-something-to-tag-table',
 but must be a toplevel semantic tag stream that contains TAG.
 A Package Hierarchy is defined in UML by the way classes and methods
 are organized on disk.  Some languages use this concept such that a
@@ -213,7 +213,7 @@ semantic-tag-full-package-default
 
 (define-overloadable-function semantic-tag-full-name (tag &optional stream-or-buffer)
   "Return the fully qualified name of TAG in the package hierarchy.
-STREAM-OR-BUFFER can be anything convertible by `semantic-something-to-stream',
+STREAM-OR-BUFFER can be anything convertible by `semantic-something-to-tag-table',
 but must be a toplevel semantic tag stream that contains TAG.
 A Package Hierarchy is defined in UML by the way classes and methods
 are organized on disk.  Some languages use this concept such that a
@@ -233,9 +233,6 @@ semantic-tag-full-name
                  (or stream-or-buffer tag))))
     (:override-with-args (tag stream))))
 
-(make-obsolete-overload 'semantic-nonterminal-full-name
-                        'semantic-tag-full-name "23.2")
-
 (defun semantic-tag-full-name-default (tag stream)
   "Default method for `semantic-tag-full-name'.
 Return the name of TAG found in the toplevel STREAM."
@@ -287,9 +284,6 @@ semantic-tag-protection
        (setq parent (semantic-tag-calculate-parent tag)))
   (:override))
 
-(make-obsolete-overload 'semantic-nonterminal-protection
-                        'semantic-tag-protection "23.2")
-
 (defun semantic-tag-protection-default (tag &optional parent)
   "Return the protection of TAG as a child of PARENT default action.
 See `semantic-tag-protection'."
@@ -377,9 +371,6 @@ semantic-tag-abstract-p
 The default behavior (if not overridden with `tag-abstract-p'
 is to return true if `abstract' is in the type modifiers.")
 
-(make-obsolete-overload 'semantic-nonterminal-abstract
-                        'semantic-tag-abstract-p "23.2")
-
 (defun semantic-tag-abstract-p-default (tag &optional parent)
   "Return non-nil if TAG is abstract as a child of PARENT default action.
 See `semantic-tag-abstract-p'."
@@ -400,9 +391,6 @@ semantic-tag-leaf-p
 The default behavior (if not overridden with `tag-leaf-p'
 is to return true if `leaf' is in the type modifiers.")
 
-(make-obsolete-overload 'semantic-nonterminal-leaf
-                        'semantic-tag-leaf-p "23.2")
-
 (defun semantic-tag-leaf-p-default (tag &optional parent)
   "Return non-nil if TAG is leaf as a child of PARENT default action.
 See `semantic-tag-leaf-p'."
diff --git a/lisp/cedet/semantic/tag.el b/lisp/cedet/semantic/tag.el
index ca5c068d34..e677264c5a 100644
--- a/lisp/cedet/semantic/tag.el
+++ b/lisp/cedet/semantic/tag.el
@@ -1328,26 +1328,6 @@ semantic-token-version
 (defconst semantic-token-incompatible-version
   semantic-tag-incompatible-version)
 
-(defsubst semantic-token-type-parent (tag)
-  "Return the parent of the type that TAG describes.
-The return value is a list.  A value of nil means no parents.
-The `car' of the list is either the parent class, or a list
-of parent classes.  The `cdr' of the list is the list of
-interfaces, or abstract classes which are parents of TAG."
-  (cons (semantic-tag-get-attribute tag :superclasses)
-        (semantic-tag-type-interfaces tag)))
-
-(make-obsolete 'semantic-token-type-parent
-	       "\
-use `semantic-tag-type-superclass' \
-and `semantic-tag-type-interfaces' instead" "23.2")
-
-(semantic-alias-obsolete 'semantic-tag-make-assoc-list
-                         'semantic-tag-make-plist "23.2")
-
-(semantic-varalias-obsolete 'semantic-expand-nonterminal
-                            'semantic-tag-expand-function "23.2")
-
 (provide 'semantic/tag)
 
 ;; Local variables:
diff --git a/lisp/cedet/semantic/util.el b/lisp/cedet/semantic/util.el
index c64d56b2e2..7df7dfcb75 100644
--- a/lisp/cedet/semantic/util.el
+++ b/lisp/cedet/semantic/util.el
@@ -79,9 +79,6 @@ semantic-file-tag-table
 	(with-current-buffer (find-file-noselect file)
 	  (semantic-fetch-tags))))))
 
-(semantic-alias-obsolete 'semantic-file-token-stream
-			 'semantic-file-tag-table "23.2")
-
 (declare-function semanticdb-abstract-table-child-p "semantic/db" (obj) t)
 (declare-function semanticdb-refresh-table "semantic/db")
 (declare-function semanticdb-get-tags "semantic/db" (arg &rest args) t)
@@ -137,9 +134,6 @@ semantic-something-to-tag-table
    ;; don't know what it is
    (t nil)))
 
-(semantic-alias-obsolete 'semantic-something-to-stream
-			 'semantic-something-to-tag-table "23.2")
-
 ;;; Completion APIs
 ;;
 ;; These functions provide minibuffer reading/completion for lists of
@@ -307,7 +301,6 @@ semantic-describe-buffer
 		      semantic-init-db-hook
 		      semantic-unmatched-syntax-hook
 		      semantic--before-fetch-tags-hook
-		      semantic-after-toplevel-bovinate-hook
 		      semantic-after-toplevel-cache-change-hook
 		      semantic-before-toplevel-cache-flush-hook
 		      semantic-dump-parse
diff --git a/lisp/cedet/semantic/wisent.el b/lisp/cedet/semantic/wisent.el
index 527a35c9ae..15d1313dfa 100644
--- a/lisp/cedet/semantic/wisent.el
+++ b/lisp/cedet/semantic/wisent.el
@@ -43,11 +43,6 @@ wisent-lex-lookahead
   "Extra lookahead token.
 When non-nil it is directly returned by `wisent-lex-function'.")
 
-;; Maintain this alias for compatibility until all WY grammars have
-;; been translated again to Elisp code.
-(semantic-alias-obsolete 'wisent-lex-make-token-table
-                         'semantic-lex-make-type-table "23.2")
-
 (defmacro wisent-lex-eoi ()
   "Return an End-Of-Input lexical token.
 The EOI token is like this: ($EOI \"\" POINT-MAX . POINT-MAX)."
-- 
2.28.0

Re: Delete variables obsolete since Emacs 23

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> The next step here is the obsoletions in semantic.
>
> Please find attached a patch to remove obsolete items also from cedet.
>
> Any comments?

I just skimmed it, but it looks good to me.  Perhaps this comment should
also be deleted?

>  ;; NOTE: DELETE THIS SOMEDAY SOON
>  
> -(semantic-alias-obsolete 'semantic-flex-start 'semantic-lex-token-start "23.2")

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Delete variables obsolete since Emacs 23

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Please find attached a patch to remove obsolete items also from cedet.
>
> I just skimmed it, but it looks good to me.  Perhaps this comment should
> also be deleted?
>
>>  ;; NOTE: DELETE THIS SOMEDAY SOON
>>
>> -(semantic-alias-obsolete 'semantic-flex-start 'semantic-lex-token-start "23.2")

Thanks for the review.

I removed the above line and pushed to master as commit 91a221b279.

Re: Deleting old `:version` from defcustoms (was: master b76cdd0: Delete libraries obsolete since 23.1 and 23.2)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I did a quick `grep` to see the functions/vars that were obsoleted in
> the same time frame, and I saw instead a deluge of
>
>     :version "23.1"
>
> If we consider those thingies old enough to remove their obsolete
> functions, shouldn't we also remove the corresponding `:version`
> thingies from `defcustom`s?
>
> I see:
> - 1 defcustom with a :version of 19.29
> - >200 defcustoms with a :version of 20.x
> - a bit less than 200 defcustoms with a :version of 21.x
> - >400 defcustoms with a :version of 22.x
> - >200 defcustoms with a :version of 23.x

I realize I don't fully understand the purpose of the :version
declarations.  If it's mostly to let users know what changed, they can
probably be removed for these very old versions.

But are they also intended to provide a historical record?  If so, maybe
they should be kept indefinitely.

Best regards,
Stefan Kangas

Re: Deleting old `:version` from defcustoms (was: master b76cdd0: Delete libraries obsolete since 23.1 and 23.2)

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 16 May 2020 20:18:54 -0700
> 
> I realize I don't fully understand the purpose of the :version
> declarations.  If it's mostly to let users know what changed, they can
> probably be removed for these very old versions.
> 
> But are they also intended to provide a historical record?  If so, maybe
> they should be kept indefinitely.

I think it's good to have a facility that can tell us in which Emacs
version was some option introduced.

Re: Deleting old `:version` from defcustoms

[Stefan Monnier]

> I think it's good to have a facility that can tell us in which Emacs
> version was some option introduced.

The :version doesn't quite tell us that: it tells us when it was
last changed.


        Stefan

Re: bug#41438: [PATCH] Allow windmove keys to be bound without prefix or modifiers

[Philip K.]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

Juri Linkov <juri@linkov.net> writes:

>> Another question I'd like to ask before trying it out: Would there be
>> any interest in adding user options for the "default" modifiers that
>> windmove should use? If yes, one could add a :set function that
>> automatically calls the apropriate bindings function, when it's value
>> is non-nil. I have a very customize-centric configuration, where something
>> this would fit in very well.
>
> I think that adding defcustoms (including a new const `none')
> would be the most natural way to customize windmove modifiers
> (provided that the existing functions should remain).

I made a first concept, but it doesn't work unless windmove is explicity
required (see below). As far as I know, autoloading variables isn't good
style, so is there any other way to invoke the :set property of a user
option?

-- 
	Philip K.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Implement-user-options-for-windmove-modifiers-and-pr.patch --]
[-- Type: text/x-diff, Size: 8273 bytes --]

From b35173936e681e170172eb3d4548a2f91f70dad3 Mon Sep 17 00:00:00 2001
From: Philip K <philip@warpmail.net>
Date: Fri, 22 May 2020 20:21:59 +0200
Subject: [PATCH] Implement user options for windmove modifiers and prefixes

---
 lisp/windmove.el | 103 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 95 insertions(+), 8 deletions(-)

diff --git a/lisp/windmove.el b/lisp/windmove.el
index 3d7f86b9af..95f74bfb81 100644
--- a/lisp/windmove.el
+++ b/lisp/windmove.el
@@ -162,6 +162,19 @@ windmove-window-distance-delta
 (make-obsolete-variable 'windmove-window-distance-delta
                         "no longer used." "27.1")
 
+(defconst windmove-modifier-type
+  '(choice (set :tag "Modifier Symbols"
+                :greedy t
+                ;; See `(elisp) Keyboard Events'
+                (const :tag "Meta" meta)
+                (const :tag "Control" control)
+                (const :tag "Shift" shift)
+                (const :tag "Hyper" hyper)
+                (const :tag "Super" super)
+                (const :tag "Alt" alt))
+           (const :tag "No modifier" none))
+  "Customisation type for windmove modifiers")
+
 \f
 ;; Note:
 ;;
@@ -419,6 +432,7 @@ windmove-down
   (interactive "P")
   (windmove-do-window-select 'down (and arg (prefix-numeric-value arg))))
 
+(defvar windmove-modifiers)
 
 ;;; set up keybindings
 ;; Idea for this function is from iswitchb.el, by Stephen Eglen
@@ -433,9 +447,10 @@ windmove-default-keybindings
 where MODIFIERS is either a list of modifiers or a single modifier.
 If MODIFIERS is `none', the keybindings will be directly bound to
 the arrow keys.
-Default value of MODIFIERS is `shift'."
+Default value of MODIFIERS is stored in `windmove-modifiers'."
   (interactive)
-  (unless modifiers (setq modifiers 'shift))
+  (unless modifiers
+    (setq modifiers windmove-modifiers))
   (when (eq modifiers 'none) (setq modifiers nil))
   (unless (listp modifiers) (setq modifiers (list modifiers)))
   (global-set-key (vector (append modifiers '(left)))  'windmove-left)
@@ -443,6 +458,25 @@ windmove-default-keybindings
   (global-set-key (vector (append modifiers '(up)))    'windmove-up)
   (global-set-key (vector (append modifiers '(down)))  'windmove-down))
 
+;; has to be declared AFTER windmove-default-keybindings, or else
+;; windmove is recursivly loaded
+;;;###autoload
+(defcustom windmove-modifiers '(shift super)
+  "Modifiers for `windmove-default-keybindings'.
+Can either be a symbol or list of modifier symbols,
+i.e. `meta',`control', `shift', `hyper', `super', or `alt'
+representing modifier keys to use with the arrow keys.
+
+If the value is just `none', the arrow keys will be directly
+bound to the windmove functions."
+  :type windmove-modifier-type
+  :require 'windmove
+  :initialize #'custom-initialize-changed
+  :set (lambda (sym val)
+         (when val
+           (windmove-swap-states-default-keybindings val))
+         (set-default sym val)))
+
 \f
 ;;; Directional window display and selection
 
@@ -565,6 +599,8 @@ windmove-display-new-tab
   (interactive "P")
   (windmove-display-in-direction 'new-tab arg))
 
+(defvar windmove-display-modifiers)
+
 ;;;###autoload
 (defun windmove-display-default-keybindings (&optional modifiers)
   "Set up keybindings for directional buffer display.
@@ -573,7 +609,7 @@ windmove-display-default-keybindings
 where MODIFIERS is either a list of modifiers or a single modifier.
 If MODIFIERS is `none', the keybindings will be directly bound to
 the arrow keys.
-Default value of MODIFIERS is `shift-meta'."
+Default value of MODIFIERS is stored in `windmove-display-modifiers'."
   (interactive)
   (unless modifiers (setq modifiers '(shift meta)))
   (when (eq modifiers 'none) (setq modifiers nil))
@@ -586,6 +622,17 @@ windmove-display-default-keybindings
   (global-set-key (vector (append modifiers '(?f)))    'windmove-display-new-frame)
   (global-set-key (vector (append modifiers '(?t)))    'windmove-display-new-tab))
 
+(defcustom windmove-display-modifiers '(shift meta)
+  "Modifiers for `windmove-display-default-keybindings'.
+Analogous to `windmove-modifiers'."
+  :type windmove-modifier-type
+  :require 'windmove
+  :initialize #'custom-initialize-changed
+  :set (lambda (sym val)
+         (when val
+           (windmove-display-default-keybindings val))
+         (set-default sym val)))
+
 \f
 ;;; Directional window deletion
 
@@ -640,6 +687,9 @@ windmove-delete-down
   (interactive "P")
   (windmove-delete-in-direction 'down arg))
 
+(defvar windmove-delete-prefix)
+(defvar windmove-delete-modifiers)
+
 ;;;###autoload
 (defun windmove-delete-default-keybindings (&optional prefix modifiers)
   "Set up keybindings for directional window deletion.
@@ -649,12 +699,13 @@ windmove-delete-default-keybindings
 a single modifier.
 If PREFIX is `none', no prefix is used. If MODIFIERS is `none', the keybindings
 are directly bound to the arrow keys.
-Default value of PREFIX is `C-x' and MODIFIERS is `shift'."
+The default values for PREFIX and MODIFIERS are stored in `windmove-delete-prefix'
+and `windmove-delete-modifiers' respectively."
   (interactive)
-  (unless prefix (setq prefix '(?\C-x)))
+  (unless prefix (setq prefix (list windmove-delete-prefix)))
   (when (eq prefix 'none) (setq prefix nil))
   (unless (listp prefix) (setq prefix (list prefix)))
-  (unless modifiers (setq modifiers '(shift)))
+  (unless modifiers (setq modifiers windmove-delete-modifiers))
   (when (eq modifiers 'none) (setq modifiers nil))
   (unless (listp modifiers) (setq modifiers (list modifiers)))
   (global-set-key (vector prefix (append modifiers '(left)))  'windmove-delete-left)
@@ -662,6 +713,28 @@ windmove-delete-default-keybindings
   (global-set-key (vector prefix (append modifiers '(up)))    'windmove-delete-up)
   (global-set-key (vector prefix (append modifiers '(down)))  'windmove-delete-down))
 
+(defcustom windmove-delete-prefix (kbd "C-x")
+  "Prefix for `windmove-delete-default-keybindings'."
+  :type 'key-sequence
+  :require 'windmove
+  :initialize #'custom-initialize-changed
+  :set (lambda (sym val)
+         (windmove-delete-default-keybindings
+          val windmove-delete-modifiers)
+         (set-default sym val)))
+
+(defcustom windmove-delete-modifiers '(shift)
+  "Modifiers for `windmove-delete-default-keybindings'.
+See `windmove-modifiers' for more details"
+  :type windmove-modifier-type
+  :require 'windmove
+  :initialize #'custom-initialize-changed
+  :set (lambda (sym val)
+         (when val
+           (windmove-delete-default-keybindings
+            windmove-delete-prefix val))
+         (set-default sym val)))
+
 \f
 ;;; Directional window swap states
 
@@ -700,6 +773,8 @@ windmove-swap-states-right
   (interactive)
   (windmove-swap-states-in-direction 'right))
 
+(defvar windmove-swap-states-modifiers)
+
 ;;;###autoload
 (defun windmove-swap-states-default-keybindings (&optional modifiers)
   "Set up keybindings for directional window swap states.
@@ -709,9 +784,10 @@ windmove-swap-states-default-keybindings
 or a single modifier.
 If MODIFIERS is `none', the keybindings will be directly bound to the
 arrow keys.
-Default value of MODIFIERS is `shift-super'."
+Default value of MODIFIERS is stored in `windmove-swap-states-modifiers'."
   (interactive)
-  (unless modifiers (setq modifiers '(shift super)))
+  (unless modifiers
+    (setq modifiers windmove-swap-states-modifiers))
   (when (eq modifiers 'none) (setq modifiers nil))
   (unless (listp modifiers) (setq modifiers (list modifiers)))
   (global-set-key (vector (append modifiers '(left)))  'windmove-swap-states-left)
@@ -719,6 +795,17 @@ windmove-swap-states-default-keybindings
   (global-set-key (vector (append modifiers '(up)))    'windmove-swap-states-up)
   (global-set-key (vector (append modifiers '(down)))  'windmove-swap-states-down))
 
+(defcustom windmove-swap-states-modifiers
+  '(shift super)
+  "Modifiers for `windmove-swap-states-default-keybindings'.
+Analogous to `windmove-modifiers'."
+  :type windmove-modifier-type
+  :require 'windmove
+  :initialize #'custom-initialize-changed
+  :set (lambda (sym val)
+         (windmove-swap-states-default-keybindings val)
+         (set-default sym val)))
+
 \f
 (provide 'windmove)
 
-- 
2.20.1

bug#41438: [PATCH] Allow windmove keys to be bound without prefix or modifiers

[Drew Adams]

Please don't send the same mail to both the bug list and emacs-devel.  Thx.

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> I think the gentle annoyance we have now strikes the right balance
> between not letting people forget the fact of obsolescence and not
> annoying them too much.  We are talking to adults, not to children, so
> we can rely on them doing TRT.

+1.  It's obsolete, but we leave it in the product,
so as to not annoy those who don't, or can't easily,
work around its absence.

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

>> I think the gentle annoyance we have now strikes the right balance
>> between not letting people forget the fact of obsolescence and not
>> annoying them too much.  We are talking to adults, not to children, so
>> we can rely on them doing TRT.
> +1.  It's obsolete, but we leave it in the product,

That is not what he said.

> so as to not annoy those who don't, or can't easily,
> work around its absence.

If there's no intention to remove it in the future, then we don't
declare it obsolete.


        Stefan

RE: Delete variables obsolete since Emacs 23

[Drew Adams]

> >> I think the gentle annoyance we have now strikes the right balance
> >> between not letting people forget the fact of obsolescence and not
> >> annoying them too much.  We are talking to adults, not to children, so
> >> we can rely on them doing TRT.
> >
> > +1.  It's obsolete, but we leave it in the product,
> 
> That is not what he said.

Eli can weigh in on whether it is or isn't.

Isn't that exactly what "we have now": declaration
of obsolescence, without removal?

That's what I understood by "the gentle annoyance
we have now".

And I agree that that "strikes the right balance"
by (1) reminding about obsolescence and (2) not
annoying too much.

> > so as to not annoy those who don't, or can't
> > easily, work around its absence.
> 
> If there's no intention to remove it in the future,
> then we don't declare it obsolete.

Is that written on the foundation stone of GNU
Emacs somewhere?

That's not how deprecation/obsolescence is viewed
in general, e.g. outside Emacs.

Something is deprecated or declared obsolete
because we no longer want to commit to its active
development, for WHATEVER reason (and the reasons
can be several).

Often, that's because some other, better thing
replaces it, but that's not a requirement.

Not wanting to commit to active development (e.g.
improvement, new features) is not the same thing
as an intention to remove.  I think you have a
very (unnecessarily) strong view of deprecation.

One of the main ideas behind deprecation is to
NOT tie your hands wrt future decisions.  All
that's decided is, so far, to not develop the
thing further.

The message to users is (1) the thing is still
supported (it's not removed), but only AS IS, so
(2) don't expect further development.

Removing something is what's normally called
"desupport": XYZ is no longer supported.  Trying
to use it raises an error, or is ignored (no-op),
etc.

You _cannot_ use something that's _desupported_.
You're advised/warned that you might not want to
use something that's deprecated.

Even for commercial software, there are quite
often features that are deprecated with NO
intention to _ever_ remove them.  Why?  Because
of an existing customer base, with legacy code.

Re: Delete variables obsolete since Emacs 23

[Gregory Heytings via Emacs development discussions.]

>
>> so as to not annoy those who don't, or can't easily, work around its 
>> absence.
>
> If there's no intention to remove it in the future, then we don't 
> declare it obsolete.
>

I think the discussion is too broad.  It depends on what is declared 
obsolete.

If it's a whole package, or a complex function, declaring that it is 
obsolete means that it won't be developed or supported in the future, and 
that because of this it might become non-functional at some point because 
the conditions it assumes will not hold anymore.  At that moment there is 
a good reason to remove it: it does not work anymore.

If on the contrary it's a function as trivial as

(defun interactive-p () (called-interactively-p 'interactive))

then declaring that it is obsolete means that new code should preferably 
use the new style.  But there is no reason to remove it.

Gregory

Re: Delete variables obsolete since Emacs 23

[Stefan Monnier]

> I think the discussion is too broad.  It depends on what is
> declared obsolete.

IMNSHO it doesn't: if we don't intend to remove it ever, then we don't
declare it obsolete.

Of course, declaring it obsolete doesn't guarantee that we will remove
it: it's just a declaration of intention.


        Stefan

Re: Delete variables obsolete since Emacs 23

[Jeff Norden]

I said:
>> The elisp manual says:
>>     You can mark a named function as "obsolete", meaning that it may be
>>     removed at some point in the future.  This causes Emacs ...
>> The phrase "may be removed" seems a bit vague.  Would "will be removed" or
>> "will probably be removed" be more accurate?

To which Eli Zaretskii replied:
> No, it won't.  Primarily because we don't really know whether we will
> remove it, let alone when.  It depends on too many factors that we
> cannot predict.

But then Stefan Monnier wrote:
> If there's no intention to remove it in the future, then we don't
> declare it obsolete.

I really like the use of 'obsolete' instead of 'deprecate'.
Merriam-Webster gives a two-part definition:
   Definition of obsolete:
     1 a: no longer in use or no longer useful
       b: of a kind or style no longer current : OLD-FASHIONED

I personally think that there is a case for keeping a limited number of
obsolete functions/variables long term, for backward compatibility.
You can apply definition (b) to these, although it's not a perfect fit,
since obsolete functions shouldn't be used in new code.

Windows still supports 8.3 file names (a horrible example, since they
never should have been created in the first place).  My pickup truck
still uses a carburetor, which was certainly made obsolete by fuel
injection, but still works fine for me.  Of course, obsolete features
won't be maintained, updated, etc.  (My local parts stores no longer
stock an air filter that fits my truck.)

But, if a function or variable fits (a), and is really no longer used or
useful, then it only make sense to remove it.

It may not be possible or practical to precisely label each obsolete
emacs item with a category.  But, it would be good to be clear as to
whether or not (make-obsolete) should be regarded as including both
possible cases.

Thanks,
-Jeff

Proposal for an Emacs User Survey

[Adrien Brochard]

Hi everyone,

I posted on reddit.com/r/emacs yesterday a proposal to craft and
distribute a survey for Emacs users to better grasp the diversity and
various usages out there. I got some very good feedback but there could
always be more.

The main points that need to be worked out are:
1. the actual questions to be asked
2. the platform to collect responses

Regarding the questions, I have a decent skeleton and a lot of suggestions.

But the for the platform to collect responses, this is where it gets
difficult. Something like a Google survey might the easiest/cheapest to
make and distribute and will work on most devices, but it obviously has
privacy concerns. On the other hand, doing something like self-hosting a
no-JS form is very time consuming. I was also thinking of allowing
responses via email for people who do not want to bother with a survey
link and then manually reconcile responses.

Links to the posts with the tentative questions:
-
https://www.reddit.com/r/emacs/comments/j6x7ad/proposal_for_an_emacs_user_survey/
-
https://www.reddit.com/r/emacs/comments/b2fh4y/help_reviewprovide_feedback_on_state_of_emacs/

Thank you for reading!
Best,
Adrien

Re: Proposal for an Emacs User Survey

[Philip K.]

Adrien Brochard <abrochard@gmx.com> writes:

> Hi everyone,
>
> I posted on reddit.com/r/emacs yesterday a proposal to craft and
> distribute a survey for Emacs users to better grasp the diversity and
> various usages out there. I got some very good feedback but there could
> always be more.
>
> The main points that need to be worked out are:
> 1. the actual questions to be asked
> 2. the platform to collect responses

I think a major problem is how to get as many emacs users -- and not
only reddit -- to participate. There are plenty of Emacs users who
neither follow the mailing lists, nor use sites like Reddit. Many of
them have had similar setups for years, and don't need to think about
it. Of course I can inform those I am antiquated with, but that approach
has it's limits.

There is a disconnect between these two groups. Of course, neither is
"right". I've seen "traditional users" who didn't know that M-x could be
invoked using the alt key (instead pressing escape and then "x"), and
newer users who use try to get as much of emacs out of the way, to use
things like org or magit. My fear is that those in the latter group are
more likely to hear about a poll like this, and be overrepresented at
the expense of the former. And without a way to prove anything, either
way, no real progress could be achieved -- one group will keep on saying
that "it's time to become vs code" and the others will invoke a virtual
"silent minority".

> Regarding the questions, I have a decent skeleton and a lot of suggestions.
>
> But the for the platform to collect responses, this is where it gets
> difficult. Something like a Google survey might the easiest/cheapest to
> make and distribute and will work on most devices, but it obviously has
> privacy concerns. On the other hand, doing something like self-hosting a
> no-JS form is very time consuming. I was also thinking of allowing
> responses via email for people who do not want to bother with a survey
> link and then manually reconcile responses.

Why not just create a simple HTML form with a captcha (not Google's
thing, but a simple library that generates an image)?

> Links to the posts with the tentative questions:
> -
> https://www.reddit.com/r/emacs/comments/j6x7ad/proposal_for_an_emacs_user_survey/
> -
> https://www.reddit.com/r/emacs/comments/b2fh4y/help_reviewprovide_feedback_on_state_of_emacs/
>
> Thank you for reading!
> Best,
> Adrien
>
>

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you for your response!

>
> I think a major problem is how to get as many emacs users -- and not
> only reddit -- to participate. There are plenty of Emacs users who
> neither follow the mailing lists, nor use sites like Reddit. Many of
> them have had similar setups for years, and don't need to think about
> it. Of course I can inform those I am antiquated with, but that approach
> has it's limits.

There is a real risk indeed of selection bias because it is hard to
reach out to people who are not on the mailing list or on popular
channels. I identified this issue on the reddit post, but aside from
sharing the survey on the mailing list/reddit/irc/telegram/blogs and
keeping the survey open for 1 month, I am not sure how to mitigate more.
It is in a way the fundamental problem of surveys though. And I should
specify that there's value in just getting any response, and that we
should be mindful of this selection bias risk and not make carefree
decisions based on the survey results.

> Why not just create a simple HTML form with a captcha (not Google's
> thing, but a simple library that generates an image)?

I know it sounds easy, but it can rapidly become a considerable amount
of work. The captcha is not trivial and then there's the question of the
backend and where the data goes. To limit barrier of entry, the survey
must be as easy to fill out as possible. That means that we can't have
downtime, lag, and work on mobile too. I am more inclined to trust that
work to professionals and have to pay a little for it.

Re: Proposal for an Emacs User Survey

[Philip K.]

Adrien Brochard <abrochard@gmx.com> writes:

>> Why not just create a simple HTML form with a captcha (not Google's
>> thing, but a simple library that generates an image)?
>
> I know it sounds easy, but it can rapidly become a considerable amount
> of work. The captcha is not trivial and then there's the question of the
> backend and where the data goes. To limit barrier of entry, the survey
> must be as easy to fill out as possible. That means that we can't have
> downtime, lag, and work on mobile too. I am more inclined to trust that
> work to professionals and have to pay a little for it.

For the fun of it, I wrote a simple survey application[0] that includes
a self-hosted captcha (without tracking anyone), requires no Javascript,
is mobile friendly and should be fairly fast. Of course it can be
improved. I wrote it as a quick and dirty evening project, but I
think it demonstrates that this kind of an approach is practicable.

[0] https://git.sr.ht/~zge/survey

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[tomas]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

On Fri, Oct 09, 2020 at 01:30:48PM +0200, Philip K. wrote:

[...]

> For the fun of it, I wrote a simple survey application[0] [...]

Thanks! I'll look into it. Perhaps I'll even steal something
from it -- I'll let you know.

Who knows. We might need something akin to suckless, for the web :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

> For the fun of it, I wrote a simple survey application[0] that includes
> a self-hosted captcha (without tracking anyone), requires no Javascript,
> is mobile friendly and should be fairly fast. Of course it can be
> improved. I wrote it as a quick and dirty evening project, but I
> think it demonstrates that this kind of an approach is practicable.
>
> [0] https://git.sr.ht/~zge/survey

Thank you! I really like reading go code (no sarcasm).
It it clear that this approach will work but the remaining 20% are the
hardest. I'm concerned with:
- cleaner UI, not just HTML embedded in the code
- mysql instead of sqlite, which also implies a mysql instance running
- DOS protection, maybe some rate-limiting and IP blocking
- HTTPS, thankfully it's easier now
- monitoring, how do we know the service is running as expected
- logging, and how to store logs for debug

My greatest fear is people clicking on the link, filling out the form,
trying to submit, seeing an error page, and never coming back to it.

Best,
Adrien

Re: Proposal for an Emacs User Survey

[Philip K.]

Adrien Brochard <abrochard@gmx.com> writes:

>> For the fun of it, I wrote a simple survey application[0] that includes
>> a self-hosted captcha (without tracking anyone), requires no Javascript,
>> is mobile friendly and should be fairly fast. Of course it can be
>> improved. I wrote it as a quick and dirty evening project, but I
>> think it demonstrates that this kind of an approach is practicable.
>>
>> [0] https://git.sr.ht/~zge/survey
>
> Thank you! I really like reading go code (no sarcasm).
> It it clear that this approach will work but the remaining 20% are the
> hardest.  I'm concerned with:
> - cleaner UI, not just HTML embedded in the code

What would you have in mind? I know that my example is simple, but I
prefer that to sites that take forever to load and reload everything all
the time, for no apparent reason.

> - mysql instead of sqlite, which also implies a mysql instance running

SQLite is actually surprisingly resilient, according to [0]:

> SQLite works great as the database engine for most low to medium traffic
> websites (which is to say, most websites). The amount of web traffic
> that SQLite can handle depends on how heavily the website uses its
> database. Generally speaking, any site that gets fewer than 100K
> hits/day should work fine with SQLite. The 100K hits/day figure is a
> conservative estimate, not a hard upper bound. SQLite has been
> demonstrated to work with 10 times that amount of traffic.

But either way, I used sqlite to avoid setting up a RDBMS.

[0] https://www.sqlite.org/whentouse.html

> - DOS protection, maybe some rate-limiting and IP blocking
> - HTTPS, thankfully it's easier now

AFAIK these things can usually be handled by a frond-end such as NGINX.

> - monitoring, how do we know the service is running as expected
> - logging, and how to store logs for debug

If there is any interest, extending the example to support this would be
feasible. The "20%" you mention aren't easy, but from what I see it
shouldn't be too hard either.

> My greatest fear is people clicking on the link, filling out the form,
> trying to submit, seeing an error page, and never coming back to it.

The reason I suggested using a simple HTML form and implemented the demo
was because I think simplicity helps avoid a lot of issues. With fewer
dependencies, secondary services, modules, etc. the chance of one of
these failing decreases. And having a smaller footprint should also
reduce the network load.

> Best,
> Adrien

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

>> - cleaner UI, not just HTML embedded in the code
>
> What would you have in mind? I know that my example is simple, but I
> prefer that to sites that take forever to load and reload everything all
> the time, for no apparent reason.

The most obvious reason to me is that user error handling is pretty
poor. Because there is no JS, we cannot offer front-end validation, that
means that the backend server is responsible for validating fields
submitted. If validation does not pass, the page must "reload" for the
user and it needs to show exactly what went wrong and preserve the user
input. That's my definition of a site that reloads all the time.


>> - mysql instead of sqlite, which also implies a mysql instance running
>
> SQLite is actually surprisingly resilient, according to [0]:
>
>> SQLite works great as the database engine for most low to medium traffic
>> websites (which is to say, most websites). The amount of web traffic
>> that SQLite can handle depends on how heavily the website uses its
>> database. Generally speaking, any site that gets fewer than 100K
>> hits/day should work fine with SQLite. The 100K hits/day figure is a
>> conservative estimate, not a hard upper bound. SQLite has been
>> demonstrated to work with 10 times that amount of traffic.
>
> But either way, I used sqlite to avoid setting up a RDBMS.
>
> [0] https://www.sqlite.org/whentouse.html

SQLite is resilient but it's dangerous to use it with multiple threads
like how your go server does. You could use a single channel model to
write records one at a time but now you have a risk to lose data from
memory if your service restarts.

>
>> - DOS protection, maybe some rate-limiting and IP blocking
>> - HTTPS, thankfully it's easier now
>
> AFAIK these things can usually be handled by a frond-end such as NGINX.

That's true it can, but that means you need to deploy and configure one.
https://www.nginx.com/blog/rate-limiting-nginx/
https://nginx.org/en/docs/http/configuring_https_servers.html


>> - monitoring, how do we know the service is running as expected
>> - logging, and how to store logs for debug
>
> If there is any interest, extending the example to support this would be
> feasible. The "20%" you mention aren't easy, but from what I see it
> shouldn't be too hard either.

You're absolutely right. None of it is "too hard". But the problem is
just how much time and effort are we willing to put into it. If the goal
is to make an open source polling platform that we can re-use in the
future, then absolutely all of this can be done. But if we treat this as
a one-off and see how it turns out, the scope seems exaggerated.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The most obvious reason to me is that user error handling is pretty
  > poor. Because there is no JS, we cannot offer front-end validation, that
  > means that the backend server is responsible for validating fields
  > submitted.

If we want to learn what users think, we should not limit their
responses to a small set of 'valid" possible answers.  The plan
I designed for inquiries asks users to answer in their own words.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Philip K.]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > The most obvious reason to me is that user error handling is pretty
>   > poor. Because there is no JS, we cannot offer front-end validation, that
>   > means that the backend server is responsible for validating fields
>   > submitted.
>
> If we want to learn what users think, we should not limit their
> responses to a small set of 'valid" possible answers.  The plan
> I designed for inquiries asks users to answer in their own words.

But wouldn't that make it needlessly hard to analyse the results,
especially if the question should be numerically quantified? I get the
value of plain text responses, but recognizing that the answer to the
question "how long have you been using emacs", could result in:

- "Since 1996"
- "24 Years"
- "January 1996"
- "Around the second time Clinton got sworn into office"
- "1896" (but actually a typo)

All of these basically mean the same, but the effort to recognize this
automatically (which would be necessary, if you want to see how factors
correlate) would not be worth it.

-- 
	Philip K.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> > If we want to learn what users think, we should not limit their
> > responses to a small set of 'valid" possible answers.  The plan
> > I designed for inquiries asks users to answer in their own words.
> 
> But wouldn't that make it needlessly hard to analyse the results,

Ease in handling the results shouldn't be a major
consideration.  Quality of the feedback and quality
of its analysis is more important that speed or ease.

> especially if the question should be numerically quantified?

Quantification of bad-quality information doesn't
help.  Better to get a high-quality understanding,
and if some quantification _then_ helps then do it.

> I get the value of plain text responses, but recognizing that the answer to the
> question "how long have you been using emacs", could result in:
> 
> - "Since 1996"
> - "24 Years"
> - "January 1996"
> - "Around the second time Clinton got sworn into office"
> - "1896" (but actually a typo)

A very specific question like that can be asked in
specific terms: how many years.

But even that is likely to be misleading, if kept
apart from considerations of kind and degree of use.

Getting a good picture of Emacs users and uses is not
easy, nor should it be.  And again, users _reasons_
for actions, preferences, etc. are important.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 10.10.2020 19:33, Drew Adams wrote:
> Ease in handling the results shouldn't be a major
> consideration.  Quality of the feedback and quality
> of its analysis is more important that speed or ease.

Difficulty in analysis is likely to result in a lack of transparency of 
such analysis's results.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Philip K. <philipk@posteo.net> [2020-10-10 12:37]:
> Richard Stallman <rms@gnu.org> writes:
> 
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> >   > The most obvious reason to me is that user error handling is pretty
> >   > poor. Because there is no JS, we cannot offer front-end validation, that
> >   > means that the backend server is responsible for validating fields
> >   > submitted.
> >
> > If we want to learn what users think, we should not limit their
> > responses to a small set of 'valid" possible answers.  The plan
> > I designed for inquiries asks users to answer in their own words.
> 
> But wouldn't that make it needlessly hard to analyse the results,
> especially if the question should be numerically quantified?

As I have done larger surveys for public relations and I know methods,
I know how tedious it is to evaluate such survey, we have been
employing many people, like 20 people, to just analyze what exactly
did people check or did not check, what did they write, to read their
handwriting, and then to properly analyze it.

However, Emacs feature requests or survey about using Emacs need live
user, not user as a number.

> I get the value of plain text responses, but recognizing that the
> answer to the question "how long have you been using emacs", could
> result in:
> 
> - "Since 1996"
> - "24 Years"
> - "January 1996"
> - "Around the second time Clinton got sworn into office"
> - "1896" (but actually a typo)
> 
> All of these basically mean the same, but the effort to recognize this
> automatically (which would be necessary, if you want to see how factors
> correlate) would not be worth it.

It is better not to do it automatically.

Emacs is developed through number of developers, they are not doing
development automatically, they are well coordinating and
collaborating with each other.

Any user who is sending a request is also collaborating in such
development by proposing features or telling about dislikes, telling
their opinion.

Those users are valuable, their answers should not remain as numbers
on the paper, or communication that comes only one time.

Kind answer on their email or requests, likes or dislikes, will
involve users into collaboration and coordination and better
understanding of what such users were thinking.

We already have mailing lists for Emacs, but we do not have it well
integrated into Emacs, that users can find out about it.

So I am proposing to follow the friendly GNU Hyperbole
https://www.gnu.org/s/hyperbole example, where
Bob Weiner have implemented Hyperbole menu, from which every user is
enabled to subscribe to Hyperbole mailing list or to send email to
Hyperbole mailing list.

Thus menu item like "Tell us how to improve Emacs?" in Help menu could
be the initiation to such collaboration.

And I think that many Emacs users do not use Emacs to send email, so
there shall be option by using eww, to send a form, and form can be
already packaged with Emacs.

Thus user could decide to:

- subscribe to improvement mailing list

- send email with or without subscription, just as almost any GNU
  list,

- unsubscribe

- or send a WWW form, by using eww function

The form would be in HTML format, accessible to eww easily.

Over time, the form could be improved or changed in new versions.

It could include free field to write what one wants, or it could
include set of question as well.

Now this opens the question that the other mailing list,
help-gnu-emacs should also be integrated in the menu, just as the user
survey.

I recommend everybody to install hyperbole package and to see how it
was implemented, it gives incentive to user to collaborate and
coordinate, and if it would not be implemented in the menu, user would
hardly find the mailing list.

Jean

Re: Proposal for an Emacs User Survey

[Philip K.]

Jean Louis <bugs@gnu.support> writes:

> * Philip K. <philipk@posteo.net> [2020-10-10 12:37]:
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> >
>> >   > The most obvious reason to me is that user error handling is pretty
>> >   > poor. Because there is no JS, we cannot offer front-end validation, that
>> >   > means that the backend server is responsible for validating fields
>> >   > submitted.
>> >
>> > If we want to learn what users think, we should not limit their
>> > responses to a small set of 'valid" possible answers.  The plan
>> > I designed for inquiries asks users to answer in their own words.
>> 
>> But wouldn't that make it needlessly hard to analyse the results,
>> especially if the question should be numerically quantified?
>
> As I have done larger surveys for public relations and I know methods,
> I know how tedious it is to evaluate such survey, we have been
> employing many people, like 20 people, to just analyze what exactly
> did people check or did not check, what did they write, to read their
> handwriting, and then to properly analyze it.
>
> However, Emacs feature requests or survey about using Emacs need live
> user, not user as a number.

Of course, but there are still numbers that describe aggregate
phenomenons that individual users don't actively notice. A question I
would be interested in is what the correlation is between people who use
specific configuration-templates (Doom, Spacemacs, etc.) and how long
they have been using Emacs/Age. Depending on what the results are, we
would have a batter guess as to whether the popularity of these
templates is just because newer users aren't secure in configuring their
own Emacs, or if people just like these templates in general (what they
like is individual, that's where plain text responses are interesting).

Other than that, I don't see why both approaches should be possible.
Mixed, or separated, you can ask multiple/single choice questions for
"hard data", and plain text for individual opinions.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Philip K. <philipk@posteo.net> [2020-10-11 21:42]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Philip K. <philipk@posteo.net> [2020-10-10 12:37]:
> >> Richard Stallman <rms@gnu.org> writes:
> >> 
> >> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> >> > [[[ whether defending the US Constitution against all enemies,     ]]]
> >> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >> >
> >> >   > The most obvious reason to me is that user error handling is pretty
> >> >   > poor. Because there is no JS, we cannot offer front-end validation, that
> >> >   > means that the backend server is responsible for validating fields
> >> >   > submitted.
> >> >
> >> > If we want to learn what users think, we should not limit their
> >> > responses to a small set of 'valid" possible answers.  The plan
> >> > I designed for inquiries asks users to answer in their own words.
> >> 
> >> But wouldn't that make it needlessly hard to analyse the results,
> >> especially if the question should be numerically quantified?
> >
> > As I have done larger surveys for public relations and I know methods,
> > I know how tedious it is to evaluate such survey, we have been
> > employing many people, like 20 people, to just analyze what exactly
> > did people check or did not check, what did they write, to read their
> > handwriting, and then to properly analyze it.
> >
> > However, Emacs feature requests or survey about using Emacs need live
> > user, not user as a number.
> 
> Of course, but there are still numbers that describe aggregate
> phenomenons that individual users don't actively notice. A question I
> would be interested in is what the correlation is between people who use
> specific configuration-templates (Doom, Spacemacs, etc.) and how long
> they have been using Emacs/Age. Depending on what the results are, we
> would have a batter guess as to whether the popularity of these
> templates is just because newer users aren't secure in configuring their
> own Emacs, or if people just like these templates in general (what they
> like is individual, that's where plain text responses are
> interesting).

I have the book of I guess Robert Kyosaki here, I cannot find at
moment exact reference, the idea I got from the book (or maybe not),
is that surveys can be done easily, one can start asking even less
number of users and with simple questions, and one will very soon find
a pattern that becomes very probable final survey result.

As those configuration templates are located on Microsoft Github, one
way to make the survey or find out that information is to submit the
issue there in each configuration, and simply ask them, you will find
out.

Another fact is that developers of those configurations obviously
already did their own analysis on what is needed and wanted and
produced that what became popular configuration. Their survey is
already done. Thus if Spacemacs configuration is most popular, then
core Emacs developers could use those results of established famous
features and see if anything could be implemented easier from
upstream. 

My personal view point on Spacesmacs, Doom, etc. is that those group
of people like shiny, like showing off their theming, they have time
for their hobby, but I have not researched enough, did not install any
of those, I don't even install external themes, I use only those built
ins.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> If we want to learn what users think, we should not limit their
> responses to a small set of 'valid" possible answers.  The plan
> I designed for inquiries asks users to answer in their own words.

That's my suggestion as well.

Re: Proposal for an Emacs User Survey

[Philip K.]

Adrien Brochard <abrochard@gmx.com> writes:

>>> - cleaner UI, not just HTML embedded in the code
>>
>> What would you have in mind? I know that my example is simple, but I
>> prefer that to sites that take forever to load and reload everything all
>> the time, for no apparent reason.
>
> The most obvious reason to me is that user error handling is pretty
> poor. Because there is no JS, we cannot offer front-end validation, that
> means that the backend server is responsible for validating fields
> submitted. If validation does not pass, the page must "reload" for the
> user and it needs to show exactly what went wrong and preserve the user
> input. That's my definition of a site that reloads all the time.

That could be mitigated with a "graceful degradation" approach, since
most people do have javascript activated by default. HTML5 also has a
few attributes that could help, such as pattern or required, depending
on what questions are being asked.

>>> - mysql instead of sqlite, which also implies a mysql instance running
>>
>> SQLite is actually surprisingly resilient, according to [0]:
>>
>>> SQLite works great as the database engine for most low to medium traffic
>>> websites (which is to say, most websites). The amount of web traffic
>>> that SQLite can handle depends on how heavily the website uses its
>>> database. Generally speaking, any site that gets fewer than 100K
>>> hits/day should work fine with SQLite. The 100K hits/day figure is a
>>> conservative estimate, not a hard upper bound. SQLite has been
>>> demonstrated to work with 10 times that amount of traffic.
>>
>> But either way, I used sqlite to avoid setting up a RDBMS.
>>
>> [0] https://www.sqlite.org/whentouse.html
>
> SQLite is resilient but it's dangerous to use it with multiple threads
> like how your go server does. You could use a single channel model to
> write records one at a time but now you have a risk to lose data from
> memory if your service restarts.

I'm familiar with the danger, but it shouldn't be a problem. As pointed
out in [0], one can activate the SQLite mutex per flag. But at the same
time, even sites as popular as Hacker News are alleged to use SQLite as
their backend.

[0] https://github.com/mattn/go-sqlite3/issues/249

>>
>>> - DOS protection, maybe some rate-limiting and IP blocking
>>> - HTTPS, thankfully it's easier now
>>
>> AFAIK these things can usually be handled by a frond-end such as NGINX.
>
> That's true it can, but that means you need to deploy and configure one.
> https://www.nginx.com/blog/rate-limiting-nginx/
> https://nginx.org/en/docs/http/configuring_https_servers.html

Perhaps using Guix could make that easier:
https://guix.gnu.org/manual/devel/en/html_node/Web-Services.html#NGINX?

>>> - monitoring, how do we know the service is running as expected
>>> - logging, and how to store logs for debug
>>
>> If there is any interest, extending the example to support this would be
>> feasible. The "20%" you mention aren't easy, but from what I see it
>> shouldn't be too hard either.
>
> You're absolutely right. None of it is "too hard". But the problem is
> just how much time and effort are we willing to put into it. If the goal
> is to make an open source polling platform that we can re-use in the
> future, then absolutely all of this can be done. But if we treat this as
> a one-off and see how it turns out, the scope seems exaggerated.

Depending on how many people participate, it would be interesting the
repeat the survey year-by-year, so that trends and changes in the
user-base can be recognized.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The most obvious reason to me is that user error handling is pretty
  > > poor. Because there is no JS, we cannot offer front-end validation, that
  > > means that the backend server is responsible for validating fields
  > > submitted. If validation does not pass, the page must "reload" for the
  > > user and it needs to show exactly what went wrong and preserve the user
  > > input. That's my definition of a site that reloads all the time.

  > That could be mitigated with a "graceful degradation" approach, since
  > most people do have javascript activated by default. HTML5 also has a
  > few attributes that could help, such as pattern or required, depending
  > on what questions are being asked.

It is ok to send some JS code, which is labeled as free such that
LibreJS recognizes it, which does nothing except validate the inputs.
So that a person who enters valid inputs would see no difference between
running the JS code and not running it.

  > >>> - mysql instead of sqlite, which also implies a mysql instance running
  > >>
  > >> SQLite is actually surprisingly resilient, according to [0]:

It would be a mistake to put the input into any sort of database
program.  That would only make things difficult and increase the
special knowledge people would need in order to work on this.

Here's the simple and clean way to do it.

Each time a user responds, convert all the answers into a formulaic
piece text which states the questions and answers.  Email that to a
certain address, which will accumulate these messages in one inbox
file.

Then it will be simple to do all sorts of counting or analysis on that
file.

Database programs are useful when there are hundreds of thousands of
records and you need to search for any one of them.  Or when the data
are being altered.  We will not have so many answers, and each one
will never change once sent.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Philip K.]

Richard Stallman <rms@gnu.org> writes:

>   > >>> - mysql instead of sqlite, which also implies a mysql instance running
>   > >>
>   > >> SQLite is actually surprisingly resilient, according to [0]:
>
> It would be a mistake to put the input into any sort of database
> program.  That would only make things difficult and increase the
> special knowledge people would need in order to work on this.
>
> Here's the simple and clean way to do it.
>
> Each time a user responds, convert all the answers into a formulaic
> piece text which states the questions and answers.  Email that to a
> certain address, which will accumulate these messages in one inbox
> file.
>
> Then it will be simple to do all sorts of counting or analysis on that
> file.

The issue I see here is that if you expert plain-text responses, someone
might just submit something that breaks the format, or even submit an
email written in HTML. So the data couldn't just be stored in plain
text, but would have to at least have some decoding to correctly parse
the message, and some encoding the safely store the message. You
wouldn't want someone who could submit 1000 fake or troll responses,
just because they know what what's going on behind the scenes.

> Database programs are useful when there are hundreds of thousands of
> records and you need to search for any one of them.  Or when the data
> are being altered.  We will not have so many answers, and each one
> will never change once sent.

Databases are useful whenever you've got data to work with. SQLite is
useful at whatever sale one is working at, from browsers to cars. And
besides, it's safer to use than directly writing to the file system (see
https://danluu.com/file-consistency/ for more details). And generally
speaking, file systems are also just databases, just that they are
directory-oriented, not record-oriented and usually have better support
in regards to OS primitives.

All in all, I don't this that this should be an issue, SQLite is well
documented, and can easily be extracted into whatever format someone
might need or want, just easier than with a mailbox or other classical
unix-like approaches.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The issue I see here is that if you expert plain-text responses, someone
  > might just submit something that breaks the format, or even submit an
  > email written in HTML.

This is not a major problem.  The only thing that breaks the format of
an inbox file is "From " in column 0, and the worst thing that does is
generate a junk message.

The program that generates the message can easily prevent this from
happening.

  > You
  > wouldn't want someone who could submit 1000 fake or troll responses,
  > just because they know what what's going on behind the scenes.

It's not likely that people will do this, and if they do,
it would not be hard to delete those fake messages from the file.

  > All in all, I don't this that this should be an issue, SQLite is well
  > documented,

Even if it is very good documentation, reading it and learning will
take work.  I don't want to have to read it, or learn how to do
something with SQLite.  It would be an extra unnecessary layer to
cause confusion or problems, and there is no need for that complexity.

                and can easily be extracted into whatever format someone
  > might need or want, just easier than with a mailbox or other classical
  > unix-like approaches.

It is not as easy for people who don't know or use database systems as
it is for you.

If some people find it helpful to analyze the data through a database
system, I have nothing against their doing so.  It won't be hard to
have a program parse the inbox file and enter the data.  The script
can detect when a message was split, and disregard it.

Or people can fix the broken messages with Emacs, either by hand
with a Lisp program.

It is also possible to handle each user's input in two ways
when it is entered: put it into a database _and_ email it.
That way, everyone's preferences are satisfied.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Tomas Hlavaty]

On Sat 10 Oct 2020 at 12:35, "Philip K." <philipk@posteo.net> wrote:
> That could be mitigated with a "graceful degradation" approach, since
> most people do have javascript activated by default.

it should be possible to fill in the survey from emacs

emacs does not support executing javascript code

many people do have javascript disabled by default

many people just close and ignore pages that require javascript

would not it be nice to be able to reply to the survey from emacs?

for example, could the survey work similar to report-emacs-bug where the
template is taken from somewhere dynamically from existing
infrastructure (instead of extra web server), e.g. using a reference to
a message on emacs-survey@gnu.org?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Tomas Hlavaty <tom@logand.com> [2020-10-11 10:33]:
> On Sat 10 Oct 2020 at 12:35, "Philip K." <philipk@posteo.net> wrote:
> > That could be mitigated with a "graceful degradation" approach, since
> > most people do have javascript activated by default.
> 
> it should be possible to fill in the survey from emacs
> 
> emacs does not support executing javascript code

If any website functions only with Javascript, people would launch
external browser from Emacs, thus jumping to Javascript which could be
non-free.

> many people do have javascript disabled by default

Many is vague, and I am sure it is true, but "many" does not represent
majority, you have to think of default settings in major browsers, if
they have Javascript turned on, majority have Javascript turned on,
regardless how many have it disabled.

> many people just close and ignore pages that require javascript

I agree, though many is not equal to majority.

> would not it be nice to be able to reply to the survey from emacs?

Yes, please.

> for example, could the survey work similar to report-emacs-bug where
> the template is taken from somewhere dynamically from existing
> infrastructure (instead of extra web server), e.g. using a reference
> to a message on emacs-survey@gnu.org?

Exactly good idea.

Re: Proposal for an Emacs User Survey

[Philip K.]

Tomas Hlavaty <tom@logand.com> writes:

> On Sat 10 Oct 2020 at 12:35, "Philip K." <philipk@posteo.net> wrote:
>> That could be mitigated with a "graceful degradation" approach, since
>> most people do have javascript activated by default.
>
> it should be possible to fill in the survey from emacs

Of course, that's what "graceful degradation" implies. If Javascript is
available, it will be used to improve the user experience. If not,
everything still continues working, just eg. without the interactivity
that Javascript might provide.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Adrien Brochard <abrochard@gmx.com> [2020-10-08 20:51]:
> > Why not just create a simple HTML form with a captcha (not Google's
> > thing, but a simple library that generates an image)?

I am not using any captcha, simple HTML and server-side spam
blocking. For that, two activities are necessary:

- administrator to watch for eventual spam, and it will take time
  until spam starts coming

- adding list of keywords into the spam-definition

In Common lisp I have something like this that accepts CGI forms:

(defparameter *spam-sets*
  '(("gratitude" "website" "magnificent" "investigation")
    ("face" "mask" "coronavirus")
    ("face" "mask")
    ("blackhat.to")
    ("disposable" "mask")
    ("medical" "mask")
    ("mask" "protect")
    ("mask" "virus")))

Only if all keywords appear, the text is rejected.

In my opinion, it is better that administrator does administration
then having all users checked for "being human" and spending time with
the form.

Jean

Re: Proposal for an Emacs User Survey

[Karl Fogel]

On 08 Oct 2020, Adrien Brochard wrote:
>But the for the platform to collect responses, this is where it gets
>difficult. Something like a Google survey might the easiest/cheapest to
>make and distribute and will work on most devices, but it obviously has
>privacy concerns. On the other hand, doing something like self-hosting a
>no-JS form is very time consuming. I was also thinking of allowing
>responses via email for people who do not want to bother with a survey
>link and then manually reconcile responses.

There is a free software web-based survey tool named LimeSurvey:

  - https://en.wikipedia.org/wiki/LimeSurvey

  - https://github.com/LimeSurvey/LimeSurvey

  - https://www.limesurvey.org

LimeSurvey.org seems to actually be a company that provides commercial hosting.  It's not clear to me whether the version they host is just the free software version or whether it has some proprietary bits added, but in any case one could just run the free software version oneself.  (I know a project that did so many years ago, and they had a good experience with it, FWIW.)

Best regards,
-Karl

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

> There is a free software web-based survey tool named LimeSurvey:
>
>    - https://en.wikipedia.org/wiki/LimeSurvey
>
>    - https://github.com/LimeSurvey/LimeSurvey
>
>    - https://www.limesurvey.org
>
> LimeSurvey.org seems to actually be a company that provides commercial hosting.  It's not clear to me whether the version they host is just the free software version or whether it has some proprietary bits added, but in any case one could just run the free software version oneself.  (I know a project that did so many years ago, and they had a good experience with it, FWIW.)
>
> Best regards,
> -Karl

Thank you for the tip! I did look into that after someone recommended
it. I like it, the main problem is indeed that I would have to
self-host, which is not something I am comfortable doing at this point,
at least not with the first iteration. And on the other side, their
hosted version is very expensive unfortunately.

Best,
Adrien

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There is a free software web-based survey tool named LimeSurvey:

Perhaps it would be ethical to use a platform like that, when what we
want is a formulaic survey.  But I think it is the wrong sort of tool
for this job.

We had a long discussion on this list, in April and/or May I think,
about how to do polling about a simple change in features.  What we
need is to invite people to explain their preferences or wishes.

Sorry, I don't have a pointer to any of those messages so I can't
say what to search for, except the words "polling" and "inquiry".


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

Hi Richard,

On 10.10.2020 06:52, Richard Stallman wrote:
> We had a long discussion on this list, in April and/or May I think,
> about how to do polling about a simple change in features.  What we
> need is to invite people to explain their preferences or wishes.

Are you going to do that survey, the one discussed several months ago?

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We had a long discussion on this list, in April and/or May I think,
  > > about how to do polling about a simple change in features.  What we
  > > need is to invite people to explain their preferences or wishes.

  > Are you going to do that survey, the one discussed several months ago?

That was an inquiry about using tabs for indentation.

I have been too busy but I will try to get it moving.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think using Google to communicate the users is bad because Google
does so many bad things.  We should not endorse it with our actions.

In addition, most Google services require users to run nonfree
software (Javascript code).  If answering our questions requires the
user to run some nonfree JS code, that would put us in a moral
contradiction.

Our goal is much more than just developing a successful editor; GNU
Emacs is a GNU package, and the purpose of the GNU operating system is
to win freedom for the users.  We have to set an example of respecting
and defending freedom in the methods we use, not just the program we
are working on.  See https://gnu.org/philosophy/free-sw.html and
https://gnu.org/philosophy/free-software-even-more-important.html.

I think Reddit is the wrong place to discuss this plan, for a similar
reason.  As I understand it, Reddit requires nonfree software to post
on Reddit nowadays.  Is that not so?

The place to discuss the questions is here.  Would you please post
here your proposal for the questions to ask?


Meanwhile,

Here's the method I propose for inquiries (we used to call them "polls")
about users' views about possible changes in well-known behaviors.

* Make a file for the replies to go into.

* Make a mailing address which drops all mail into the file.

* Write a inquiry statement which describes the proposed change in
sufficient detail that people can judge it, what kind of information
we seek, and where to email the response.  Include a deadline for
replies at least 6 weeks in the future.

If possible, we should tell users how to select various behaviors, so
that they can state their opinions based on comparing actual
experiences.

End the inquiry statement with the following text.

    We do not seek "votes", but rather understanding.  If you are for
    the change, please explain why.  Would it help you directly?  If
    so, in what scenarios?  How often, and how strongly, would it
    benefit you?  What would the benefit be?

    Or is it that you think it will improve Emacs, or speed Emacs
    development, by helping other users?  How so?

    Please distinguish between what you know and what you predict.

    Likewise, if you are against the change, please explain why.
    Would it inconvenience you directly?  If so, in what scenarios?
    How often, and how strongly, would it inconvenience you?  What
    would the inconvenience be?

    Or is it that you think it will harm Emacs or Emacs development by
    inconveniencing others?  How so?

    We invite you also to propose alterations in the proposed change that
    you think would improve it -- saying in what scenario that would be an
    improvement, and how so, etc.

    Please post the URL of this page in forums where it is appropriate,
    and resend it to Emacs users and mailing lists where you know people
    will be glad to receive it.

It is crucial to urge people repeatedly to explain their positions
because there is a strong tendency for people not to bother.

* Put the inquiry statement in a web page under gnu.org/software/emacs.

* Mail the inquiry statement info-gnu-emacs, help-gnu-emacs and
emacs-tangents, with the URL of the web page _and_ the full text
of the inquiry statement.

Also post a note referring to the web page on reddit.com/r/emacs, and
any other suitable places.  Mail the URL to Sacha Chua
<sacha@sachachua.com>.

* Wait until at least two weeks after the deadline, then study and
record the responses.  Note down all interesting comments, since they
are the most important information in the responses.

* Do count how many people support each position that people support,
but it would be a mistake to make the actual decision based simply on
counting.  A given change can affect one user very often, and affect
another user only rarely, but they could both state a "strong
preference".

* We are not compelled to choose between "make that change" and "no
change".  The best outcome of the inquiry is that the responses show
us how to design a way to please almost all users, almost all the
time, and not displease any user very much.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-09 06:37]:
> * Make a file for the replies to go into.
> 
> * Make a mailing address which drops all mail into the file.

That was specific proposal, it sounds good, I wish to understand who
will implement it?

Jean

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you for your response. Please find the list of questions I have
gathered at the end of this email.

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I think using Google to communicate the users is bad because Google
> does so many bad things.  We should not endorse it with our actions.
>
> In addition, most Google services require users to run nonfree
> software (Javascript code).  If answering our questions requires the
> user to run some nonfree JS code, that would put us in a moral
> contradiction.
>
> Our goal is much more than just developing a successful editor; GNU
> Emacs is a GNU package, and the purpose of the GNU operating system is
> to win freedom for the users.  We have to set an example of respecting
> and defending freedom in the methods we use, not just the program we
> are working on.  See https://gnu.org/philosophy/free-sw.html and
> https://gnu.org/philosophy/free-software-even-more-important.html.
>
> I think Reddit is the wrong place to discuss this plan, for a similar
> reason.  As I understand it, Reddit requires nonfree software to post
> on Reddit nowadays.  Is that not so?

I understand the position. My primary concern though is that without
some convenience, it is difficult to convince a lot of people to
respond. Which is why I am proposing two channels: an a dedicated email
address that will accept plain text responses (at the risk of data
quality), and an online form. That plain text form to be email can be
hosted on a static site that people can access from Emacs itself.

> If possible, we should tell users how to select various behaviors, so
> that they can state their opinions based on comparing actual
> experiences.
>
> End the inquiry statement with the following text.
>
>      We do not seek "votes", but rather understanding.  If you are for
>      the change, please explain why.  Would it help you directly?  If
>      so, in what scenarios?  How often, and how strongly, would it
>      benefit you?  What would the benefit be?
>
>      Or is it that you think it will improve Emacs, or speed Emacs
>      development, by helping other users?  How so?
>
>      Please distinguish between what you know and what you predict.
>
>      Likewise, if you are against the change, please explain why.
>      Would it inconvenience you directly?  If so, in what scenarios?
>      How often, and how strongly, would it inconvenience you?  What
>      would the inconvenience be?
>
>      Or is it that you think it will harm Emacs or Emacs development by
>      inconveniencing others?  How so?
>
>      We invite you also to propose alterations in the proposed change that
>      you think would improve it -- saying in what scenario that would be an
>      improvement, and how so, etc.
>
>      Please post the URL of this page in forums where it is appropriate,
>      and resend it to Emacs users and mailing lists where you know people
>      will be glad to receive it.
>
> It is crucial to urge people repeatedly to explain their positions
> because there is a strong tendency for people not to bother.

I am all in favor of this. Thank you for putting it together. Let me
know if you think the wording is still coherent given the questions.

> * Put the inquiry statement in a web page under gnu.org/software/emacs.
>
> * Mail the inquiry statement info-gnu-emacs, help-gnu-emacs and
> emacs-tangents, with the URL of the web page _and_ the full text
> of the inquiry statement.
>
> Also post a note referring to the web page on reddit.com/r/emacs, and
> any other suitable places.  Mail the URL to Sacha Chua
> <sacha@sachachua.com>.
>
> * Wait until at least two weeks after the deadline, then study and
> record the responses.  Note down all interesting comments, since they
> are the most important information in the responses.

Yes to all of that.

> * Do count how many people support each position that people support,
> but it would be a mistake to make the actual decision based simply on
> counting.  A given change can affect one user very often, and affect
> another user only rarely, but they could both state a "strong
> preference".
>
> * We are not compelled to choose between "make that change" and "no
> change".  The best outcome of the inquiry is that the responses show
> us how to design a way to please almost all users, almost all the
> time, and not displease any user very much.

I also agree with this. I think no matter what we are going to get
selection bias and we should not rush any decision based on the survey
results.


* Survey Questions
(Please note that any "other" should allow for free text entry for the
respondent to elaborate)

** How would you characterize your use of Emacs?
    - Use it for work
    - I use it for serious "hobby" projects
    - I'm just tinkering
    - I use it for my studies
    - Other

** What do you use Emacs for?
     - Software development
     - Research writing
     - Data science
     - Writing
     - Other

** How long have you been using Emacs?

** Which version of Emacs do you use?

** What OS do you primary use?
     - Linux
     - Windows
     - MacOS
     - BSD
     - Other

** How do you use Emacs?
     - GUI
     - Terminal (TUI)
     - Both

** How do you run Emacs?
     - Client/Server mode only
     - Standalone application only
     - Both

** Describe your configuration
    - I am using vanilla emacs (little to no configuration).
    - Custom configuration
    - Spacemacs - https://github.com/syl20bnr/spacemacs
    - Doom Emacs - https://github.com/hlissner/doom-emacs
    - Prelude - https://github.com/bbatsov/prelude
    - purcell emacs.d - https://github.com/purcell/emacs.d
    - magnars emacs.d - https://github.com/magnars/.emacs.d
    - Emacs Starter Kit - https://github.com/eschulte/emacs-starter-kit
    - oh-my-emacs - https://github.com/xiaohanyu/oh-my-emacs
    - Better Defaults - https://github.com/technomancy/better-defaults
    - Graphene - https://github.com/rdallasgray/graphene
    - ohai-emacs - https://github.com/bodil/ohai-emacs
    - ergoemacs-mode - https://github.com/ergoemacs/ergoemacs-mode
    - Rho Emacs - https://github.com/GChristensen/rho-emacs
    - Radian - https://github.com/raxod502/radian
    - Centaur Emacs - https://github.com/seagle0128/.emacs.d
    - Other

** What keybindings do you use?
    -  Emacs defaults
    -  Evil/Spacemacs/Doom-Emacs (all the vim-likes)
    -  CUA-mode
    -  God-mode
    -  Boon
    -  Xah-Fly-Keys
    -  Custom modal (ryo-modal, etc)
    -  Custom modifiers (Emacs from scratch)
    -  Other

** Prior to using Emacs what was your primary editor?
    - VIM
    - VScode
    - Eclipse
    - Notepad++
    - Sublime
    - Intelij
    - Other

** org-mode usage
     - I use emacs only for org-mode
     - dayly
     - from time to time
     - Not a org-mode user

** Which completion/selection framework do you use?
    - helm
    - ivy
    - ido
    - icomplete
    - other
    - i don't use a completion framework

** How do you manage third-party elisp?
     - built-in package.el
     - Spacemacs does it for me
     - straight.el
     - borg
     - leaf
     - el-get
     - Nix
     - git submodules without Borg
     - git subtrees
     - git clones
     - guix
     - quelpa
     - cask
     - No third-party deps
     - other

** How do you get emacs packages(if applicable)?
     - No repos
     - Gnu Elpa
     - Melpa/Melpa Stable
     - Directly from the source (e. g. using straight).

** Can you list some of your favorite package?

** What package do you use for error checking?
    - Flymake
    - Flycheck
    - None

** Do you use TRAMP?

** DO you use Magit?

** What package do you use for project management?
     - project.el
     - projectile
     - Other(mention)
     - None

** Do you use shell/terminal emulator in Emacs?
     - eshell
     - shell
     - term
     - ansi-term
     - do not use.

** Do you use mail client in Emacs?
     - Mu4e
     - Gnus
     - Mut
     - notmuch
     - do not use

** What is your Elisp proficiency?
     - Beginner/No knowledge
     - Basic elisp understanding
     - Intermediate
     - Advanced
     - Expert

** If you use Emacs for programming, which languages do you program in?

** do you use a language server with lsp-mode? With what languages?

** Do you use Emacs for debugging? What do you use? (Gdb, dap-mode etc)

** Which theme do you use?

** Have you ever contributed to GNU Emacs core/Elpa?
     - No
     - No, but I would do that if the process is changed(e. g. using
github pull requests instead of the mailing list, no papers, etc**.
     - I do PRs from time to time
     - I provide PRs regularly
     - I am active contributor/maintainer

** Have you ever contributed to Melpa package?
     - No
     - I do PRs from time to time
     - I provide PRs regularly
     - I am a package maintainer

** What Emacs community forums have you visited in the past year?
Answers would be things like
     - r/emacs
     - Emacs mailing list
     - irc
     - Emacs meetups
     - I follow twitter Emacs related accounts
     - Other

** What are some of the Emacs improvements you are the most interested in?
** what do you think are Emacs' greatest strengths?
** can you recall any difficulties you faced initially learning Emacs?
** What is the one thing you would like emacs to do differently?
** How did you hear about this survey?
** If there is another survey in 2021, would you be opposed to it
containing optional & general demographics questions?
    It could include age backets, gender, country or language
** Do you have a preferred platform for filling out the survey in the
future?
** General feedback about the survey process?

RE: Proposal for an Emacs User Survey

[Drew Adams]

As a start, lose all of the multiple choices.
Just have open questions, letting users say
what they do/use/prefer, etc.

The multiple choices, even with an "other",
bias the results.

Ask users to express themselves, then work
with whatever info they provide for the
questions.

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

> As a start, lose all of the multiple choices.
> Just have open questions, letting users say
> what they do/use/prefer, etc.
>
> The multiple choices, even with an "other",
> bias the results.

How does it bias results exactly?
There is always a series of trade-off to make when surveying: free text
is more open ended and allows for discovery, but multiple choices have
higher completion rate and give specific answers. The set of questions
is organized such that we have:
- multiple choices regarding factual and technical practices (which
version of Emacs/OS/packages etc)
- open ended question with broader topics and free text
- meta questions about the survey itself

> Ask users to express themselves, then work
> with whatever info they provide for the
> questions.

That's the point of the more open ended questions with free text along
with the general feedback section.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> > As a start, lose all of the multiple choices.
> > Just have open questions, letting users say
> > what they do/use/prefer, etc.
> >
> > The multiple choices, even with an "other",
> > bias the results.
> 
> How does it bias results exactly?

Suggestions of what is possible.

My suggestion is to just leave the questions open,
and let people express themselves however they
think best, rather than being incited to choose
among suggested alternatives.

Just a suggestion.

> multiple choices have higher completion rate and give specific answers. 

Higher completion rate and specific answers can
mean little, compared to thoughtful responses.
Emacs has always preferred the latter.

And explicitly solicit reasons - everywhere.
Reasons can be important for guiding decisions here.

> The set of questions is organized such that we have:
> - multiple choices regarding factual and technical practices (which
> version of Emacs/OS/packages etc)

I've said what I think.  Asking about packages is
a bit different from asking about Emacs version
and OS.  The former, at least, should be an open
question, IMHO.

> - open ended question with broader topics and free text
> - meta questions about the survey itself
> 
> > Ask users to express themselves, then work
> > with whatever info they provide for the questions.
> 
> That's the point of the more open ended questions with free text along
> with the general feedback section.

Yes.  My suggestion is more of that and less of
citing multiple things that a user might consider
choosing.  Choosing OS's and Emacs versions is
less likely to be opinion-based than choosing
packages etc., and it's unlikely that someone will
come up with a new (unlisted) OS or Emacs version.
It's a different terrain, IMO - more choices, more
movement/evolution, more individual preference-based.

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

>>> As a start, lose all of the multiple choices.
>>> Just have open questions, letting users say
>>> what they do/use/prefer, etc.
>>>
>>> The multiple choices, even with an "other",
>>> bias the results.
>>
>> How does it bias results exactly?
>
> Suggestions of what is possible.
>
> My suggestion is to just leave the questions open,
> and let people express themselves however they
> think best, rather than being incited to choose
> among suggested alternatives.
>
> Just a suggestion.

I understand what you are suggesting, but I do not understand the "bias
the results" you are referring to.

> Higher completion rate and specific answers can
> mean little, compared to thoughtful responses.
> Emacs has always preferred the latter.
>
> And explicitly solicit reasons - everywhere.
> Reasons can be important for guiding decisions here.

How about having a "why" optional free-text section after each
non-factual multiple choice question? Like "which package manager do you
use? and why?" or "What are some of your favorite packages and why?"

I also could argue that if you present a paragraph input to users, only
those with the time and energy will make the effort to write a
comprehensive answer, which might create a real bias. On the other hand,
if more people can emit their preference for a particular option, you do
not know why, but you know where to start looking for why.

You do point out something very real: the survey questions are mixing
facts with opinions. We are asking factual things, along with general
opinions on the state of things. There's obviously a trade-off to make
between the two. Factual gives us quick results but without reason. Open
ended decreases the response rate and increases the analysis time, but
opens for more meaningful answers.

RE: Proposal for an Emacs User Survey

[Drew Adams]

I'm not going to argue about it.  I think I made
my suggestion clear, and I think you understand
it well enough.

Just one opinion.

Re: Proposal for an Emacs User Survey

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]

* 2020-10-09 19:12:23-04, Adrien Brochard wrote:

> Please find the list of questions I have gathered at the end of this
> email.

Great! I would like to see this survey conducted. It would be
interesting to see what people use Emacs for and some distribution of
answers. It's not easy to find good sample of users but if the survey is
published in some main communication platforms or communities it will be
good enough, I think.

Some Emacs proposals cause only talking and nobody actually does
anything. So perhaps your survey will be the best Emacs survey ever.
Best options are those that get implemented.

> ** What package do you use for error checking?
>     - Flymake
>     - Flycheck
>     - None

Some rare people use wcheck-mode which is available from GNU Elpa. I'm
the author and would like to see if it gets even 1 % of users against
the other more popular options.

https://github.com/tlikonen/wcheck-mode

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: Proposal for an Emacs User Survey

[Rasmus]

Hi Teemu,

Teemu Likonen <tlikonen@iki.fi> writes:

> Some rare people use wcheck-mode which is available from GNU 
> Elpa. I'm the author and would like to see if it gets even 1 % 
> of users against the other more popular options. 
> 
> https://github.com/tlikonen/wcheck-mode 

Aside: I used to use wheck. I think I might even have reached out 
to you many years ago.

Hunspell support (e.g. ‘ispell-hunspell-add-multi-dic’, thanks 
Eli!)  and the general functionality and speed of ‘flyspell’ has 
grown so good that I don’t need anything but vanilla Emacs.

Many thanks for creating wheck; it allowed me to get good 
spell-checking at a time when it was otherwise hard to set up!

[I see that whceck does a lot more and I wonder if it could be 
used for 
 languagetool (the grammar checker)]

Rasmus

-- 
Together we'll stand, divided we'll fall

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Teemu Likonen <tlikonen@iki.fi> [2020-10-10 11:35]:
> * 2020-10-09 19:12:23-04, Adrien Brochard wrote:
> 
> > Please find the list of questions I have gathered at the end of this
> > email.
> 
> Great! I would like to see this survey conducted. It would be
> interesting to see what people use Emacs for and some distribution of
> answers. It's not easy to find good sample of users but if the survey is
> published in some main communication platforms or communities it will be
> good enough, I think.

If survey is published, for example on Reddit, such survey is narrow
and specific to Reddit users, and would represent only that
communication channel.

My proposal on how to publish a survey is to include that in the Help
menu, and let people do it straight from Emacs. That is similar to bug
reporting, but it is not a bug, it is feature request.

Something like:

Tell us how we can improve?

in the Help menu, would lead to description as how RMS said, and from
there on, users would submit the email with their descriptions, likes,
hates, whatever they wish to say.

It could go straight to specific mailing list.

Jean

RE: Proposal for an Emacs User Survey

[Drew Adams]

> If survey is published, for example on Reddit, such survey is narrow
> and specific to Reddit users, and would represent only that
> communication channel.
> 
> My proposal on how to publish a survey is to include that in the Help
> menu, and let people do it straight from Emacs. That is similar to bug
> reporting, but it is not a bug, it is feature request.
                                  ^^^^^^^^^^^^^^^^^^^^^
> Something like:
>   Tell us how we can improve?
> in the Help menu, would lead to description as how RMS said, and from
> there on, users would submit the email with their descriptions, likes,
> hates, whatever they wish to say.
> 
> It could go straight to specific mailing list.

I agree that the `Help' menu could/should have an item
for something like this.

Actually, we already have this functionality, which is
submitting free-form feedback of any kind.  We call it
an enhancement request, and we tell users to use
`M-x report-emacs-bug' for it.  This is not as well
known as it should be, IMO.

It would be good for the `Help' menu to have an item
that explicitly calls this `Suggest Improvements' or
similar.  It would be bound to `report-emacs-bug'.
___

I think that corresponds to what you are suggesting.

However, providing individual feedback to the "bug"
list, as helpful as it might be, inviting a thread
of discussion by interested parties, including some
Emacs core developers, does not also involve or
encourage any broader discussion/communication.

That might be OK, and in fact that aspect hasn't yet
been raised in this current thread.  So far, we've
discussed only submitting a survey, getting results,
and analyzing/discussing them here, on emacs-devel.

So as far as that goes, I think this suggestion from
Jean-Louis does fit the bill.

Well, it doesn't constitute a "survey", but it does
correspond to what RMS, I, and Jean-Louis have
emphasized here: getting users to say, in their own
words, what they want and what they do.

I've said this before (dunno whether I've proposed
a separate `Help' menu item).  We should make
`report-emacs-bug' better known as a recommended
way to submit ANY feedback about Emacs.

The command name can mislead wrt this role.  A
separate `Help' menu item would help (even though
redundant with item `Send Bug Report').  Or change
that item to `Send Feedback or Bug Report'.

It might also make sense, for discoverability etc.,
to add a `send-feedback' alias for `send-bug-report'.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-11 18:15]:
> Actually, we already have this functionality, which is
> submitting free-form feedback of any kind.  We call it
> an enhancement request, and we tell users to use
> `M-x report-emacs-bug' for it.  This is not as well
> known as it should be, IMO.

Only that "report Emacs bug" does not sound as free form feedback of
any kind. I know it is, but it does not sound as enhancement
request. It takes time for person to realize that.

> It would be good for the `Help' menu to have an item
> that explicitly calls this `Suggest Improvements' or
> similar.  It would be bound to `report-emacs-bug'.

Such menu in Help could lead users to subscribe, unsubscribe, review
or send email to help-gnu-emacs mailing list, send improvement or
enhancement suggestions or anything, similar like GNU Hyperbole is
doing it since decades.

Survey in the context of the opinion poll:
https://en.wikipedia.org/wiki/Survey_(human_research)#Opinion_poll

The Help menu that would easier enable communication channel with
users is necessary.

Survey itself is not as much important as the ability or capacity in
Emacs development to analyse the data that is already there in public,
as I mentioned, bug report database is there, and there are numbers of
stars or likes on various public projects, then there are search
engines, and various comments of users on other public channels.

Subject of stars/likes and considering that users really like
something because of number of stars/likes is doubtful. Those who have
spent few  thousand dollars on advertising of their services or
products, they will know why is number of stars/likes doubtful.

Example is Spacemacs configuration, I do not know if it is
"configuration" or "text editor built on top of Emacs" as it says on
Archlinux website: https://wiki.archlinux.org/index.php/Spacemacs

What I may say about that, even if Spacemacs developers did not
advertise, they could as well advertise for it and obtain those
stars/likes and apparent online popularity. But number of stars does
not necessarily mean number of users.

Number of users publishing online that they use Spacemacs, does not
necessarily mean that is less then number of users who do not use
Spacemacs, maybe those groups are not willing to publish their
statements, or find it silly, like I do. I find it silly to argue
about theming, fonts, but maybe is that exactly what is attracting
users, some nice theme, flashy, shiny stuff.

That is up to Emacs analysis department (emacs-devel) to accept as
challenge.

> I've said this before (dunno whether I've proposed a separate `Help'
> menu item).  We should make `report-emacs-bug' better known as a
> recommended way to submit ANY feedback about Emacs.

Definitely. 

> The command name can mislead wrt this role.  A
> separate `Help' menu item would help (even though
> redundant with item `Send Bug Report').  Or change
> that item to `Send Feedback or Bug Report'.
> 
> It might also make sense, for discoverability etc.,
> to add a `send-feedback' alias for `send-bug-report'.

That is right.

> Well, it doesn't constitute a "survey", but it does correspond to
> what RMS, I, and Jean-Louis have emphasized here: getting users to
> say, in their own words, what they want and what they do.

Let me emphasize again, there is number of users who will never speak
in public or publish their opinion, why should they? (as thinking from
their view point). In my opinion that is greater number of Emacs
users. It has to be taken in consideration if then developers are
working only for those who are online louder and more exposed to talk,
or if they should be maybe working on some set of principles for
general users.

Let me give you few practical examples from:
https://www.debian.org/users/

There are listed among others, following organizations:

* Electronics Research Group, University of Aberdeen, Aberdeen, Scotland
* Department of Informatics, University of Zurich, Zurich, Switzerland
* General Students' Committee (AStA), Saarland University, Saarbrücken, Germany
* Athénée Royal de Gembloux, Gembloux, Belgium
* Computer Science, Brown University, Providence, RI, USA
* Sidney Sussex College, University of Cambridge, UK
* Centro de Comunicación Científica, Universidad de Buenos Aires, Argentina
* CEIC, Scuola Normale Superiore di Pisa, Italy
* Mexican Space Weather Service (SCiESMEX), Geophysics Institute campus Morelia (IGUM), National University of Mexico (UNAM), Mexico
* COC Araraquara, Brazil
* Department of Computer Science, Savitribai Phule Pune University, India
* Departamento de Arquitectura y Tecnología de Sistemas Informáticos (Facultad de Informática), Universidad Politécnica de Madrid, Madrid, Spain
* Department of Control Engineering, Faculty of Electrical Engineering, Czech Technical University, Czech Republic
* Swiss Federal Institute of Technology Zurich, Department of Physics, ETH Zurich, Switzerland
* Formation Canadienne, Université Libre de Guinée (UG), Conakry, Republic of Guinea
* Genomics Research Group, CRIBI - Università di Padova, Italy
* Department of Geological Sciences and Geotechnology, University of Milano-Bicocca, Italy
* Dipartimento di Geoscienze, Università degli Studi di Padova, Italy
* Nucleo Lab, Universidad Mayor de San Andrés, Bolivia
* Department of Physics, Harvard University, USA

I am to assume that among those organizations there are many people
using Emacs, people of different characteristics, not of those groups
who like theming and online popularities, so those people in those
organizations, who use Emacs, in my opinion, would be less loud online
to promote Emacs in the manner how Doom/Spacemacs or others are doing
it. Such people would be impossible to reach by publishing some online
survey anywhere, but they could be reached through Help menu.

There are various groups of people, now we have groups of people who
think that Spacemacs is editor built on top of Emacs, maybe it is
true, so it is written, but that group of people is now mentioning
"Spacemacs" and not Emacs, so they will even ask classic Emacs
questions under Spacemacs subjects online. To introduce those users to
Emacs development and helpful mailing lists, such menu Help -->
Contact us or Tell us your opinion, would be helpful.

But nevertheless Spacemacs as example of online apparent popularity,
not the real popularity. What if Spacemacs have employed advertising
to reach stars and likes? That is not hard to do.

I do not think that Emacs ever advertised as commercial companies are
doing it, it is self-advertised and promoted through OS distributions.

Real popularity could be somehow obtained if GNU/Linux distributions
and BSD and other operating system distributions and Emacs main
website would publish their statistics.

Real data useful for surveys cannot be accurately obtained by
publishing such survey online anywhere else but in Emacs itself,
including with email and with the anonymous form submission where no
personal user information will be entered. I would even state
explicitly that no IP logs will be kept, all would go to /dev/null and
systems like captcha would not be necessary, as Emacs could formulate
the buffer with the HTML rendered and viewed with eww that already
contains inside tokens that will recognize that such form is coming
from within Emacs and from nowhere else. Separate form could be made
for online public.

Summary:

- there are different groups of users, in my opinion, development is
  rather looking onto feedback of those who are more loud online, as
  those who don't communicate they cannot be reached anyway

- those groups of users who are publicly telling about their ideas
  related to Emacs, may not be, or may be, representative or majority
  group of users, but myself I doubt they are. Relying that those
  groups are representative for majority of Emacs users is doubtful as
  well. 

- it is better to develop for majority, just as by principle was done
  for years for and within Emacs, and I would say to skip all the
  surveys, and to increase the analysis and enhancement
  capacities. Data is already there as mentioned.

- if any survey is done, such survey should target to find out what
  the majority really wants, in order to bring about more Emacs
  popularity. But looking into present popular projects, one can
  easily draw conclusions without vias.

- add the button in the Help menu to contact help-gnu-emacs mailing
  list, subscribe, unsubscribe, send email anyway, or send feedback or
  send bug report. The "Send bug report" menu could be converted to
  multi menu doing that feature.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> > Actually, we already have this functionality, which is
> > submitting free-form feedback of any kind.  We call it
> > an enhancement request, and we tell users to use
> > `M-x report-emacs-bug' for it.  This is not as well
> > known as it should be, IMO.
> 
> Only that "report Emacs bug" does not sound as free form feedback of
> any kind. I know it is, but it does not sound as enhancement
> request. It takes time for person to realize that.

Agreed.  Along with the suggestion to (also) pitch
the command as a way to provide feedback of any kind,
we should work on the command's description and even
some of its behavior.

The fact that it is also for suggestions should be as
front-and-center as is the fact that it is the preferred
way to submit a bug report.

> > It would be good for the `Help' menu to have an item
> > that explicitly calls this `Suggest Improvements' or
> > similar.  It would be bound to `report-emacs-bug'.
> 
> Such menu in Help could lead users to subscribe, unsubscribe, review
> or send email to help-gnu-emacs mailing list, send improvement or
> enhancement suggestions or anything, similar like GNU Hyperbole is
> doing it since decades.

Those kinds of things are indeed also possible.
We could add a submenu of `Help' for things
related to the existing mailing lists (reading,
subscribing, submitting).

> The Help menu that would easier enable communication channel with
> users is necessary.
> 
> Survey itself is not as much important as the ability or capacity in
> Emacs development to analyse the data that is already there in public,
> as I mentioned, bug report database is there, and there are numbers of
> stars or likes on various public projects, then there are search
> engines, and various comments of users on other public channels.
> 
> Subject of stars/likes and considering that users really like
> something because of number of stars/likes is doubtful. Those who have
> spent few  thousand dollars on advertising of their services or
> products, they will know why is number of stars/likes doubtful.

I tend to agree.  Almost anything can be of some value,
but extracting real, useful value from some things can be
unobvious, difficult, or otherwise not worth the trial.

> > I've said this before (dunno whether I've proposed a separate `Help'
> > menu item).  We should make `report-emacs-bug' better known as a
> > recommended way to submit ANY feedback about Emacs.
> 
> Definitely.
> 
> > The command name can mislead wrt this role.  A
> > separate `Help' menu item would help (even though
> > redundant with item `Send Bug Report').  Or change
> > that item to `Send Feedback or Bug Report'.
> >
> > It might also make sense, for discoverability etc.,
> > to add a `send-feedback' alias for `send-bug-report'.
> 
> That is right.
> 
> > Well, it doesn't constitute a "survey", but it does correspond to
> > what RMS, I, and Jean-Louis have emphasized here: getting users to
> > say, in their own words, what they want and what they do.
> 
> Let me emphasize again, there is number of users who will never speak
> in public or publish their opinion, why should they? (as thinking from
> their view point). In my opinion that is greater number of Emacs
> users. It has to be taken in consideration if then developers are
> working only for those who are online louder and more exposed to talk,
> or if they should be maybe working on some set of principles for
> general users.

Of course.  For active input we're pretty much limited
to voluntary contribution.  That can be encouraged, but
we can be sure, at most, only that it represents the
views of the volunteers.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-11 22:48]:
> Those kinds of things are indeed also possible.  We could add a
> submenu of `Help' for things related to the existing mailing lists
> (reading, subscribing, submitting).

Please install Hyperbole and see how they done that.

> > > Well, it doesn't constitute a "survey", but it does correspond to
> > > what RMS, I, and Jean-Louis have emphasized here: getting users to
> > > say, in their own words, what they want and what they do.
> > 
> > Let me emphasize again, there is number of users who will never speak
> > in public or publish their opinion, why should they? (as thinking from
> > their view point). In my opinion that is greater number of Emacs
> > users. It has to be taken in consideration if then developers are
> > working only for those who are online louder and more exposed to talk,
> > or if they should be maybe working on some set of principles for
> > general users.
> 
> Of course.  For active input we're pretty much limited to voluntary
> contribution.  That can be encouraged, but we can be sure, at most,
> only that it represents the views of the volunteers.

Those who don't want to tell anything, should speak out louder...

How to request changes in Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Only that "report Emacs bug" does not sound as free form feedback of
  > any kind. I know it is, but it does not sound as enhancement
  > request. It takes time for person to realize that.

That is a good point.  Do people have concrete suggestions for
how to make it clearer how to request changes etc?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: How to request changes in Emacs

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-13 06:50]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > Only that "report Emacs bug" does not sound as free form feedback of
>   > any kind. I know it is, but it does not sound as enhancement
>   > request. It takes time for person to realize that.
> 
> That is a good point.  Do people have concrete suggestions for
> how to make it clearer how to request changes etc?

We discussed that in this same thread, concrete proposal would be to
expand the menu item Help -> Send Bug Report

The item "Send Bug Report" could be placeholder for something like
"Tell us how to improve" which would expand in more submenu options,
something like:

- Send Bug Report

- Suggest Emacs improvements

- Send email to Help GNU Emacs

- Join the Help GNU Emacs mailing list

- Leave Help GNU Emacs mailing list

The item "Suggest Emacs improvements" would be similar like
report-emacs-bug, with different wordings, as user is not speaking
about the bug, email could be sent to mailing list. It would be free
form, and it could also include various information of the system
similar as report-emacs-bug

GNU Hyperbole has similar features built in.

Drew Adams could give maybe better references to emails in this
thread, you could review the thread by the subject How to make Emacs
popular again.

Re: How to request changes in Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The item "Send Bug Report" could be placeholder for something like
  > "Tell us how to improve" which would expand in more submenu options,
  > something like:

I guess so.  Perhaps it should be renamed to "Give Feedback."

I always find submenus inconvenient to select from; am I the only
one?  Is it because I rarely do that?

  > - Send Bug Report

  > - Suggest Emacs improvements

  > - Send email to Help GNU Emacs

I suggest

  Mail a question to help-gnu-emacs.

  > - Join the Help GNU Emacs mailing list

  > - Leave Help GNU Emacs mailing list

I think the last two are not needed.  If a user finds out about
help-gnu-emacs, perse can decide to join.  Also, that command could
display, in some way, guidance on how to subscribe or unsubscribe.

Having fewer options could permit a more convenient interface for
choosing an option.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: How to request changes in Emacs

[Drew Adams]

>> - Send Bug Report
>> - Suggest Emacs improvements

I suggested it's also possible to combine those two:

  A separate `Help' menu item would help (even though
  redundant with item `Send Bug Report').  Or change
  that item to `Send Feedback or Bug Report'.

RE: How to request changes in Emacs

[Drew Adams]

>   > Only that "report Emacs bug" does not sound as free form feedback of
>   > any kind. I know it is, but it does not sound as enhancement
>   > request. It takes time for person to realize that.
> 
> That is a good point.  Do people have concrete suggestions for
> how to make it clearer how to request changes etc?

There have been concrete suggestions for this.
For instance:

https://lists.gnu.org/archive/html/emacs-devel/2020-10/msg00552.html

The suggestions there include:

1. Alias `report-emacs-bug' to a name that suggests
   sending feedback in general, not just reporting a bug.

2. Change the description of that command, to reflect
   the fact that it is for _both_ sending general feedback
   and reporting bugs.  And change the descriptions in
   manuals accordingly.

3. Changing the Help menu item for reporting a but,
   so it suggests either reporting a bug or sending
   general feedback.

Re: How to request changes in Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

emacs-feedback could be a good name, but perhaps it should not insert
into the message some of the things that report-emacs-bug inserts.
They are relevant if the subject is a bug that just occurred, but not
in other circumstances.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: How to request changes in Emacs

[Drew Adams]

> emacs-feedback could be a good name, but perhaps it should not insert
> into the message some of the things that report-emacs-bug inserts.
> They are relevant if the subject is a bug that just occurred, but not
> in other circumstances.

Agreed.  They are not necessarily relevant, not all anyway.
___

[I suggested, in 2013 and 2016, that we modularize the
possible info to automatically gather and send, and let
users choose which such pieces they want to send.

That modularization would also make it clearer & easier to
handle a new command for sending feedback a bit differently
from `report-emacs-bug', in terms of what to include.

https://lists.gnu.org/archive/html/emacs-devel/2016-05/msg00238.html

https://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00431.html]

RE: Proposal for an Emacs User Survey

[Drew Adams]

> I've said this before....  We should make
> `report-emacs-bug' better known as a recommended
> way to submit ANY feedback about Emacs.
...
> It might also make sense, for discoverability etc.,
> to add a `send-feedback' alias for `send-bug-report'.

I'll mention also that I often suggest to users
(e.g. on SE.emacs or Reddit) that they use `M-x
report-emacs-bug' to send an enhancement request.

And pretty much each time I feel obliged to add
something like "(that's for enhancement reports,
as well as bug reports)" so they aren't scared
away by the command name, thinking that it's
something too serious or that they might get
criticized for not submitting a report of a
proper bug.

An alias that emphasizes general feedback to
improve Emacs would be good, I think.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-11 23:20]:
> > I've said this before....  We should make
> > `report-emacs-bug' better known as a recommended
> > way to submit ANY feedback about Emacs.
> ...
> > It might also make sense, for discoverability etc.,
> > to add a `send-feedback' alias for `send-bug-report'.
> 
> I'll mention also that I often suggest to users
> (e.g. on SE.emacs or Reddit) that they use `M-x
> report-emacs-bug' to send an enhancement request.
> 
> And pretty much each time I feel obliged to add
> something like "(that's for enhancement reports,
> as well as bug reports)" so they aren't scared
> away by the command name, thinking that it's
> something too serious or that they might get
> criticized for not submitting a report of a
> proper bug.
> 
> An alias that emphasizes general feedback to
> improve Emacs would be good, I think.

You are putting much energy into that.

Then, did you review the bug database, how many enhancements really
arrived?

Now, speaking of Emacs improvements, those popular configurations are
already showing what some users wish and want, they also do good work
in attracting new users to Emacs.

Speaking of MELPA, that is also something users want, so I hope we
start somehow doing elpa.nongnu.org where such packages can be
invited, so I understood, polished and distributed.

So, all those popular configurations could be then forked into
elpa.nongnu.org including thousands of other packages, after careful
review.

That is for me the summary of this talk, there is need for a shelf
with new shiny shoes.

Re: Proposal for an Emacs User Survey

[Philip K.]

Jean Louis <bugs@gnu.support> writes:

> * Teemu Likonen <tlikonen@iki.fi> [2020-10-10 11:35]:
>> * 2020-10-09 19:12:23-04, Adrien Brochard wrote:
>> 
>> > Please find the list of questions I have gathered at the end of this
>> > email.
>> 
>> Great! I would like to see this survey conducted. It would be
>> interesting to see what people use Emacs for and some distribution of
>> answers. It's not easy to find good sample of users but if the survey is
>> published in some main communication platforms or communities it will be
>> good enough, I think.
>
> If survey is published, for example on Reddit, such survey is narrow
> and specific to Reddit users, and would represent only that
> communication channel.
>
> My proposal on how to publish a survey is to include that in the Help
> menu, and let people do it straight from Emacs. That is similar to bug
> reporting, but it is not a bug, it is feature request.

But if you have to have mail configured, you're also only going to have
a specific subset of all users.

I think the idea of having a survey that could be filled out from Emacs
is interesting (although we would only get to hear from current users,
not previous users). But it would have to work regardless of what the
user has configured or not. From what I see, that would either mean a
web form that can be filled out using EWW or a package that could be
downloaded from ELPA.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Philip K. <philipk@posteo.net> [2020-10-11 21:33]:
> I think the idea of having a survey that could be filled out from Emacs
> is interesting (although we would only get to hear from current users,
> not previous users). But it would have to work regardless of what the
> user has configured or not. From what I see, that would either mean a
> web form that can be filled out using EWW or a package that could be
> downloaded from ELPA.

To reach previous users, we may ask Vim and VSCode developers to
include the Emacs survey option.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > To reach previous users, we may ask Vim and VSCode developers to
  > include the Emacs survey option.

It could be useful to ask those users a different survey.  However,
the Microsoft management would surely not cooperate; I am not going to
ask them.

We could reach plenty of former Emacs users in other ways.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Proposal for an Emacs User Survey

[Drew Adams]

> > My proposal on how to publish a survey is to include that in the Help
> > menu, and let people do it straight from Emacs. That is similar to bug
> > reporting, but it is not a bug, it is feature request.
> 
> But if you have to have mail configured, you're also only going to have
> a specific subset of all users.

Nothing _limits_ input to email.  That's all we
handle today, and we should at least continue to
handle such input.  But if we want to entertain
supporting other input methods (smoke signals...
whatever), that's not impossible.

> I think the idea of having a survey that could be filled out from Emacs
> is interesting (although we would only get to hear from current users,
> not previous users). But it would have to work regardless of what the
> user has configured or not. From what I see, that would either mean a
> web form that can be filled out using EWW or a package that could be
> downloaded from ELPA.

It's not only either/or.

The point of the suggestion was to leverage the
existing `report-emacs-bug' input of general,
free-form feedback.  Where "leverage" also means
encourage.  And that in turn means new names
(command alias, menu items,... whatever) and
some new description (e.g. rework the wording of
the existing bug-reporting instructions).

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-11 22:48]:
> > > My proposal on how to publish a survey is to include that in the Help
> > > menu, and let people do it straight from Emacs. That is similar to bug
> > > reporting, but it is not a bug, it is feature request.
> > 
> > But if you have to have mail configured, you're also only going to have
> > a specific subset of all users.
> 
> Nothing _limits_ input to email.  That's all we handle today, and we
> should at least continue to handle such input.  But if we want to
> entertain supporting other input methods (smoke signals...
> whatever), that's not impossible.

Is hard for me to understand the above.

Let me say that trends changed, there are now more people using
Internet, and there are more people using Internet without email or
not knowing what email does, or having email, but never using it and
not knowing what it does, including people having email and using it
exclusively on their mobile devices, not knowing they can use it from
computers.

Suspicious is also Windows and other systems but Unix-like systems,
I did not try, I just don't know how would report-emacs-bug function
on Windows, I doubt it is well integrated.

I would just make it as a form, either as Emacs forms from forms.el
library, plus some POST to URL, or as HTML form by using eww.

In such form, user could enter email, but if user does not have email,
need not enter such, neither the name. User could still communicate,
it would be one way, but better some opinions then none, hey.

> > I think the idea of having a survey that could be filled out from Emacs
> > is interesting (although we would only get to hear from current users,
> > not previous users). But it would have to work regardless of what the
> > user has configured or not. From what I see, that would either mean a
> > web form that can be filled out using EWW or a package that could be
> > downloaded from ELPA.
> 
> It's not only either/or.
> 
> The point of the suggestion was to leverage the existing
> `report-emacs-bug' input of general, free-form feedback.  Where
> "leverage" also means encourage.  And that in turn means new names
> (command alias, menu items,... whatever) and some new description
> (e.g. rework the wording of the existing bug-reporting
> instructions).

Not sure, I would leave report-emacs-bug in place, and add feedback or
suggestions with different wording, including access to friendly
mailing lists and IRC channel.

So far fastest way to get live help for Emacs is IRC.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> > > But if you have to have mail configured, you're also only going to have
> > > a specific subset of all users.
> >
> > Nothing _limits_ input to email.  That's all we handle today, and we
> > should at least continue to handle such input.  But if we want to
> > entertain supporting other input methods (smoke signals...
> > whatever), that's not impossible.
> 
> Is hard for me to understand the above.

It's hard for me to understand what you write, below.

My point there was only that nothing limits acceptance
of user input only in the form of email.  That's all.

To be clear, I'm in favor of continuing to accept
input by email.  I'm strongly in favor of that.
And in particular by `report-emacs-bug'.

I was just saying that nothing prevents accepting
feedback in other ways.  Of course, some ways would
no doubt require some support development.  (Something
like snail mail would not.)

> Let me say that trends changed, there are now more people using
> Internet, and there are more people using Internet without email or
> not knowing what email does, or having email, but never using it and
> not knowing what it does, including people having email and using it
> exclusively on their mobile devices, not knowing they can use it from
> computers.
> 
> Suspicious is also Windows and other systems but Unix-like systems,
> I did not try, I just don't know how would report-emacs-bug function
> on Windows, I doubt it is well integrated.
> 
> I would just make it as a form, either as Emacs forms from forms.el
> library, plus some POST to URL, or as HTML form by using eww.
> 
> In such form, user could enter email, but if user does not have email,
> need not enter such, neither the name. User could still communicate,
> it would be one way, but better some opinions then none, hey.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-12 02:25]:
> > > > But if you have to have mail configured, you're also only going to have
> > > > a specific subset of all users.
> > >
> > > Nothing _limits_ input to email.  That's all we handle today, and we
> > > should at least continue to handle such input.  But if we want to
> > > entertain supporting other input methods (smoke signals...
> > > whatever), that's not impossible.
> > 
> > Is hard for me to understand the above.
> 
> It's hard for me to understand what you write, below.
> 
> My point there was only that nothing limits acceptance
> of user input only in the form of email.  That's all.
> 
> To be clear, I'm in favor of continuing to accept
> input by email.  I'm strongly in favor of that.
> And in particular by `report-emacs-bug'.

Yes, I have confirmed that.

Additionally for users who did not set up email system, I said that
form can be used to submit information, which in turn is again
converted to email. A form submission can be anonymous, or with email
address again, but email system need not be set.

I am not using Windows, but there I think on Windows mostly. I doubt
GNU/Linux systems all have set up email transport, but they would
rather have WWW access. Is it now clearer?

RE: Proposal for an Emacs User Survey

[Drew Adams]

> Additionally for users who did not set up email system,

Most users have an email client, and they can use
that client for this.  But it is a valid point that
some users, who can use email, might nevertheless
not want to, or might prefer another way of sending
feedback.  We can accept feedback in more than one way.

> I said that
> form can be used to submit information, which in turn is again
> converted to email. A form submission can be anonymous, or with email
> address again, but email system need not be set.

Email system need not be set, to send email feedback.
Depending on what you might mean by "email system"
and "set".  It's enough to have a mail client.

> I am not using Windows, but there I think on Windows mostly. I doubt
> GNU/Linux systems all have set up email transport, but they would
> rather have WWW access. Is it now clearer?

I take the point that not everyone will have email
set up on the computer they use for Emacs.  OK.

And some users might prefer not to use email to
provide Emacs feedback.

Other ways to do that can be accepted/supported.
I have nothing specific to say about any such
other ways.
___

On another major question in this thread: I'm in
favor of emphasizing solicitation and consideration
of free-form feedback, over multiple-choice votes
(check-boxes, radio buttons).

And I'm in favor of asking that those submitting
suggestions to give their reasons, for better
understanding.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-12 20:37]:
> > Additionally for users who did not set up email system,
> 
> Most users have an email client, and they can use that client for
> this.  But it is a valid point that some users, who can use email,
> might nevertheless not want to, or might prefer another way of
> sending feedback.  We can accept feedback in more than one way.

Just thinking of those Windows users and GNU/Linux and BSD users who
would not have the Emacs set for sending email, what if they just use
smartphones to send emails? There are different groups of users today,
is not same as before 20 years when email was so much popular. There
are users using almost exclusively chat systems, disregarding email.

I wonder if users on Windows or GNU/Linux who do use email on their
system would be properly served with the option 'mail-client'.

They would face 3 options, mail-client, transport, smtp, right?

I have just tried it on my Hyperbola GNU/Linux-libre where I have set
mutt as my mail client, so what happened is following:

- I have chosen 'mail client'

- epiphany browser opened, I do not know why exactly

- thereafter xterm opened, with mutt

- mutt asked me pre-defined emails as I specified it in the Emacs M-x
  mail, I guess it would be same as report-emacs-bug

- I found my double signature inside of the new email, and when I
  wanted to send email, my from email was changed to other email
  address

- instead of having Unix style line endings, I could see ^M when
  editing that new email from mutt, very ugly

Overall, on GNU/Linux I am not satisfied with mail-client option, it
can be my own bad configuration.

And I wonder how well would that work on side of other people using
various mail clients, and if at all it would work.

I could see following in the new email that was invoked through mail
client option, those annoying ^M

--Text follows this line--^M
^M
^M
-- ^M
Thanks,^M



Jean

RE: Proposal for an Emacs User Survey

[Drew Adams]

> Just thinking of those Windows users and GNU/Linux and BSD users who
> would not have the Emacs set for sending email, what if they just use
> smartphones to send emails? There are different groups of users today,
> is not same as before 20 years when email was so much popular. There
> are users using almost exclusively chat systems, disregarding email.
> 
> I wonder if users on Windows or GNU/Linux who do use email on their
> system would be properly served with the option 'mail-client'.

I can't speak to all that you say/ask.  I can say this:

I use Emacs on MS Windows, and I use MS Outlook as
my mail client.  And I have no problem using it with
`report-emacs-bug'.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Philip K. <philipk@posteo.net> [2020-10-11 21:34]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Teemu Likonen <tlikonen@iki.fi> [2020-10-10 11:35]:
> >> * 2020-10-09 19:12:23-04, Adrien Brochard wrote:
> >> 
> >> > Please find the list of questions I have gathered at the end of this
> >> > email.
> >> 
> >> Great! I would like to see this survey conducted. It would be
> >> interesting to see what people use Emacs for and some distribution of
> >> answers. It's not easy to find good sample of users but if the survey is
> >> published in some main communication platforms or communities it will be
> >> good enough, I think.
> >
> > If survey is published, for example on Reddit, such survey is narrow
> > and specific to Reddit users, and would represent only that
> > communication channel.
> >
> > My proposal on how to publish a survey is to include that in the Help
> > menu, and let people do it straight from Emacs. That is similar to bug
> > reporting, but it is not a bug, it is feature request.
> 
> But if you have to have mail configured, you're also only going to have
> a specific subset of all users.
> 
> I think the idea of having a survey that could be filled out from Emacs
> is interesting (although we would only get to hear from current users,
> not previous users). But it would have to work regardless of what the
> user has configured or not. From what I see, that would either mean a
> web form that can be filled out using EWW or a package that could be
> downloaded from ELPA.

If the eww's eww-submit function get dissected, maybe there will be
simplest POST function, then library forms.el can be used to POST the
information to GNU servers, which in turn arrives by email
easily. Even the CGI script can be written in Emacs Lisp to accept the
email and send into the queue for approval before publishing it to
mailing list.

That would not need to load full eww, it can display users few fields,
if one wish to enter some personal information or not, it would be
equal.

Form would send it to server.

Server side Emacs CGI would send email to queue mailman or some
mailing list manager for approval or censorship :-)

Emails would be sent from form-submission@gnu.org to mailing list with
Reply-To: email address of user who submitted the form. But what if
email address is not working?!

Other users or developers could collaborate or answer questions.

See: share/emacs/28.0.50/etc/forms

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you!
I added wcheck-mode to the list.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Hi,

To me it looks like a pretty solid list of questions. I'd just suggest
a few changes (assuming we go with multiple-choice):


> ** How do you run Emacs?
>      - Client/Server mode only
>      - Standalone application only
>      - Both

I would add a "I don't know" item here.

> ** Describe your configuration
>     - I am using vanilla emacs (little to no configuration).
>     - Custom configuration
>     - Spacemacs - https://github.com/syl20bnr/spacemacs
>     - Doom Emacs - https://github.com/hlissner/doom-emacs
>     - Prelude - https://github.com/bbatsov/prelude
>     - purcell emacs.d - https://github.com/purcell/emacs.d
>     - magnars emacs.d - https://github.com/magnars/.emacs.d
>     - Emacs Starter Kit - https://github.com/eschulte/emacs-starter-kit
>     - oh-my-emacs - https://github.com/xiaohanyu/oh-my-emacs
>     - Better Defaults - https://github.com/technomancy/better-defaults
>     - Graphene - https://github.com/rdallasgray/graphene
>     - ohai-emacs - https://github.com/bodil/ohai-emacs
>     - ergoemacs-mode - https://github.com/ergoemacs/ergoemacs-mode
>     - Rho Emacs - https://github.com/GChristensen/rho-emacs
>     - Radian - https://github.com/raxod502/radian
>     - Centaur Emacs - https://github.com/seagle0128/.emacs.d
>     - Other

This list is significantly longer than others in the questionnaire.
Would it make sense to group the less common ones in "other"?

I also don't know if I would include better-defaults, ergoemacs, etc,
in that list: they are standard packages, they don't require replacing
your .emacs.d.

> ** What keybindings do you use?
>     -  Emacs defaults
>     -  Evil/Spacemacs/Doom-Emacs (all the vim-likes)
>     -  CUA-mode
>     -  God-mode
>     -  Boon
>     -  Xah-Fly-Keys
>     -  Custom modal (ryo-modal, etc)
>     -  Custom modifiers (Emacs from scratch)
>     -  Other

Same doubt regarding the less common options.

Also, I'd add Viper (built-in) to the vim-likes.

>
> ** Prior to using Emacs what was your primary editor?
>     - VIM
>     - VScode
>     - Eclipse
>     - Notepad++
>     - Sublime
>     - Intelij
>     - Other

I'd add an entry "Text editor supplied by the OS (notepad, gedit,
kate...)" and an entry "None".

> ** Do you use shell/terminal emulator in Emacs?
>      - eshell
>      - shell
>      - term
>      - ansi-term
>      - do not use.

I'd add "vterm" and "other".

>
> ** Do you use mail client in Emacs?
>      - Mu4e
>      - Gnus
>      - Mut
>      - notmuch
>      - do not use

Here too, "other" should be an option.

Nice work !

Re: Proposal for an Emacs User Survey

[Philip K.]

Thibaut Verron <thibaut.verron@gmail.com> writes:

>> ** Do you use mail client in Emacs?
>>      - Mu4e
>>      - Gnus
>>      - Mut
>>      - notmuch
>>      - do not use
>
> Here too, "other" should be an option.

The list is incomplete, but are there really that many mail clients that
they couldn't be listed. Emacswiki[0] lists:

- Rmail
- Gnus
- MH-E
- Wanderlust
- Mew
- VM
- Notmuch
- mu4e
- mutt

and a few more unmaintained ones. I guess if other could stand for those?

[0]  https://www.emacswiki.org/emacs/CategoryMail

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you! Added!

On 10/10/20 9:50 AM, Philip K. wrote:
> Thibaut Verron <thibaut.verron@gmail.com> writes:
>
>>> ** Do you use mail client in Emacs?
>>>       - Mu4e
>>>       - Gnus
>>>       - Mut
>>>       - notmuch
>>>       - do not use
>>
>> Here too, "other" should be an option.
>
> The list is incomplete, but are there really that many mail clients that
> they couldn't be listed. Emacswiki[0] lists:
>
> - Rmail
> - Gnus
> - MH-E
> - Wanderlust
> - Mew
> - VM
> - Notmuch
> - mu4e
> - mutt
>
> and a few more unmaintained ones. I guess if other could stand for those?
>
> [0]  https://www.emacswiki.org/emacs/CategoryMail
>

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-10 13:56]:

> I would add a "I don't know" item here.
> 
> > ** Describe your configuration
> >     - I am using vanilla emacs (little to no configuration).
> >     - Custom configuration
> >     - Spacemacs - https://github.com/syl20bnr/spacemacs
> >     - Doom Emacs - https://github.com/hlissner/doom-emacs
> >     - Prelude - https://github.com/bbatsov/prelude
> >     - purcell emacs.d - https://github.com/purcell/emacs.d
> >     - magnars emacs.d - https://github.com/magnars/.emacs.d
> >     - Emacs Starter Kit - https://github.com/eschulte/emacs-starter-kit
> >     - oh-my-emacs - https://github.com/xiaohanyu/oh-my-emacs
> >     - Better Defaults - https://github.com/technomancy/better-defaults
> >     - Graphene - https://github.com/rdallasgray/graphene
> >     - ohai-emacs - https://github.com/bodil/ohai-emacs
> >     - ergoemacs-mode - https://github.com/ergoemacs/ergoemacs-mode
> >     - Rho Emacs - https://github.com/GChristensen/rho-emacs
> >     - Radian - https://github.com/raxod502/radian
> >     - Centaur Emacs - https://github.com/seagle0128/.emacs.d
> >     - Other

The Emacs bug report system already include packages employed by
various users, so the survey data of who uses which package, on which
system, which Emacs version, is already there in the bug system.

It needs only be evaluated, Emacs Lisp could crawl through the bug
report and find out which packages are used mostly, which
configurations, themes are used mostly by those who report bugs.

Help menu -> Tell us how to improve Emacs, could automatically ask
user to send whatever list of packages user is using, of course by
asking user for permission, in similar way how that does the function
{M-x report-emacs-bug}

I am against advertising of the Microsoft® Github® on any official
Emacs communication lines.

I find the above Microsoft Github list biased for the reason that the
list have been made in specific order by somebody and I do not see why
should Centaur be last and Spacemacs first.

Additionally, whoever made the list, already have got it by
popularity, in fact, there is nothing to be asked, as Github is asking
users to LIKE the project, there are "stars" given by users, so if you
wish to know which Github project is more popular, use the following
search:

https://github.com/search?o=desc&q=emacs&s=stars&type=Repositories

The evaluation is right there, as it is sorted by stars given by its
users. Thus the above list is absolutely not necessary.

Emacs, as free software, in general should not be pointing users to
Github for reasons of non-free Javascript.

Jean

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you! I've added your suggestions to the survey.

Re: Proposal for an Emacs User Survey

[Rasmus]

Hi Adrien,

Thank you for putting together the questions.

I think they are some good questions there, although I am not 
quite sure what exactly the “research question” is.  But I guess 
it’s just understanding users broadly...

Adrien Brochard <abrochard@gmx.com> writes:

> ** How would you characterize your use of Emacs? 
>    - Use it for work - I use it for serious "hobby" projects - 
>    I'm just tinkering - I use it for my studies - Other 

I would’t want to limit my choice to one here.  I use Emacs for 
work (when I can) and for hobby projects.

IOW: Multiple choices would be nice.
 
> ** What do you use Emacs for? 
>     - Software development - Research writing - Data science - 
>     Writing - Other 

As above.
 
> ** What OS do you primary use? 
>     - Linux - Windows - MacOS - BSD - Other 

As above.
 
> ** Describe your configuration 
>    - I am using vanilla emacs (little to no configuration).  - 
>    Custom configuration - Spacemacs - 
>    https://github.com/syl20bnr/spacemacs - Doom Emacs - 
>    https://github.com/hlissner/doom-emacs - Prelude - 
>    https://github.com/bbatsov/prelude - purcell emacs.d - 
>    https://github.com/purcell/emacs.d - magnars emacs.d - 
>    https://github.com/magnars/.emacs.d - Emacs Starter Kit - 
>    https://github.com/eschulte/emacs-starter-kit - oh-my-emacs - 
>    https://github.com/xiaohanyu/oh-my-emacs - Better Defaults - 
>    https://github.com/technomancy/better-defaults - Graphene - 
>    https://github.com/rdallasgray/graphene - ohai-emacs - 
>    https://github.com/bodil/ohai-emacs - ergoemacs-mode - 
>    https://github.com/ergoemacs/ergoemacs-mode - Rho Emacs - 
>    https://github.com/GChristensen/rho-emacs - Radian - 
>    https://github.com/raxod502/radian - Centaur Emacs - 
>    https://github.com/seagle0128/.emacs.d - Other 

Well done finding all of those.  I haven’t heard of most of those.
 
> ** What keybindings do you use? 
>    -  Emacs defaults -  Evil/Spacemacs/Doom-Emacs (all the 
>    vim-likes) -  CUA-mode -  God-mode -  Boon -  Xah-Fly-Keys - 
>    Custom modal (ryo-modal, etc) -  Custom modifiers (Emacs from 
>    scratch) -  Other 

It may also be interesting to ask which keybinding you initially 
used.
I would guess a lot of people started with CUA and moved to 
vanilla keybindings.
 
> ** Prior to using Emacs what was your primary editor? 
>    - VIM - VScode - Eclipse - Notepad++ - Sublime - Intelij - 
>    Other 

None/don’t know? (I used TeXnicCenter but it is hardly an editor).
 
> ** How do you manage third-party elisp? 
>     - built-in package.el - Spacemacs does it for me - 
>     straight.el - borg - leaf - el-get - Nix - git submodules 
>     without Borg - git subtrees - git clones - guix - quelpa - 
>     cask - No third-party deps - other 

Again, I’m impressed that you put together such a long list!
 
> ** How do you get emacs packages(if applicable)? 
>     - No repos - Gnu Elpa - Melpa/Melpa Stable - Directly from 
>     the source (e. g. using straight). 

GNU ELPA is configured by default so is no repos necessary?
 
> ** Can you list some of your favorite package? 
> 
> ** What package do you use for error checking? 
>    - Flymake - Flycheck - None 
> 
> ** Do you use TRAMP? 
> 
> ** DO you use Magit? 

What is the purpose of this question?  There are many VC programs 
and Magit can only be used for git.  So if we care about “do you 
control VC from Emacs” then the question might not be precise 
enough. 

> ** Do you use shell/terminal emulator in Emacs? 
>     - eshell - shell - term - ansi-term - do not use. 

“None” for consistency?
 
> ** Do you use mail client in Emacs? 
>     - Mu4e - Gnus - Mut - notmuch - do not use 


“mu4e” and “Mutt” I guess?
And “None” for consistency.
 
> ** What is your Elisp proficiency? 
>     - Beginner/No knowledge - Basic elisp understanding - 
>     Intermediate - Advanced - Expert 

I would find it very hard to decide here.  Could something like 
this be split into more easily quantifiable choices? Like do you 
know how to use ‘defmacro’?
 
> ** do you use a language server with lsp-mode? With what 
> languages? 

Maybe mention eglot as well?
 
> ** Have you ever contributed to GNU Emacs core/Elpa? 
>     - No - No, but I would do that if the process is 
>     changed(e. g. using 
> github pull requests instead of the mailing list, no papers, 
> etc**. 
>     - I do PRs from time to time - I provide PRs regularly - I 
>     am active contributor/maintainer 

Nitpick: would “patches” be more well-understood than PR?
 
> ** Have you ever contributed to Melpa package? 
>     - No - I do PRs from time to time - I provide PRs regularly 
>     - I am a package maintainer 

Nitpick: I guess you mean something more broad than “melpa” here? 
I guess I should understand it as “non-core packages” or “packages 
not shipped with Emacs” (that’s also not entirely correct as it 
may cover ELPA).
 
> ** What Emacs community forums have you visited in the past 
> year?  Answers would be things like 
>     - r/emacs - Emacs mailing list - irc - Emacs meetups - I 
>     follow twitter Emacs related accounts - Other 


Maybe include the Emacs stack exchange or does that not qualify as 
a forum?

    https://emacs.stackexchange.com/

It would also be good to have a banner there once the survey is 
“up”.


> ** If there is another survey in 2021, would you be opposed to 
> it containing optional & general demographics questions? 
>    It could include age backets, gender, country or language

Can’t you just put those in and make them optional?

Kind regards,
Rasmus

-- 
Human: An animal that complicates things more than strictly 
necessary

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you!
I've added your suggestions in. For the lists, you can thank the Reddit
community, they did all the hard work.

>> ** What is your Elisp proficiency?     - Beginner/No knowledge - Basic
>> elisp understanding -     Intermediate - Advanced - Expert
>
> I would find it very hard to decide here.  Could something like this be
> split into more easily quantifiable choices? Like do you know how to use
> ‘defmacro’?

** What is your Elisp proficiency?
     - No knowledge
     - I copy paste some code here and there, mostly for my configuration
     - I can write my own simple functions
     - I can write my own packages

What about these levels?


>> ** If there is another survey in 2021, would you be opposed to it
>> containing optional & general demographics questions?    It could
>> include age backets, gender, country or language
>
> Can’t you just put those in and make them optional?

Some people were vocal about their preference for the survey not to
include demographic questions, even optional, as it was inviting trouble.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Since this is a general survey rather than a focused inquiry, some of
my plans don't apply.  A multiple choice answer-taker would
be usable in principel for these questions.  But it needs to
meet our ethical criteria -- not sending nonfree JS code to the
browser.

Also, each question should offer a way to answer "none of the above"
or "I'm a strange case".

Your questions seem pretty good overall for a survey.  Here are a few specific
critiques.

  > ** What OS do you primary use?
  >      - Linux

Linux is a kernel, not an operating system.  So it should say,
"GNU/Linux."  See https://gnu.org/gnu/linux-and-gnu.html and
https://gnu.org/gnu/gnu-linux-faq.html, plus the history in
https://gnu.org/gnu/the-gnu-project.html.

  > ** How do you use Emacs?
  >      - GUI
  >      - Terminal (TUI)

Some people will not know those terms, so let's use names that anyone
will understand.

  > ** Which completion/selection framework do you use?

Is it correct to assume a person uses only one?

  > ** Have you ever contributed to Melpa package?

If it mentions Melpa, it must explain the ethical reason we cannot
recommend anything about Melpa.

I would rather not mention that name at all.

  > ** can you recall any difficulties you faced initially learning Emacs?

Add, "Please be as specific and concrete as your memories permit."

  > ** If there is another survey in 2021, would you be opposed to it
  > containing optional & general demographics questions?
  >     It could include age backets, gender, country or language

Let's not ask that.  We should not invite people to make difficulties
for us.

  > ** Do you have a preferred platform for filling out the survey in the
  > future?

That question would be useless -- people would recommend unjust
platforms based on practical convenience.

Let's add a short test: we could offer multiple choice answers for
each of these.  For some questions the user should be able to select
multiple answers, and should select all that are valid.

  > What is GNU?

  > What is GNU/Linux?

  > How does GNU Emacs related to GNU?

  > What is free software (libre software)?

  > Which of these are free liceenses?

  > How does GNU relate to free software?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 1462 bytes --]

> If it mentions Melpa, it must explain the ethical reason we cannot
> recommend anything about Melpa.
> 
> I would rather not mention that name at all.

Not mentioning MELPA as an option because you feel uncomfortable about
it will distort survey results, especially for the part where you ask
what package archives people use.  Worse, it leads to a self-reinforcing
effect:  If the true extent of people using MELPA is unknown,
emacs-devel will continue making choices on the false premise that MELPA
isn't a thing at all.  In the worst case GNU ELPA will be eclipsed by
MELPA and cease to be of any practical concern (and it sure seems to me
that this is where things are heading currently).  Remember, this is a
survey to find out how Emacs users use their editor, not about how much
they adhere to FSF ideology.

> Let's add a short test: we could offer multiple choice answers for
> each of these.  For some questions the user should be able to select
> multiple answers, and should select all that are valid.
> 
>   > What is GNU?
> 
>   > What is GNU/Linux?
> 
>   > How does GNU Emacs related to GNU?
> 
>   > What is free software (libre software)?
> 
>   > Which of these are free liceenses?
> 
>   > How does GNU relate to free software?

See my last sentence above.  What's the point of this quiz?  Finding out
what percentage of Emacs users fully believes in GNU's goals?  What will
that information be used for?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Vasilij Schneidermann <mail@vasilij.de> [2020-10-11 10:38]:
> > If it mentions Melpa, it must explain the ethical reason we cannot
> > recommend anything about Melpa.
> > 
> > I would rather not mention that name at all.
> 
> Not mentioning MELPA as an option because you feel uncomfortable about
> it will distort survey results, especially for the part where you ask
> what package archives people use.  Worse, it leads to a self-reinforcing
> effect:  If the true extent of people using MELPA is unknown,
> emacs-devel will continue making choices on the false premise that MELPA
> isn't a thing at all.  In the worst case GNU ELPA will be eclipsed by
> MELPA and cease to be of any practical concern (and it sure seems to me
> that this is where things are heading currently).  Remember, this is a
> survey to find out how Emacs users use their editor, not about how much
> they adhere to FSF ideology.

Does MELPA contains non-free software?

Re: Proposal for an Emacs User Survey

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

> Does MELPA contains non-free software?

No, it doesn't.  Neither does the website contain non-free JS, it runs
fine with LibreJS.  The only concern here is that MELPA does not adhere
to GNU's coding guidelines, such as never ever referring, linking or
otherwise supporting non-free software (for example by hosting packages
that wrap non-free executables or are tied into working with non-free
operating systems).  And why should they, they are not GNU, they are a
community-provided package archive and a wildly successful one, too.

Even if it contained non-free software, this is beside the point.  What
use is a survey if it doesn't strive to accurately capture the status
quo and instead basks in ideological purity?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Vasilij Schneidermann <mail@vasilij.de> [2020-10-11 15:51]:
> > Does MELPA contains non-free software?
> 
> No, it doesn't.  Neither does the website contain non-free JS, it runs
> fine with LibreJS.  The only concern here is that MELPA does not adhere
> to GNU's coding guidelines, such as never ever referring, linking or
> otherwise supporting non-free software (for example by hosting packages
> that wrap non-free executables or are tied into working with non-free
> operating systems).

In other words packages on MELPA would or could refer, link and
support non-free software including wrapping it.

Hypothetical example could be the emacs package that supports
proprietary speech system, such package could be interesting for
developers, but without proprietary speech system it could not run,
thus disabling freedom of users.

Emacs is free software project, forerunner, so I guess that developers
by GNU principles would not support, improve, develop, those software
packages that are beyond free software principles. What each developer
does in their private life is their own decision, each is free to use
proprietary software how they wish, yet in the written or not written
agreement for Emacs development, you all do not develop or support
proprietary software.

Thus a survey that brings up the issue that somebody would like to see
Emacs Lisp package that supports proprietary speech software on
Windows, would simply not work, as that would be waste of time.

Nevertheless such inputs may come anyway, if good communication
channel is offered, but there is no need to support inputs from users
who would use or do not mind using proprietary wrapped software, as
project is GNU, and Emacs is inseparable of principles of free
software and GNU.

> And why should they, they are not GNU, they are a community-provided
> package archive and a wildly successful one, too.
> 
> Even if it contained non-free software, this is beside the point.
> What use is a survey if it doesn't strive to accurately capture the
> status quo and instead basks in ideological purity?

In my opinion, if GNU project is asking for input by well established
communication line, and I suggest that such is implemented in Help
menu, then it is inseparable from the movement to push free software
philosophy onto users, as that is good thing to do.

Ideological purity is impossible attainment.

Every person is free to make surveys, any company or any individual, a
user could use even proprietary software to make survey for Emacs,
Microsoft would be free to begin its Windows by asking if their editor
is better or Emacs. Any private person can turn on advertising on
Internet and capture Emacs users and ask them any questions. This
freedom of making surveys exist. I am saying this as "Emacs Survey"
can be done independently on global level, but that is not for GNU to
do it in that manner.

MELPA probably collects information of IP addresses, locations, and
number of downloads, thus they have enough data and possibilities for
their own survey. By the way, it disallows me to browse the MELPA.ORG
website without Javascript, I do not know if it is free or not.

Let us imagine Microsoft providing Emacs script repository, that is
quite possible, they do it now indirectly through Github, but we can
imagine that such abusive company could lead users to many non-free
software, and that was never the purpose of Emacs. Would then GNU ask
Microsoft to ask users about that?! I don't think so, so GNU and GNU
Emacs are inseparable from free software enlightenment and should not
lead users to repositories where users may download or otherwise get
abused by proprietary software.

I like looking into extreme hypothetical situations to recognize if
the direction of any action is supportive or not supportive for the
group of people I personally belong, in this case free software
movement. And that does not mean bashing onto anybody who bundles
proprietary software, it just means promoting more of those free
software principles to guide people gently into abuse free future.

Jean

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> MELPA probably collects information of IP addresses, locations, and
> number of downloads, thus they have enough data and possibilities for
> their own survey. By the way, it disallows me to browse the MELPA.ORG
> website without Javascript, I do not know if it is free or not.

LibreJS only blocks two scripts on melpa.org: a twitter widget, and a
google analytics snippet.

Neither of those are necessary for using the site.

Re: Proposal for an Emacs User Survey

[Brett Gilio]

Thibaut Verron <thibaut.verron@gmail.com> writes:

>
> LibreJS only blocks two scripts on melpa.org: a twitter widget, and a
> google analytics snippet.
>
> Neither of those are necessary for using the site.
>

It would be great if melpa.org didn't require Javascript for
functionality, at all. But that is just my opinion.

It is not possible to browse packages on M-x eww, for example. Perhaps a
fallback would be nice.

-- 
Brett M. Gilio
<brettg@gnu.org>
https://brettgilio.com

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> It would be great if melpa.org didn't require Javascript for
> functionality, at all. But that is just my opinion.
>
> It is not possible to browse packages on M-x eww, for example. Perhaps a
> fallback would be nice.

M-x list-packages ?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 21:28]:
> > It would be great if melpa.org didn't require Javascript for
> > functionality, at all. But that is just my opinion.
> >
> > It is not possible to browse packages on M-x eww, for example. Perhaps a
> > fallback would be nice.
> 
> M-x list-packages ?

It is possible that MELPA webmaster have setup that website with
intention not to let Emacs users browse packages through Emacs, but to
force them to install MELPA into Emacs

As the link to packages list is impossible to reach, but the link on
how to add MELPA to Emacs is quite reachable through Emacs.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le dim. 11 oct. 2020 à 20:47, Jean Louis <bugs@gnu.support> a écrit :
>
> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 21:28]:
> > > It would be great if melpa.org didn't require Javascript for
> > > functionality, at all. But that is just my opinion.
> > >
> > > It is not possible to browse packages on M-x eww, for example. Perhaps a
> > > fallback would be nice.
> >
> > M-x list-packages ?
>
> It is possible that MELPA webmaster have setup that website with
> intention not to let Emacs users browse packages through Emacs, but to
> force them to install MELPA into Emacs
>
> As the link to packages list is impossible to reach, but the link on
> how to add MELPA to Emacs is quite reachable through Emacs.

I think you're making an unfair "trial of motives" here.

You can add Melpa to the list of repositories in an emacs instance and
then browse it with list-packages. That will not "install Melpa" if
you don't add the repository line in your init.el.

For the record, here is the (short) discussion on the topic:
https://github.com/melpa/melpa/issues/3483

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 21:58]:
> Le dim. 11 oct. 2020 à 20:47, Jean Louis <bugs@gnu.support> a écrit :
> >
> > * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 21:28]:
> > > > It would be great if melpa.org didn't require Javascript for
> > > > functionality, at all. But that is just my opinion.
> > > >
> > > > It is not possible to browse packages on M-x eww, for example. Perhaps a
> > > > fallback would be nice.
> > >
> > > M-x list-packages ?
> >
> > It is possible that MELPA webmaster have setup that website with
> > intention not to let Emacs users browse packages through Emacs, but to
> > force them to install MELPA into Emacs
> >
> > As the link to packages list is impossible to reach, but the link on
> > how to add MELPA to Emacs is quite reachable through Emacs.
> 
> I think you're making an unfair "trial of motives" here.

Slight act of distracting and magic is done, instead of letting users
understand what MELPA is, like does it at all offer free software (it
is not described on the page or mentioned), it just says: do this and
install this in Emacs, then proceed with installation of packages. It
does not link back to Emacs software page, which I would as
responsible webmaster always do.

As I am not a judge, I make no trials, my impression though remains
the same that it was intentional. Now you say the bug report from 2016
to make MELPA browsable by Emacs still remains same.

> For the record, here is the (short) discussion on the topic:
> https://github.com/melpa/melpa/issues/3483

I see some pretty trivial nonsense there:
> "There is therefore no easy way to fix this, and I don't believe it
> is worth a bigger effort,"

How it was done by using Javascript, it can be done by using Emacs
Lisp or any other programming language, and generate static pages,
quite automatically.

> You can add Melpa to the list of repositories in an emacs instance and
> then browse it with list-packages. That will not "install Melpa" if
> you don't add the repository line in your init.el.

I am speaking of accessibility subject.

It will be remedied with elpa.nongnu.org

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 20:37]:
> > MELPA probably collects information of IP addresses, locations, and
> > number of downloads, thus they have enough data and possibilities for
> > their own survey. By the way, it disallows me to browse the MELPA.ORG
> > website without Javascript, I do not know if it is free or not.
> 
> LibreJS only blocks two scripts on melpa.org: a twitter widget, and a
> google analytics snippet.
> 
> Neither of those are necessary for using the site.

I expect that any site promoting Emacs based software is accessible
through Emacs and that such site should not expose me to Google
Intelligence and Twitter, I don't like being tracked across other
websites.

I can perfectly use http://elpa.gnu.org through Emacs

I cannot at all use https://melpa.org through Emacs, it is
impossible. It is common sense not to design sites in such stunned way
(sorry I was looking for synonyms of "stupid" not to be too bold).

Website of MELPA is ridiculous. If it promotes Emacs, it should be
accessible through Emacs. And now we discuss what Emacs users want,
example me, I don't want to be pushed around. 

MELPA 

* Packages <--- I can click, but nothing happens
* Getting started <--- I can click, but nothing happens
* GitHub <--- brings me to Microsoft Github (THE LINK WORKS!!! YEAH!!!)

MELPA (Milkypostman’s Emacs Lisp Package Archive)

* Up-to-date packages built on our servers from upstream source
* Installable in any Emacs with 'package.el' - no local version-control tools needed
* Curated - no obsolete, renamed, forked or randomly hacked packages
* Comprehensive - more packages than any other archive
* Automatic updates - new commits result in new packages
* Extensible - contribute recipes via github, and we'll build the packages

This site requires JavaScript to function. You can browse and install packages directly from Emacs (see MELPA's README.md on GitHub).

Source code for this page → 
JavaScript license information 

-------- end of inaccessible MELPA website --------

I cannot wait for non-GNU ELPA to begin.


Jean

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> I can perfectly use http://elpa.gnu.org through Emacs
>
> I cannot at all use https://melpa.org through Emacs, it is
> impossible. It is common sense not to design sites in such stunned way
> (sorry I was looking for synonyms of "stupid" not to be too bold).

It seems reasonable to me that a list that changes every now and then
(Elpa) is easier to serve statically than a list which is updated
several times a day (Melpa).

But if the list of GNU Elpa is generated automatically, maybe the code
to do that can be useful to Melpa.

> Website of MELPA is ridiculous. If it promotes Emacs, it should be
> accessible through Emacs. And now we discuss what Emacs users want,
> example me, I don't want to be pushed around.

Or maybe Emacs should have a web browser which can run (free) javascript.

I don't think so, for obvious security reasons, but then again I don't
expect to be able to access all emacs-related websites from within
Emacs.

And, again, Melpa is accessible through Emacs: the entire content of
the website is served on an API which is fully supported by Emacs
(list-packages).

> This site requires JavaScript to function.

At least that's pretty explicit.

> JavaScript license information

And they went through the trouble of using only free javascript for
the website (except for the, admittedly, unnecessary twitter feed and
analytics, which a lot of users will block with their adblocker or
LibreJS).

Re: Proposal for an Emacs User Survey

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

> 
> Or maybe Emacs should have a web browser which can run (free) javascript.
> 
Emacs with —with-xwidget build already does. And I was working on some
patches to port librejs and potentially other WebExtensions plugins
to it. The copyright office of my school is a bit slow so it’s been stopped
for a while, but should not take much time once the copyright assignment
is complete.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Or maybe Emacs should have a web browser which can run (free) javascript.
  > > 
  > Emacs with —with-xwidget build already does.

Please do not do anything along those lines.
What we already did may have been a grave ethical mistake.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-11 22:16]:
> > I can perfectly use http://elpa.gnu.org through Emacs
> >
> > I cannot at all use https://melpa.org through Emacs, it is
> > impossible. It is common sense not to design sites in such stunned way
> > (sorry I was looking for synonyms of "stupid" not to be too bold).
> 
> It seems reasonable to me that a list that changes every now and then
> (Elpa) is easier to serve statically than a list which is updated
> several times a day (Melpa).

What can be generated online, it can be generated statically, it is
matter of seconds. Thus I am not underestimating the webmaster of
MELPA. He did not want it in 2016, he does not want it now. There is
no "discussion" on the bug report, he closed the bug, finished. It
does not seem to be user friendly.

There is nothing interactive on MELPA to be served dynamically, there
is no user input other than clicking, there are no interactions. I do
not underestimate programmers and their intelligence. If MELPA would
be easily browsable, it would be easily duplicated as a website.

> But if the list of GNU Elpa is generated automatically, maybe the
> code to do that can be useful to Melpa.

Of course it can. For now it seem trivial, and can be done through
Emacs Lisp. 

> > Website of MELPA is ridiculous. If it promotes Emacs, it should be
> > accessible through Emacs. And now we discuss what Emacs users want,
> > example me, I don't want to be pushed around.
> 
> Or maybe Emacs should have a web browser which can run (free)
> javascript.

But why only javascript, maybe Emacs should have Emacs Lisp instead of
Javascript, that way we could run programs on Emacs by installing them
straight from websites without knowing what is going to happen, within
some safe environment so that it does not affect the
system. So far I know, any script could be included in web pages, it
all depends of plugins to browsers, I remember Perl could be included
in the web pages, and other programming languages, why not Emacs Lisp
then.

> I don't think so, for obvious security reasons, but then again I
> don't expect to be able to access all emacs-related websites from
> within Emacs.

That's what I am talking about, it should be accessible through Emacs.

> And, again, Melpa is accessible through Emacs: the entire content of
> the website is served on an API which is fully supported by Emacs
> (list-packages).
> 
> > This site requires JavaScript to function.
> 
> At least that's pretty explicit.

And how that can be good, we speak of Emacs, not of whatever
else. People do use Internet through Emacs.

> > JavaScript license information
> 
> And they went through the trouble of using only free javascript for
> the website (except for the, admittedly, unnecessary twitter feed and
> analytics, which a lot of users will block with their adblocker or
> LibreJS).

Which makes the MELPA website non-free website, not recommended for
freedom lovers.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Even if it contained non-free software, this is beside the point.  What
  > use is a survey if it doesn't strive to accurately capture the status
  > quo and instead basks in ideological purity?

That's not a fair description of what we are doing, so it is a misled
criticism.  You can't convince me that what I am doing is mistaken by
describing it wrong.  And please don't be so harsh in tone.

Please review the Kind Communications Guidelines,
https://gnu.org/philosophy/kind-communication.html,
and try to express your point in a way that is less harsh.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

>
>   > Even if it contained non-free software, this is beside the point.  What
>   > use is a survey if it doesn't strive to accurately capture the status
>   > quo and instead basks in ideological purity?
>
> That's not a fair description of what we are doing, so it is a misled
> criticism.


Just as a reminder, here is what the email starter suggested:

> a survey for Emacs users to better grasp the diversity and various usages
out there

There is nothing about taking practical decisions or encouraging free
software (or anything) there.

And I would be worried about selection bias in a survey which voluntarily
omits the most popular options from the lists of choices: how many people
would choose not to complete the survey of they feel that the questions are
biased?

Would it be acceptable to put all major package repos in the list, with a
warning saying that melpa is not a free software repository? The same
warning could be used when mentioning windows, non-free IDEs, etc.

[-- Attachment #2: Type: text/html, Size: 1501 bytes --]

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 06:33]:
> >
> >   > Even if it contained non-free software, this is beside the point.  What
> >   > use is a survey if it doesn't strive to accurately capture the status
> >   > quo and instead basks in ideological purity?
> >
> > That's not a fair description of what we are doing, so it is a misled
> > criticism.
> 
> 
> Just as a reminder, here is what the email starter suggested:
> 
> > a survey for Emacs users to better grasp the diversity and various usages
> out there
> 
> There is nothing about taking practical decisions or encouraging free
> software (or anything) there.

Every opinion poll survey has a purpose, normally the purpose is to
find out what majority wish and want to improve the product or service
and thus reach or gain more customers, strike it, users.

- Encouraging free software is within the context of GNU Emacs subject.

- Encouraging or supporting usage of non-free software is not within
  the context.

Thus the survey or opinion polls cannot be evaluated in the manner to
encourage usage of proprietary software, it is vice versa. Those
opinions vouching for proprietary software would or should be
enlightened of the benefits and safety of free software.

A survey from the context of GNU Emacs development naturally should
involve questions related to free software as the purpose of Emacs was
historically to distribute free software,
https://www.gnu.org/gnu/thegnuproject.html

> Would it be acceptable to put all major package repos in the list,
> with a warning saying that melpa is not a free software repository?

Are there many package repositories? I know of three, there will be
fourth soon elpa.nongnu.org or similar.

They do provide free software naturally as packages should be GPL so
far I understand (not sure), but if they wrap non free software or
have pointers to non free software, such recommendation would be
contrary to principles why GNU Emacs have been made as free software.

> The same warning could be used when mentioning windows, non-free
> IDEs, etc.

Warning you mention is used on Emacs download page, see
https://www.gnu.org/software/emacs/download.html

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> > Just as a reminder, here is what the email starter suggested:
> >
> > > a survey for Emacs users to better grasp the diversity and various usages
> > out there
> >
> > There is nothing about taking practical decisions or encouraging free
> > software (or anything) there.
>
> Every opinion poll survey has a purpose, normally the purpose is to
> find out what majority wish and want to improve the product or service
> and thus reach or gain more customers, strike it, users.

Yes it has a purpose, I quoted it above. To understand "the diversity
and various usages out there". Specifically excluding some popular
usages defeats that purpose.

If anything, wouldn't we want to get an idea how many Emacs users
currently use a non-free package repository?

On the other hand, there are no questions in the suggested list about
what people want.

> > Would it be acceptable to put all major package repos in the list,
> > with a warning saying that melpa is not a free software repository?
>
> Are there many package repositories? I know of three, there will be
> fourth soon elpa.nongnu.org or similar.

Yes, that sounds about right.

> They do provide free software naturally as packages should be GPL so
> far I understand (not sure), but if they wrap non free software or
> have pointers to non free software, such recommendation would be
> contrary to principles why GNU Emacs have been made as free software.

I don't understand why a question in a survey would be seen as a recommendation.

That's similar, in a sense, to those social surveys asking people if
they have done drugs. I don't think those want to encourage people to
take drugs.

> > The same warning could be used when mentioning windows, non-free
> > IDEs, etc.
>
> Warning you mention is used on Emacs download page, see
> https://www.gnu.org/software/emacs/download.html

Great, so there is precedent ! Why is it acceptable and sufficient on
the main download page but not acceptable in a survey?

Those words are a bit too harsh when applied to Melpa, I hope that we
agree on that. And nobody likes to read propaganda (no matter how
justified) in a survey, so having such a long tirade could again lead
to selection bias.

But, for example, wouldn't something like below be both short and
explicit enough?

"- Melpa (Note: Emacs and the GNU project DO NOT ENDORSE package
repositories which encourage non-free software, see
https://www.gnu.org/philosophy/)"

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 08:34]:
> > > Just as a reminder, here is what the email starter suggested:
> > >
> > > > a survey for Emacs users to better grasp the diversity and various usages
> > > out there
> > >
> > > There is nothing about taking practical decisions or encouraging free
> > > software (or anything) there.
> >
> > Every opinion poll survey has a purpose, normally the purpose is to
> > find out what majority wish and want to improve the product or service
> > and thus reach or gain more customers, strike it, users.
> 
> Yes it has a purpose, I quoted it above. To understand "the diversity
> and various usages out there". Specifically excluding some popular
> usages defeats that purpose.

Tell me examples of popular usage that you refer to?

Free form gives enough possibility for any user to explain anything
they wish.

> If anything, wouldn't we want to get an idea how many Emacs users
> currently use a non-free package repository?

I am not sure if there is any non-free package repository for
Emacs.

MELPA is fetching most packages from the Microsoft Github, and Github
dictates free licenses for any public repository, most of them are
free software. For me is hard to find particular example that uses non
free software. 

That will be work to do, to move some public packages to non-GNU ELPA.

MELPA recipes can be cloned, copy of software can be placed on
nongnu.org automatically, later revised from TODO to be TO PUBLISH,
and distributed ethically. 

> > They do provide free software naturally as packages should be GPL so
> > far I understand (not sure), but if they wrap non free software or
> > have pointers to non free software, such recommendation would be
> > contrary to principles why GNU Emacs have been made as free software.
> 
> I don't understand why a question in a survey would be seen as a
> recommendation.

Above paragraph refers to MELPA, that could wrap non free software in
the free software packages. I can then imagine links in packages pointing
to non free software, that is what was meant with recommendation. It
does not refer to questions in the opinion poll.

> That's similar, in a sense, to those social surveys asking people if
> they have done drugs. I don't think those want to encourage people to
> take drugs.
> 
> > > The same warning could be used when mentioning windows, non-free
> > > IDEs, etc.
> >
> > Warning you mention is used on Emacs download page, see
> > https://www.gnu.org/software/emacs/download.html
> 
> Great, so there is precedent ! Why is it acceptable and sufficient on
> the main download page but not acceptable in a survey?

It is not a new precedent, it was from the creation of Emacs to talk
about free software, and advise people. In my opinion GNU project
should increase the marketing of free software philosophy by power of
10, it is not enough.

> Those words are a bit too harsh when applied to Melpa, I hope that
> we agree on that. And nobody likes to read propaganda (no matter how
> justified) in a survey, so having such a long tirade could again
> lead to selection bias.

GNU project with promotion of free software is not biased as that
would mean that it is influenced in an unfair way.

GNU project is influenced in a fair way and thus should be promoting
and supporting free software and helping users of proprietary software
to understand what is free software and freedom in computing.

The word propaganda you maybe used in a negative connotation, but the
word itself means promoting information to spread some cause. Who is
not interested, would not read it. The point of propaganda that some
will get interested, so propaganda gives results for those who are.

> But, for example, wouldn't something like below be both short and
> explicit enough?
> 
> "- Melpa (Note: Emacs and the GNU project DO NOT ENDORSE package
> repositories which encourage non-free software, see
> https://www.gnu.org/philosophy/)"

I do not think that it is necessary for survey from GNU Emacs to ask
if people are using Melpa or whatever other software
repository.

Reason for that is that it is obvious that people do, people ARE using
MELPA and Marmalade software repositories; AND more important reason
not to ask is that information about usage of those repositories,
likes, number of contributors, it is already available on the
Microsoft Github "Insights" link on the MELPA page. There is no point
in asking users what is already obvious. There are stars or likes on
Github.

If there is certain disagreement between GNU and MELPA, that is valid
reason as well. 

In my opinion, the team of MELPA is preparing the list of packages
well, then GNU team working on ELPA on nongnu.org can fork all the
software and curate and remove whatever is unsafe or not ethical and
make a new repository.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> Tell me examples of popular usage that you refer to?

Downloading packages from Melpa every day. Or even using Doom or
Spacemacs (both activate the Melpa repository afaict, through Straight
in the case of Doom).

> Free form gives enough possibility for any user to explain anything
> they wish.

But do they wish to?

Imagine a poll for a presidential election, where John Doe and Richard
Miles are the frontrunners.

Now let's say that a pollster frames its question like this:

"Who do you intend to vote for?
- John Doe
- Other (please specify)"

What would you think of the data resulting from such a poll?

>
> > If anything, wouldn't we want to get an idea how many Emacs users
> > currently use a non-free package repository?
>
> I am not sure if there is any non-free package repository for
> Emacs.
>
> MELPA is fetching most packages from the Microsoft Github, and Github
> dictates free licenses for any public repository, most of them are
> free software. For me is hard to find particular example that uses non
> free software.

Okay...

> Above paragraph refers to MELPA, that could wrap non free software in
> the free software packages. I can then imagine links in packages pointing
> to non free software, that is what was meant with recommendation. It
> does not refer to questions in the opinion poll.

Okay.

> GNU project with promotion of free software is not biased as that
> would mean that it is influenced in an unfair way.

The GNU project is not biased. But the proposed amended survey would
be, similar to the hypothetical presidential poll above.

>
> GNU project is influenced in a fair way and thus should be promoting
> and supporting free software and helping users of proprietary software
> to understand what is free software and freedom in computing.
>
> The word propaganda you maybe used in a negative connotation, but the
> word itself means promoting information to spread some cause. Who is
> not interested, would not read it. The point of propaganda that some
> will get interested, so propaganda gives results for those who are.

Who is not interested might also leave the survey and skew the data.
Advertisement
campaigns disguised as surveys are legion.

> > But, for example, wouldn't something like below be both short and
> > explicit enough?
> >
> > "- Melpa (Note: Emacs and the GNU project DO NOT ENDORSE package
> > repositories which encourage non-free software, see
> > https://www.gnu.org/philosophy/)"
>
> I do not think that it is necessary for survey from GNU Emacs to ask
> if people are using Melpa or whatever other software
> repository.
>
> Reason for that is that it is obvious that people do, people ARE using
> MELPA and Marmalade software repositories;

Okay, that is a valid point (although I would be interested to
know if people still use Marmalade). But then we might as well drop
the repository question altogether.

> AND more important reason
> not to ask is that information about usage of those repositories,
> likes, number of contributors, it is already available on the
> Microsoft Github "Insights" link on the MELPA page. There is no point
> in asking users what is already obvious. There are stars or likes on
> Github.

This data might have its own selection bias.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 10:00]:
> > Tell me examples of popular usage that you refer to?
> 
> Downloading packages from Melpa every day. Or even using Doom or
> Spacemacs (both activate the Melpa repository afaict, through Straight
> in the case of Doom).

Aha.

As there is disagreement between GNU project and MELPA way of
providing software packages, and as it is obvious that many users do
use MELPA, there is no need to ask what is obvious, one has to
implement wider software package repository from GNU project itself.

Spacemacs is configuration, it is also considered software, so also
for this is obvious that 20,000+ Microsoft Github users have given
star/like on Github, as that is what I understand when I read 20.7k+
-- so it is obvious, that need not be asked, asking would be useless
compared to obvious statistical results. (Note, when I use the word
Microsoft in front of Github, it is used in the sense of warning.)

Unless Github statistics are fake, GNU Emacs developers can already
think why those 20,700+ users are using Spacemacs and think how to
improve Emacs in general. In my opinion there is no need to ask what
is obvious. There is need to act.

> > Free form gives enough possibility for any user to explain anything
> > they wish.
> 
> But do they wish to?

So far I read on this list, that was proposed by RMS and few others
acknowledged it, I also think that it should be free form and I and
Drew, we like it to be included permanently in the menu, something
like:

Help -> Tell us how to improve

Then from Tell us how to improve:

-> Report Emacs bug
-> Suggest improvements
-> Subscribe to Help GNU Emacs mailing list
-> Send email to Help GNU Emacs list
-> Unsubscribe from Help GNU Emacs mailing list

The menu item "Suggest improvements" could be similar to
report-emacs-bug just enlightening and inviting user to communicate
about enhancements.

There is no need in Emacs for those opinion poll surveys with set of
questions, it is questionable and unclear who would be doing such
survey, who would be paying for that, and who would be evaluating it.

What is successful action is to listen to users who write to
help-gnu-emacs, devel-emacs and developers also listen to users beyond
official GNU communication lines, so that what is successful is
working, and need not be changed.

Cheapest long term solution is to embed it in Emacs, that is anyway
inviting new users to collaborate. Then listening to users and
improving by understanding their needs.

> Imagine a poll for a presidential election, where John Doe and Richard
> Miles are the frontrunners.
> 
> Now let's say that a pollster frames its question like this:
> 
> "Who do you intend to vote for?
> - John Doe
> - Other (please specify)"
> 
> What would you think of the data resulting from such a poll?

I know what you mean, especially for the reason that I have been doing
surveys with people on the phone, in person face to face, on the
street, and by visiting people at their homes, I have got paid for
making surveys, and I know how to evaluate surveys, and how to write
them properly.

My answer to that proposal is same as Drew's, something like
implementation of the above menus in the Help.

> > GNU project with promotion of free software is not biased as that
> > would mean that it is influenced in an unfair way.
> 
> The GNU project is not biased. But the proposed amended survey would
> be, similar to the hypothetical presidential poll above.

So far I know, GNU project is not encompassing projects that are not
ethical.

For those questions that RMS proposed, I think they are also not
necessary to be made for the reason that every survey gives its
results or data that has to be evaluated, and it is obvious that many
users, millions probably, do not know what is Linux as kernel and what
is operating system.

In that sense, Emacs already has in its splash screen (About GNU Emacs):
Welcome to GNU Emacs, one component of the GNU/Linux operating system

Where the words "GNU/Linux" are hyperlinked to appropriate GNU page.

nd in my opinion, Emacs should contain more information and links to
free software, that should be *contained* in the Emacs distribution
and not just hyperlinked.

For example I think that GNU Manifesto should go back to GNU Emacs
where it was, it should be obtainable through Help system.

Additionally, I would include in the Help system, the option to turn
on "Free Software Menu".

Then the Free Software menu option would appear with various
submenues, delivering GNU free software philosophy straight from
Emacs, without using Internet.

My opinion is maybe in that sense more "pushy" then what RMS wants, I
think that free software philosophy has to be pushy 10 times more than
it is now, as the increase of proprietary software and troublesame
abuses of people on this planet increased as well.

> Okay, that is a valid point (although I would be interested to
> know if people still use Marmalade). But then we might as well drop
> the repository question altogether.

I would drop all the questions, and just have it embedded in Emacs
Help menu for people to complain, suggest improvements, etc. System
already exists, but "Report Emacs bug" does not incite suggesting
improvements.

I read now that Marmalade has been discontinued.

Proposal about making a survey is fine, why not? But where is the
person responsible, money, funds, methods, tactical plan how to
implement the survey, evaluator, and so on?!

All that is time and resources consuming process, while in same time
developers already know what is attracting people, what has to be done
to be improved, they are finally improving it every day.

> > AND more important reason not to ask is that information about
> > usage of those repositories, likes, number of contributors, it is
> > already available on the Microsoft Github "Insights" link on the
> > MELPA page. There is no point in asking users what is already
> > obvious. There are stars or likes on Github.
> 
> This data might have its own selection bias.

Of course. I would not trust the Microsoft Github data at all, and it
does not show logically to be true, here is short analysis:

- we assume here that Spacemacs has 20700 stars/likes on Microsoft
  Github, because it says so on the page. Does that mean those are
  "users" of Spacemacs? Hard to believe, see below why.

- if I open the main Spacemacs Github page, I can see that last
  improvement on Spacemacs was 29th February 2020, there is not much
  going on. But Spacemacs is appearing as first on Github for the
  search word "emacs".

- I can see number of contributors, number is 562 contributors, there
  are so many contributors but the last update was 29th February 2020,
  maybe I am mistaken. So many contributors are counted as such if
  they report a bug, they do not write software necessarily. I can see
  very unrelated or mixture of bugs reported:
  https://github.com/syl20bnr/spacemacs/issues?q=is%3Aissue+is%3Aclosed

- in my opinion, there is maybe 1000 - max 2000 real users of
  Spacemacs, not more than that. I am assumin that maybe each among 4
  will make "issue" and thus become contributor, about 562 x 4 people
  is my imaginative assumption of number of users who ever tried
  Spacemacs, and I am sure that bunch of them stopped using it as
  well. 

It is untrue that 20700 stars mean there are so many users, people
click how they want and wish. Spacemacs is definitely not for
beginners, it is distributed from Git, so user must have and know how
to git pull or clone it.

It could be as well possible that author have advertised Spacemacs
back in time. In general, it is doing good job to advertise Emacs.

Jean

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> Spacemacs is configuration, it is also considered software, so also
> for this is obvious that 20,000+ Microsoft Github users have given
> star/like on Github, as that is what I understand when I read 20.7k+
> -- so it is obvious, that need not be asked, asking would be useless
> compared to obvious statistical results. (Note, when I use the word
> Microsoft in front of Github, it is used in the sense of warning.)

I would think that those 20k are a gross underestimate, more on that
below. (And I understood that.)

> Unless Github statistics are fake, GNU Emacs developers can already
> think why those 20,700+ users are using Spacemacs and think how to
> improve Emacs in general. In my opinion there is no need to ask what
> is obvious. There is need to act.

The actual number is not obvious, beyond "at least 20k".


> I know what you mean, especially for the reason that I have been doing
> surveys with people on the phone, in person face to face, on the
> street, and by visiting people at their homes, I have got paid for
> making surveys, and I know how to evaluate surveys, and how to write
> them properly.
>
> My answer to that proposal is same as Drew's, something like
> implementation of the above menus in the Help.

So only free form, no multiple choice, and permanently in Emacs?
That's fine for me.

The goal is then sensibly different from that of the suggested survey,
and as you say, they are not mutually exclusive.

> My opinion is maybe in that sense more "pushy" then what RMS wants, I
> think that free software philosophy has to be pushy 10 times more than
> it is now, as the increase of proprietary software and troublesame
> abuses of people on this planet increased as well.
>
> > Okay, that is a valid point (although I would be interested to
> > know if people still use Marmalade). But then we might as well drop
> > the repository question altogether.
>
> I would drop all the questions, and just have it embedded in Emacs
> Help menu for people to complain, suggest improvements, etc. System
> already exists, but "Report Emacs bug" does not incite suggesting
> improvements.

Questions are useful to avoid writer paralysis though.
There could be three systems: "answer survey" (with a few free-form
questions to give inspiration), "send suggestion", "report bug".

>
> Of course. I would not trust the Microsoft Github data at all, and it
> does not show logically to be true, here is short analysis:
>
> - we assume here that Spacemacs has 20700 stars/likes on Microsoft
>   Github, because it says so on the page. Does that mean those are
>   "users" of Spacemacs? Hard to believe, see below why.
>
> - if I open the main Spacemacs Github page, I can see that last
>   improvement on Spacemacs was 29th February 2020, there is not much
>   going on. But Spacemacs is appearing as first on Github for the
>   search word "emacs".

Second is the emacs-mirror (and the first result for a google search
"github emacs" for me).

I don't know how github sorts its results, it is not just stars.

> - I can see number of contributors, number is 562 contributors, there
>   are so many contributors but the last update was 29th February 2020,
>   maybe I am mistaken. So many contributors are counted as such if
>   they report a bug, they do not write software necessarily. I can see
>   very unrelated or mixture of bugs reported:
>   https://github.com/syl20bnr/spacemacs/issues?q=is%3Aissue+is%3Aclosed

Some branches are more recent, see for example buffertabs, last
updated 2 weeks ago and 4k commits ahead of master.

Apparently Github defines contributor as "has published to master"
(possibly via a fork and pull-request?).

The list of contributors is not visible beyond the top 100, but those
100 all have at least one commit on master.

> It is untrue that 20700 stars mean there are so many users, people
> click how they want and wish. Spacemacs is definitely not for
> beginners, it is distributed from Git, so user must have and know how
> to git pull or clone it.

Are you sure about that? The website spacemacs.org has a big
"download" button which serves a zipped copy of master.
Does running spacemacs require knowledge from git after that point?

Regardless, it is possible to run spacemacs without ever visiting the
github page of the project, and github stars cannot track those.
So even though some of those 20k stars are probably not users, or are
no longer users, I still believe that the actual number of users may
be way more than 20k.

A properly conducted survey seems like a good way to get a better
estimate of the actual number.
But I agree that it is not a trivial task.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 11:38]:
> So only free form, no multiple choice, and permanently in Emacs?
> That's fine for me.
> 
> The goal is then sensibly different from that of the suggested survey,
> and as you say, they are not mutually exclusive.

Yes, like that.

Emacs has its website, various simple qeustion based opinion polls can
easily published on the website, that was done since inception of
websites.

> > My opinion is maybe in that sense more "pushy" then what RMS wants, I
> > think that free software philosophy has to be pushy 10 times more than
> > it is now, as the increase of proprietary software and troublesame
> > abuses of people on this planet increased as well.
> >
> > > Okay, that is a valid point (although I would be interested to
> > > know if people still use Marmalade). But then we might as well drop
> > > the repository question altogether.
> >
> > I would drop all the questions, and just have it embedded in Emacs
> > Help menu for people to complain, suggest improvements, etc. System
> > already exists, but "Report Emacs bug" does not incite suggesting
> > improvements.
> 
> Questions are useful to avoid writer paralysis though.
> There could be three systems: "answer survey" (with a few free-form
> questions to give inspiration), "send suggestion", "report bug".

That is right, that is how users should be motivated to communicate.

> > Of course. I would not trust the Microsoft Github data at all, and it
> > does not show logically to be true, here is short analysis:
> >
> > - we assume here that Spacemacs has 20700 stars/likes on Microsoft
> >   Github, because it says so on the page. Does that mean those are
> >   "users" of Spacemacs? Hard to believe, see below why.
> >
> > - if I open the main Spacemacs Github page, I can see that last
> >   improvement on Spacemacs was 29th February 2020, there is not much
> >   going on. But Spacemacs is appearing as first on Github for the
> >   search word "emacs".
> 
> Second is the emacs-mirror (and the first result for a google search
> "github emacs" for me).

Emacs mirror on Github is second, but obviously there is no movement
there, it shows that statistics are doubtful.

> I don't know how github sorts its results, it is not just stars.

Because nobody knows, and especially because it is from Microsoft, I
would not touch it or believe it. But that is up to imaginary, as for
now, survey evaluators.

> > It is untrue that 20700 stars mean there are so many users, people
> > click how they want and wish. Spacemacs is definitely not for
> > beginners, it is distributed from Git, so user must have and know how
> > to git pull or clone it.
> 
> Are you sure about that? The website spacemacs.org has a big
> "download" button which serves a zipped copy of master.
> Does running spacemacs require knowledge from git after that point?

No info on that. But it is quite easy to ask Spacemacs for website
statistics, as survey for that reason is obviously not necessary. Then
how to know if statistics are real?

By asking various configuration packages for statistics, one can find
out some ratios, but why? One has to ask WHY each time. One WHY could
be to replicate or include or invite the configuration to GNU
project. If that is reason why, then why not ask. If there is nothing
special, it is just good to let people do what they wish.

Example of discussed Spacemacs is clear to me, the data is already
there, so the result of pathetic evaluation is that there are many Vim
users who like now to use Emacs because of Spacemacs, there are many
new keybindings, so if anybody wish to improve Emacs, one could see
which are major features of Spacemacs and could ask developers to
include their settings into main stream Emacs. Why not? Just go
ahead and ask.

GNU project already invited everybody to contribute with their
projects to be included in GNU.

You may ask RMS and Spacemacs developers, if they wish to include
Spacemacs as official part of GNU Emacs, that way GNU Emacs would
possibly gain popularity.

Spacemacs configuration could be optional for user to set, similar
like a theme. Why not. I know there are ways.

> Regardless, it is possible to run spacemacs without ever visiting the
> github page of the project, and github stars cannot track those.
> So even though some of those 20k stars are probably not users, or are
> no longer users, I still believe that the actual number of users may
> be way more than 20k.

Yes, but how do you know that? Like where is information available?

Finally, why is it important? It is clear that some people like Vim
configuration in Emacs, yet it does not mean to change whole
configuration to be like theirs.

Re: Proposal for an Emacs User Survey

[Ihor Radchenko]

> Reason for that is that it is obvious that people do, people ARE using
> MELPA and Marmalade software repositories; AND more important reason
> not to ask is that information about usage of those repositories,
> likes, number of contributors, it is already available on the
> Microsoft Github "Insights" link on the MELPA page. There is no point
> in asking users what is already obvious. There are stars or likes on
> Github.

I disagree that there is no point asking. Assuming that we are
interested to know about MELPA/Marmalade usage, taking information about
usage/stars/contributors/etc from third-party sources will represent
different subset of Emacs users - it cannot be compared with other
results of the presently discussed poll.

All the MELPA/marmalade statistics is inherently biased. It only
represents Emacs users using those repositories. On the other hand,
asking about package repositories in this poll will provide us with a
concrete estimate how popular are MELPA and Mermalade. In future, when
nongnu ELPA is going to be up and running for a while, it may also be
interesting to see how the popularity changes.

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 08:34]:
>> > > Just as a reminder, here is what the email starter suggested:
>> > >
>> > > > a survey for Emacs users to better grasp the diversity and various usages
>> > > out there
>> > >
>> > > There is nothing about taking practical decisions or encouraging free
>> > > software (or anything) there.
>> >
>> > Every opinion poll survey has a purpose, normally the purpose is to
>> > find out what majority wish and want to improve the product or service
>> > and thus reach or gain more customers, strike it, users.
>> 
>> Yes it has a purpose, I quoted it above. To understand "the diversity
>> and various usages out there". Specifically excluding some popular
>> usages defeats that purpose.
>
> Tell me examples of popular usage that you refer to?
>
> Free form gives enough possibility for any user to explain anything
> they wish.
>
>> If anything, wouldn't we want to get an idea how many Emacs users
>> currently use a non-free package repository?
>
> I am not sure if there is any non-free package repository for
> Emacs.
>
> MELPA is fetching most packages from the Microsoft Github, and Github
> dictates free licenses for any public repository, most of them are
> free software. For me is hard to find particular example that uses non
> free software. 
>
> That will be work to do, to move some public packages to non-GNU ELPA.
>
> MELPA recipes can be cloned, copy of software can be placed on
> nongnu.org automatically, later revised from TODO to be TO PUBLISH,
> and distributed ethically. 
>
>> > They do provide free software naturally as packages should be GPL so
>> > far I understand (not sure), but if they wrap non free software or
>> > have pointers to non free software, such recommendation would be
>> > contrary to principles why GNU Emacs have been made as free software.
>> 
>> I don't understand why a question in a survey would be seen as a
>> recommendation.
>
> Above paragraph refers to MELPA, that could wrap non free software in
> the free software packages. I can then imagine links in packages pointing
> to non free software, that is what was meant with recommendation. It
> does not refer to questions in the opinion poll.
>
>> That's similar, in a sense, to those social surveys asking people if
>> they have done drugs. I don't think those want to encourage people to
>> take drugs.
>> 
>> > > The same warning could be used when mentioning windows, non-free
>> > > IDEs, etc.
>> >
>> > Warning you mention is used on Emacs download page, see
>> > https://www.gnu.org/software/emacs/download.html
>> 
>> Great, so there is precedent ! Why is it acceptable and sufficient on
>> the main download page but not acceptable in a survey?
>
> It is not a new precedent, it was from the creation of Emacs to talk
> about free software, and advise people. In my opinion GNU project
> should increase the marketing of free software philosophy by power of
> 10, it is not enough.
>
>> Those words are a bit too harsh when applied to Melpa, I hope that
>> we agree on that. And nobody likes to read propaganda (no matter how
>> justified) in a survey, so having such a long tirade could again
>> lead to selection bias.
>
> GNU project with promotion of free software is not biased as that
> would mean that it is influenced in an unfair way.
>
> GNU project is influenced in a fair way and thus should be promoting
> and supporting free software and helping users of proprietary software
> to understand what is free software and freedom in computing.
>
> The word propaganda you maybe used in a negative connotation, but the
> word itself means promoting information to spread some cause. Who is
> not interested, would not read it. The point of propaganda that some
> will get interested, so propaganda gives results for those who are.
>
>> But, for example, wouldn't something like below be both short and
>> explicit enough?
>> 
>> "- Melpa (Note: Emacs and the GNU project DO NOT ENDORSE package
>> repositories which encourage non-free software, see
>> https://www.gnu.org/philosophy/)"
>
> I do not think that it is necessary for survey from GNU Emacs to ask
> if people are using Melpa or whatever other software
> repository.
>
> Reason for that is that it is obvious that people do, people ARE using
> MELPA and Marmalade software repositories; AND more important reason
> not to ask is that information about usage of those repositories,
> likes, number of contributors, it is already available on the
> Microsoft Github "Insights" link on the MELPA page. There is no point
> in asking users what is already obvious. There are stars or likes on
> Github.
>
> If there is certain disagreement between GNU and MELPA, that is valid
> reason as well. 
>
> In my opinion, the team of MELPA is preparing the list of packages
> well, then GNU team working on ELPA on nongnu.org can fork all the
> software and curate and remove whatever is unsafe or not ethical and
> make a new repository.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-10-12 10:56]:
> > Reason for that is that it is obvious that people do, people ARE using
> > MELPA and Marmalade software repositories; AND more important reason
> > not to ask is that information about usage of those repositories,
> > likes, number of contributors, it is already available on the
> > Microsoft Github "Insights" link on the MELPA page. There is no point
> > in asking users what is already obvious. There are stars or likes on
> > Github.
> 
> I disagree that there is no point asking. Assuming that we are
> interested to know about MELPA/Marmalade usage, taking information about
> usage/stars/contributors/etc from third-party sources will represent
> different subset of Emacs users - it cannot be compared with other
> results of the presently discussed poll.
> 
> All the MELPA/marmalade statistics is inherently biased. It only
> represents Emacs users using those repositories. On the other hand,
> asking about package repositories in this poll will provide us with a
> concrete estimate how popular are MELPA and Mermalade. In future, when
> nongnu ELPA is going to be up and running for a while, it may also be
> interesting to see how the popularity changes.

When doing an opinion poll, it involves investment of money, time,
efforts, people, so the answers have to be evaluated with purpose to
do something.

Marmalade is anyway not working any more.

Let us say you wish to ask if users are using MELPA or what else?
There is nothing else to compare.

ELPA is already built-in, there is no other package repository. Now
you wish to find statistical information how many users use MELPA, but
why not simply ask MELPA developers to provide such statistic? They
probably have statistics of downloads or accesses to the
website. There is no point to asking around the corner, when one can
ask directly.

Let us say one wish to evaluate which packages are more popular, you
can take a list of package from MELPA, then run Emacs Lisp through the
list, and then probably use Github API or web scraper, you can find
which package has more stars/likes or number of forks. Similar results
you can get from Github search. None of results, neither survey
results nor Github API/search results are correct results, so it would
be expensive making survey in that way.

So, for number of MELPA downloads, anybody wishing to find out, one
can ask there. Another question how they are evaluating the website
statistics. But for which practical use?!

For popularity of various packages, it is possible to use API (I just
guess so) or web scraper, and obtain statistical information. But for
which practical use?!

Most practical is when user reports a bug, then it can be solved, or
user reports enhancement to software, it could be implemented.

This mailing list is a small survey itself, help-gnu-emacs mailing
list as well, various answers on non-GNU websites also represent
valuable data for improvements, at this moment there is so much data
that demands improvements.

Jean

Re: Proposal for an Emacs User Survey

[Ihor Radchenko]

> Let us say you wish to ask if users are using MELPA or what else?
> There is nothing else to compare.

At least we can get ratio between people using MELPA vs. not using
MELPA. As I stated, it might be useful later, when we have nongnu ELPA.
Or it might, for example, reveal that most of Emacs users are actually
not using MELPA (though I don't believe that it is the case).

Note that I am not arguing that we have to include this question in this
specific poll (there will be many other valuable inputs regardless of
this question). Just wanted to point out that MELPA statistics will not
answer the question what fraction of users are _not_ using MELPA. If
that fraction is significant (unlikely), taking only MELPA statistics
might be not representative.

Best,
Ihor


Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-10-12 10:56]:
>> > Reason for that is that it is obvious that people do, people ARE using
>> > MELPA and Marmalade software repositories; AND more important reason
>> > not to ask is that information about usage of those repositories,
>> > likes, number of contributors, it is already available on the
>> > Microsoft Github "Insights" link on the MELPA page. There is no point
>> > in asking users what is already obvious. There are stars or likes on
>> > Github.
>> 
>> I disagree that there is no point asking. Assuming that we are
>> interested to know about MELPA/Marmalade usage, taking information about
>> usage/stars/contributors/etc from third-party sources will represent
>> different subset of Emacs users - it cannot be compared with other
>> results of the presently discussed poll.
>> 
>> All the MELPA/marmalade statistics is inherently biased. It only
>> represents Emacs users using those repositories. On the other hand,
>> asking about package repositories in this poll will provide us with a
>> concrete estimate how popular are MELPA and Mermalade. In future, when
>> nongnu ELPA is going to be up and running for a while, it may also be
>> interesting to see how the popularity changes.
>
> When doing an opinion poll, it involves investment of money, time,
> efforts, people, so the answers have to be evaluated with purpose to
> do something.
>
> Marmalade is anyway not working any more.
>
> Let us say you wish to ask if users are using MELPA or what else?
> There is nothing else to compare.
>
> ELPA is already built-in, there is no other package repository. Now
> you wish to find statistical information how many users use MELPA, but
> why not simply ask MELPA developers to provide such statistic? They
> probably have statistics of downloads or accesses to the
> website. There is no point to asking around the corner, when one can
> ask directly.
>
> Let us say one wish to evaluate which packages are more popular, you
> can take a list of package from MELPA, then run Emacs Lisp through the
> list, and then probably use Github API or web scraper, you can find
> which package has more stars/likes or number of forks. Similar results
> you can get from Github search. None of results, neither survey
> results nor Github API/search results are correct results, so it would
> be expensive making survey in that way.
>
> So, for number of MELPA downloads, anybody wishing to find out, one
> can ask there. Another question how they are evaluating the website
> statistics. But for which practical use?!
>
> For popularity of various packages, it is possible to use API (I just
> guess so) or web scraper, and obtain statistical information. But for
> which practical use?!
>
> Most practical is when user reports a bug, then it can be solved, or
> user reports enhancement to software, it could be implemented.
>
> This mailing list is a small survey itself, help-gnu-emacs mailing
> list as well, various answers on non-GNU websites also represent
> valuable data for improvements, at this moment there is so much data
> that demands improvements.
>
> Jean

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-10-12 11:56]:
> > Let us say you wish to ask if users are using MELPA or what else?
> > There is nothing else to compare.
> 
> At least we can get ratio between people using MELPA vs. not using
> MELPA. As I stated, it might be useful later, when we have nongnu ELPA.
> Or it might, for example, reveal that most of Emacs users are actually
> not using MELPA (though I don't believe that it is the case).

Take in consideration that there are many organizations using
GNU/Linux distributions internally on many many computers, with many
many users. From time to time those users may need to use some
editors. But in general, they will not be excited for any talk we have
here, for any usage of Internet beyond their work or study. That group
of users consider would think it is immature to speak of
configurations, etc.

Debian is collecting some information of organizations using it:
https://www.debian.org/users/

Example:
https://www.debian.org/users/edu/ethz_phys

>  Swiss Federal Institute of Technology Zurich, Department of Physics, ETH Zurich, Switzerland

> We have about 160 workstations which are automatically installed
> with additional science/math/astronomical software. We also have
> about 10 servers. All machines run stable.

Now those 160 workstations with some nice software, maybe they are
used by students, I do not know, but those people whoever use them are
still users of a system, among them there will be those using GNU
Emacs, but they may not be excited for the talk we have here.

One would need to find out with them how many would be using Emacs,
and I doubt anybody would be using MELPA.

Hypothetically, university could have 160 workstations and 2000 users,
maybe some use Emacs, some not, among those who use maybe nobody use
MELPA, while for usage of ELPA, one could say "YES" because it is
embedded, but practically, maybe nobody used neither ELPA or
MELPA. And you would never find out about those 2000 users. In
addition, those 2000 users are growing each year, as there is new
number of students coming to university, those among them using Emacs
would become users of Emacs and would later probably not be using
Emacs, maybe after they finish university there are other things to
do. Yet such case would be very significant case of users of Emacs.

GNU/Linux systems are multi-user systems, one computer could have
large number of users, many would never be involved with any
development, or being proud of using Emacs, so what, it is just editor
for them or just computing environment, but they would not need to
communicate online about that.

Basically, the point of that survey is useless, as it will not reach
the true user base.

It will reach us, people who like polishing their cheap "cars", ricers
by definition, how they call computer users who polish their Window
Manager themes, icons, colors, and now Emacs. Ricers. Jargon file has
to be updated.

Rice

"Rice" is a word that is commonly used to refer to making visual
improvements and customizations on one's desktop. It was inherited
from the practice of customizing cheap Asian import cars to make them
appear to be faster than they actually were - which was also known as
"ricing". Here on /r/unixporn , the word is accepted by the majority
of the community and is used sparingly to refer to a visually
attractive desktop upgraded beyond the default.

From:
https://www.reddit.com/r/unixporn/wiki/themeing/dictionary#wiki_rice

I hope you get my point.

The survey would not be really directed to Emacs users, as the true
user base is hard to reach.

It would be directed to Emacs Ricers, people who like polishing their
Emacs.

> Note that I am not arguing that we have to include this question in this
> specific poll (there will be many other valuable inputs regardless of
> this question).

Those polls are best to be placed on Emacs website.

Or just make a package in Emacs that is similar to Debian's popularity
contest package, see statistics here:
https://popcon.debian.org/

and here is statistics for Emacs:
https://qa.debian.org/popcon.php?package=emacs

I see there about 50000 users, there could be hundreds of thousands of
users, as not everybody is using the package "popularity-contest" to
submit reports about their usage.

Vim is leading before Emacs:
https://qa.debian.org/popcon.php?package=vim

Now those statistics are indication, but not the real information, as
one has to know dynamics and definitions. Popularity contest does not
mean it is popular by use, but by number of installation.

To be popular by use is not same as to be popular by number of
installations.

One can see that on each statistics page that most popular package in
Debian is dpkg, but I really doubt that it is "most popular software
used by users" in Debian. libc6 is also most popular software, but who
knows about that? Do you see that statistics can be doubtful?

Which computer user is going around and telling how he is using libc6
on his computer, being proud, or liking to configure it for his
pleasure?! Yet the package is "popular" as number one.

Example how such popularity is blown up is that Debian package
'education-common' recommends 'vim' or other version of vim. Let us
say somebody wish to install some educational package in Debian, such
could pull 'education-common' which in turn could pull 'vim' as
recommended package, but user would not be really using 'vim', one
would be using what one wants (some other software).

Such popularity contest would counte number of installations of
packages, and so it does it automatically.

> Just wanted to point out that MELPA statistics will not answer the
> question what fraction of users are _not_ using MELPA. If that
> fraction is significant (unlikely), taking only MELPA statistics
> might be not representative.

Nothing can answer that question, as real number of Emacs users cannot
be obtained.

But hey, we can reach those who are online. Emacs Ricers.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't understand why a question in a survey would be seen as a
  > recommendation.

Simply mentioning MELPA encourages people to think we consider it
legitimate.  We do not!  And it defeats our goals if people think
that we do.

MELPA contains programs that depend on, that require installation of,
various nonfree programs.  Thus, it leads people to install those.
This encourages injustice -- the injustice of those nonfree programs.

Most Emacs users have probably never thought about this issue.  They
take for granted that nonfree programs are legitimate, they believe
everyone considers them legitimate, and they assume we do too.

It is important to educate these people that the GNU Project does not
agree with "everyone" on this.

Some Emacs users have though about this but were not convinced, and
then they probably dropped the question from their thoughts.  If we
treat MELPA as if it were as legitimate as the free archives, they
will be encouraged in considering it as legitimate as the free
archives.

Perhaps we should do something to show them what the issue is.
Perhaps with text like this:

======================================================================
GNU Emacs is part of the GNU operating system.  The GNU Project aims
to escape, then replace, all nonfree software with freedom-respecting
free software.

We maintain an Emacs package archive that consists of free packages
that can run in a free environment.  There is another Emacs package
archive called Melpa which distributes packages require the user to
install some nonfree programs.  That practice is harmful because it
encourages the installation of those nonfree programs, and that works
against our goal -- it prioritizes short-term convenience over
freedom.

Rather than prioritize short-term convenience over freedom, we
normally avoid mentioning that Malpa exists.

As a special exception, we mentioned it in a question in this survey,
to get data about its use.  That does not mean we have changed our
minds about our criticism of Melpa.
======================================================================

How about that?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

It is good to develop verified, safe, separate repository at
elpa.nongnu.org that would be included in Emacs as default, so that
people can get the safe, verified packages from repository that takes
care of users and help them understand freedom issues.

For MELPA warning it is good to be specific, as if there is some
wrapped proprietary software on MELPA, then let us find out which
software it is, and one can make a list, or even ask MELPA not to
include such.

Which one it is exactly?

The list of packages that wrap proprietary software would be anyway
required, if GNU wish to publish elpa.nongnu.org

Above is based on below.

* Richard Stallman <rms@gnu.org> [2020-10-13 06:53]:
> Some Emacs users have though about this but were not convinced, and
> then they probably dropped the question from their thoughts.  If we
> treat MELPA as if it were as legitimate as the free archives, they
> will be encouraged in considering it as legitimate as the free
> archives.
> 
> Perhaps we should do something to show them what the issue is.
> Perhaps with text like this:
> 
> ======================================================================
> GNU Emacs is part of the GNU operating system.  The GNU Project aims
> to escape, then replace, all nonfree software with freedom-respecting
> free software.
> 
> We maintain an Emacs package archive that consists of free packages
> that can run in a free environment.  There is another Emacs package
> archive called Melpa which distributes packages require the user to
> install some nonfree programs.  That practice is harmful because it
> encourages the installation of those nonfree programs, and that works
> against our goal -- it prioritizes short-term convenience over
> freedom.
> 
> Rather than prioritize short-term convenience over freedom, we
> normally avoid mentioning that Malpa exists.
> 
> As a special exception, we mentioned it in a question in this survey,
> to get data about its use.  That does not mean we have changed our
> minds about our criticism of Melpa.
> ======================================================================

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For MELPA warning it is good to be specific, as if there is some
  > wrapped proprietary software on MELPA, then let us find out which
  > software it is, and one can make a list, or even ask MELPA not to
  > include such.

  > Which one it is exactly?

I don't know, but probably someone here knows of some examples.

I have a vague recollection, though, that some of us already asked the
maintainers of Melpa to make this change, and they did not want to.

Does anyone actually know?

  > The list of packages that wrap proprietary software would be anyway
  > required, if GNU wish to publish elpa.nongnu.org

Yes and no.  We would not try to import all packages from MELPA except
those that were excluded for this reason.  Rather, we would need to
examine each package to decide whether to import it, and whether and
how change it.  In the process we would see if the package has any
problems that might be reasons not to import it.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-15 07:00]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > For MELPA warning it is good to be specific, as if there is some
>   > wrapped proprietary software on MELPA, then let us find out which
>   > software it is, and one can make a list, or even ask MELPA not to
>   > include such.
> 
>   > Which one it is exactly?
> 
> I don't know, but probably someone here knows of some examples.
> 
> I have a vague recollection, though, that some of us already asked the
> maintainers of Melpa to make this change, and they did not want to.
> 
> Does anyone actually know?

Maybe these:

- lastpass, LastPass proprietary software command wrapper, it probably
  interacts with proprietary software from Emacs, inciting users to
  try out or use or install LastPass

- sharper, dotnet CLI wrapper, I am just blindly assuming that dotnet
  CLI is Microsoft proprietary programming language

Jean

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Warning you mention is used on Emacs download page, see
  > > https://www.gnu.org/software/emacs/download.html

  > Great, so there is precedent ! Why is it acceptable and sufficient on
  > the main download page but not acceptable in a survey?

I am not sure what those words mean.  I will look at that page
and see if I can figure it out.  Would people like to explain what
part or aspect of that page is at issue here?
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-13 06:53]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > Warning you mention is used on Emacs download page, see
>   > > https://www.gnu.org/software/emacs/download.html
> 
>   > Great, so there is precedent ! Why is it acceptable and sufficient on
>   > the main download page but not acceptable in a survey?
> 
> I am not sure what those words mean.  I will look at that page
> and see if I can figure it out.  Would people like to explain what
> part or aspect of that page is at issue here?

You missed to follow the thread. Discussion was about warning users
about non-free software in the survey itself.

On that page, we are warning users with below wording.


Nonfree systems

The reason for GNU Emacs's existence is to provide a powerful editor for the GNU operating system. Versions of GNU, such as GNU/Linux, are the primary
platforms for Emacs development. 

However, GNU Emacs includes support for some other systems that volunteers choose to support.

The purpose of the GNU system is to give users the freedom that proprietary software takes away from its users. Proprietary operating systems (like other
proprietary programs) are an injustice, and we aim for a world in which they do not exist. 

To improve the use of proprietary systems is a misguided goal. Our aim, rather, is to eliminate them. We include support for some proprietary systems in GNU
Emacs in the hope that running Emacs on them will give users a taste of freedom and thus lead them to free themselves. 

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > On that page, we are warning users with below wording.

Yes.  I wrote those words.  It is crucial to have them there, because
we mention Windows and MacOS.  Where we mention them, we should explain
why they are unjust -- because otherwise people will assume we share
the usual amoral attitude towards them.

I think of it as a denunciation, not a warning.

Why is it ok to mention Windows and MacOS there?  Because everyone
using a computer already knows about them.  See node References
in the GNU Coding Standards (https://www.gnu.org/prep/standards/).

I hope that only a minority of Emacs users know about MELPA, and I'd
rather not inform the rest about it.  But if something is going to
inform them anyway, it is better to do it with a denunciation.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Thu, 15 Oct 2020 23:59:07 -0400
Richard Stallman <rms@gnu.org> wrote:

> I hope that only a minority of Emacs users know about MELPA, and I'd
> rather not inform the rest about it.  But if something is going to
> inform them anyway, it is better to do it with a denunciation.


I've been using Emacs (and MELPA) for the best part of a decade and
knew nothing about this! I'm concerned to use only free software and
actively avoid proprietary software, so this is a bit of a shock.

Is there anywhere I can read more about this issue?

I think this illustrates well why any initiative related with GNU (even
by association) should avoid mentioning proprietary software without a
clear warning and should, in fact, do its utmost to educate users about
free software.

I've used Trisquel, Parabola and Replicant for many years, and in all
of those cases, installing proprietary software is purposely made
difficult (For example, using a `your-freedom' package in Parabola. It
makes sense to force a user to actively remove `your-freedom' if he or
she wants to install a proprietary program; difficult to do so by
mistake.

In fact, I would go the extra mile and say Emacs should expressly warn
users over the dangers of installing proprietary software from
unofficial repositories (by the way, I always just assumed MELPA was
somehow official and related to ELPA, because its name is so similar to
ELPA). This survey could provide a good opportunity for such
information/education.

Over the years, as I worked on my emacs configuration, I read many
other confiruations, but none (that I recall) gave any warnings about
MELPA, so I never thought twice about it.

Any other caveats regarding unwittingly running proprietary software
inside of Emacs?

Sorry to derail the conversation.

Marcel

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 08:03, Marcel Ventosa <mve1@runbox.com> a écrit :
>
> On Thu, 15 Oct 2020 23:59:07 -0400
> Richard Stallman <rms@gnu.org> wrote:
>
> > I hope that only a minority of Emacs users know about MELPA, and I'd
> > rather not inform the rest about it.  But if something is going to
> > inform them anyway, it is better to do it with a denunciation.
>
>
> I've been using Emacs (and MELPA) for the best part of a decade and
> knew nothing about this! I'm concerned to use only free software and
> actively avoid proprietary software, so this is a bit of a shock.

As I understand it, Melpa packages cannot *be* or *install* non-free
software. But some will not work without such software, which can in
theory encourage users to install it.

So unless you yourself installed non-free software, Melpa cannot have
made your Emacs configuration non-free by accident.

> In fact, I would go the extra mile and say Emacs should expressly warn
> users over the dangers of installing proprietary software from
> unofficial repositories (by the way, I always just assumed MELPA was
> somehow official and related to ELPA, because its name is so similar to
> ELPA). This survey could provide a good opportunity for such
> information/education.

ELPA means Emacs Lisp Package Archive, so both Melpa and GNU Elpa are ELPA's.
I think that commonly referring to GNU Elpa as simply Elpa (which I am
also guilty of) is a bigger source of confusion than Melpa and GNU
Elpa sharing the same suffix.

GNU Elpa and the future Non-GNU Elpa are (will be) activated by
default as package archives, Melpa is not.
The hope is that once 99% of the packages by the community are
available in an archive activated by default, users will not rush to
install Melpa in the same proportions as today.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Fri, 16 Oct 2020 08:52:49 +0200
Thibaut Verron <thibaut.verron@gmail.com> wrote:

> Le ven. 16 oct. 2020 à 08:03, Marcel Ventosa <mve1@runbox.com> a
> écrit :
> >
> > On Thu, 15 Oct 2020 23:59:07 -0400
> > Richard Stallman <rms@gnu.org> wrote:
> >  
> > > I hope that only a minority of Emacs users know about MELPA, and
> > > I'd rather not inform the rest about it.  But if something is
> > > going to inform them anyway, it is better to do it with a
> > > denunciation.  
> >
> >
> > I've been using Emacs (and MELPA) for the best part of a decade and
> > knew nothing about this! I'm concerned to use only free software and
> > actively avoid proprietary software, so this is a bit of a shock.  
> 
> As I understand it, Melpa packages cannot *be* or *install* non-free
> software. But some will not work without such software, which can in
> theory encourage users to install it.
> 
> So unless you yourself installed non-free software, Melpa cannot have
> made your Emacs configuration non-free by accident.

I understand, thanks for the explanation. In that case, I think I'm
well informed enough to have avoived the dangers. I wonder how many
people are not. AFAIK, both Trisquel and Parabola keep out packages
that recommend or encourage proprietary software, which seems essential
to protect users' freedom. 

> > In fact, I would go the extra mile and say Emacs should expressly
> > warn users over the dangers of installing proprietary software from
> > unofficial repositories (by the way, I always just assumed MELPA was
> > somehow official and related to ELPA, because its name is so
> > similar to ELPA). This survey could provide a good opportunity for
> > such information/education.  
> 
> ELPA means Emacs Lisp Package Archive, so both Melpa and GNU Elpa are
> ELPA's. I think that commonly referring to GNU Elpa as simply Elpa
> (which I am also guilty of) is a bigger source of confusion than
> Melpa and GNU Elpa sharing the same suffix.

As written on the MELPA Github about page:

"MELPA is Milkypostman's ELPA or Milkypostman's Experimental Lisp
Package Archive if you're not into the whole brevity thing." 

Perhaps the shared `E' in `Elpa' is purely coincidental?

> GNU Elpa and the future Non-GNU Elpa are (will be) activated by
> default as package archives, Melpa is not.
> The hope is that once 99% of the packages by the community are
> available in an archive activated by default, users will not rush to
> install Melpa in the same proportions as today.

This will be good news! In the meantime, I wonder how many others are
unaware of the potential dangers of using MELPA. A prominent place to
inform Emacs users about these dangers would be useful; perhaps this
survey will provide a good first opportunity. In any case, I'm thankful
to RMS and this survey related conversation for an important discovery. 

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 09:24, Marcel Ventosa <mve1@runbox.com> a écrit :
>
> On Fri, 16 Oct 2020 08:52:49 +0200
> Thibaut Verron <thibaut.verron@gmail.com> wrote:
>
> > Le ven. 16 oct. 2020 à 08:03, Marcel Ventosa <mve1@runbox.com> a
> > écrit :
> > >
> > > On Thu, 15 Oct 2020 23:59:07 -0400
> > > Richard Stallman <rms@gnu.org> wrote:
> > >
> > > > I hope that only a minority of Emacs users know about MELPA, and
> > > > I'd rather not inform the rest about it.  But if something is
> > > > going to inform them anyway, it is better to do it with a
> > > > denunciation.
> > >
> > >
> > > I've been using Emacs (and MELPA) for the best part of a decade and
> > > knew nothing about this! I'm concerned to use only free software and
> > > actively avoid proprietary software, so this is a bit of a shock.
> >
> > As I understand it, Melpa packages cannot *be* or *install* non-free
> > software. But some will not work without such software, which can in
> > theory encourage users to install it.
> >
> > So unless you yourself installed non-free software, Melpa cannot have
> > made your Emacs configuration non-free by accident.
>
> I understand, thanks for the explanation. In that case, I think I'm
> well informed enough to have avoived the dangers. I wonder how many
> people are not.

I personally don't think many users install non-free software because
they saw it wrapped in a Melpa package.

Taking the example of emacs-lastpass given above, I don't see how
anyone would even find this package without searching for it with the
keyword "lastpass".

The audience, rather, is users who are currently using Lastpass in
their browsers but are interested in bringing some of their online
activities to Emacs, but rely on their password manager to do so.

In due time, with the new "taste of freedom", they might even switch
to Keepass or Bitwarden (note: I don't use a password manager in
Emacs, so I have no idea of the quality of each support package,
beyond their existence), but in any case, I believe that for those
users, the existence of transitional solutions is a good thing.

I absolutely support the fact that Melpa is not activated by default,
and that there should be a warning about the existence of those
packages everywhere possible. But I still consider that the value of
those packages outweigh their dangers, just like the win32 build of
Emacs.

> > ELPA means Emacs Lisp Package Archive, so both Melpa and GNU Elpa are
> > ELPA's. I think that commonly referring to GNU Elpa as simply Elpa
> > (which I am also guilty of) is a bigger source of confusion than
> > Melpa and GNU Elpa sharing the same suffix.
>
> As written on the MELPA Github about page:
>
> "MELPA is Milkypostman's ELPA or Milkypostman's Experimental Lisp
> Package Archive if you're not into the whole brevity thing."
>
> Perhaps the shared `E' in `Elpa' is purely coincidental?

I didn't know about the Experimental version, thanks. If anything, the
quoted sentence is self-contradicting for me.
Also, the title of the melpa.org page is "MELPA (Milkypostman’s Emacs
Lisp Package Archive)".

In any case, I don't know if an acronym with a different vowel but
still ending with LPA, such as Milpa, would be less confusing.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Fri, 16 Oct 2020 09:53:22 +0200
Thibaut Verron <thibaut.verron@gmail.com> wrote:

> Le ven. 16 oct. 2020 à 09:24, Marcel Ventosa <mve1@runbox.com> a
> écrit :
> >
> > On Fri, 16 Oct 2020 08:52:49 +0200
> > Thibaut Verron <thibaut.verron@gmail.com> wrote:
> >  
> > > Le ven. 16 oct. 2020 à 08:03, Marcel Ventosa <mve1@runbox.com> a
> > > écrit :  
> > > >
> > > > On Thu, 15 Oct 2020 23:59:07 -0400
> > > > Richard Stallman <rms@gnu.org> wrote:
> > > >  
> > > > > I hope that only a minority of Emacs users know about MELPA,
> > > > > and I'd rather not inform the rest about it.  But if
> > > > > something is going to inform them anyway, it is better to do
> > > > > it with a denunciation.  
> > > >
> > > >
> > > > I've been using Emacs (and MELPA) for the best part of a decade
> > > > and knew nothing about this! I'm concerned to use only free
> > > > software and actively avoid proprietary software, so this is a
> > > > bit of a shock.  
> > >
> > > As I understand it, Melpa packages cannot *be* or *install*
> > > non-free software. But some will not work without such software,
> > > which can in theory encourage users to install it.
> > >
> > > So unless you yourself installed non-free software, Melpa cannot
> > > have made your Emacs configuration non-free by accident.  
> >
> > I understand, thanks for the explanation. In that case, I think I'm
> > well informed enough to have avoived the dangers. I wonder how many
> > people are not.  
> 
> I personally don't think many users install non-free software because
> they saw it wrapped in a Melpa package.
> 
> Taking the example of emacs-lastpass given above, I don't see how
> anyone would even find this package without searching for it with the
> keyword "lastpass".
> 
> The audience, rather, is users who are currently using Lastpass in
> their browsers but are interested in bringing some of their online
> activities to Emacs, but rely on their password manager to do so.
> 
> In due time, with the new "taste of freedom", they might even switch
> to Keepass or Bitwarden (note: I don't use a password manager in
> Emacs, so I have no idea of the quality of each support package,
> beyond their existence), but in any case, I believe that for those
> users, the existence of transitional solutions is a good thing.

I understand your point, and transitional solutions may indeed be a good
thing (though they can lead both ways). However, it's a long-standing
position of GNU not to be seen to endorse these compromises, whether or
not their existence is a good thing. For me, the name GNU has always
signified a free software safe haven. The idea that one might be misled
into installing proprietary software because of well earned trust in GNU
should be avoided at all costs.

I would posit the same if I was a member of a vegan organization that
was relaxing their views on eating animals to ease the transition. While
there is little doubt transitions can be beneficial, the vegan
organization should not confuse it's tenets.

Perhaps. Transitional solutions go both ways though. A cursory glance at
Reddit's Emacs group is enough to notice not only ignorance of the
philosophy behind GNU, but quite recurrent mockery of what it stands
for. Usually in the form of deriding RMS. For the most recent example,
one user comments under abrochard's survey post: "So, will you be
censoring the survey to maintain ideological purity, like rms
insisted?", to which abrochard responds: "I agree with you. The
discussion around Melpa is a big factor as to why the survey is
happening in parallel to the gnu project."

> I absolutely support the fact that Melpa is not activated by default,
> and that there should be a warning about the existence of those
> packages everywhere possible. But I still consider that the value of
> those packages outweigh their dangers, just like the win32 build of
> Emacs.
> 
> > > ELPA means Emacs Lisp Package Archive, so both Melpa and GNU Elpa
> > > are ELPA's. I think that commonly referring to GNU Elpa as simply
> > > Elpa (which I am also guilty of) is a bigger source of confusion
> > > than Melpa and GNU Elpa sharing the same suffix.  
> >
> > As written on the MELPA Github about page:
> >
> > "MELPA is Milkypostman's ELPA or Milkypostman's Experimental Lisp
> > Package Archive if you're not into the whole brevity thing."
> >
> > Perhaps the shared `E' in `Elpa' is purely coincidental?  
> 
> I didn't know about the Experimental version, thanks. If anything, the
> quoted sentence is self-contradicting for me.
> Also, the title of the melpa.org page is "MELPA (Milkypostman’s Emacs
> Lisp Package Archive)".
> 
> In any case, I don't know if an acronym with a different vowel but
> still ending with LPA, such as Milpa, would be less confusing.

Yes, I see it as a problem when an unofficial offshoot of a project does
not make it crystal clear that it is so. In fact, and on the same topic,
a post was made about the survey yesterday on Reddit by Abrochard with
the title "The Emacs User Survey 2020 will open on Oct 19th," which
makes it sound as though it were an official survey. Further down the
thread, a user criticized RMS of trying to censor the survey, mentioning
the MELPA discussion in particular, to which Abrochard responded: "I
agree with you. The discussion around Melpa is a big factor as to why
the survey is happening in parallel to the gnu project."

I'm not saying the obfuscation is purposeful either in the case of
`MELPA' imitating the name of the existing `GNU ELPA', or of Adrien
calling his survey *The* Emacs User survey", but what I do think is that
all non-GNU initiatives that affect perception of GNU, particularly the
ones that clearly do not share the GNU philosophy (the survey referred
to `GNU/Linux' as `Linux', for example), would seem much more
transparent if they were very clearly and visibly labeled as unofficial.
My hope, of course, would be that these initiatives could respect the
GNU philosophy, even if they did not share it.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 11:25, Marcel Ventosa wrote:

> Perhaps. Transitional solutions go both ways though. A cursory glance at
> Reddit's Emacs group is enough to notice not only ignorance of the
> philosophy behind GNU, but quite recurrent mockery of what it stands
> for. Usually in the form of deriding RMS. For the most recent example,
> one user comments under abrochard's survey post: "So, will you be
> censoring the survey to maintain ideological purity, like rms
> insisted?", to which abrochard responds: "I agree with you. The
> discussion around Melpa is a big factor as to why the survey is
> happening in parallel to the gnu project."

Do you think the "mockery" is entirely without merit?

It's not even so much real mockery as probably the only way the user 
could describe the conflict.

When a survey purposefully omits a well-known and popular option, it is 
going to discount a sizable portion of our community, and ignore a 
project that has done a lot to popularize Emacs over the years.

I think it's both insulting to its developers, and stinks of thought 
police. Far from the idea of user freedom I hope to expect from GNU and FSF.

> I'm not saying the obfuscation is purposeful either in the case of
> `MELPA' imitating the name of the existing `GNU ELPA', or of Adrien
> calling his survey *The* Emacs User survey", but what I do think is that
> all non-GNU initiatives that affect perception of GNU, particularly the
> ones that clearly do not share the GNU philosophy (the survey referred
> to `GNU/Linux' as `Linux', for example), would seem much more
> transparent if they were very clearly and visibly labeled as unofficial.
> My hope, of course, would be that these initiatives could respect the
> GNU philosophy, even if they did not share it.

Calling it "THE Emacs User survey" is perhaps unfortunate. But likewise 
it is unfortunate how Emacs leadership does little to follow the 
external, "unofficial" polls.

A survey doesn't have to be "the official GNU survey" to be useful.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Fri, 16 Oct 2020 15:17:16 +0300
Dmitry Gutov <dgutov@yandex.ru> wrote:

> Do you think the "mockery" is entirely without merit?

Yes, and I think mockery precludes understanding, particularly as we
are dealing with a thoughtful man and a well developed and time tested
philosophy, successfully put into practice against the odds.

> ignore a project that has done a lot to popularize Emacs over the years.

I fail to understand the narrative that pushes popularity over all else.
What would be the use of making Emacs the most popular editor if it
discarded the philosophy that brought it about?

I've noticed a trend of speaking about Emacs and other free software
projects as if they were "commodities" and "products," but as I see
them, it is precisely because they are community driven projects that
they are not "commodities" or "products".

> I think it's both insulting to its developers, and stinks of thought 
> police. Far from the idea of user freedom I hope to expect from GNU and FSF.

You are conflating freedom, as in the freedom to do whatever you want,
with software freedom as defined by GNU, which succintly means that
users have the freedom to run, copy, distribute, study, change and
improve software. 

I don't understand how refusing to draw attention to a repository that
recommends proprietary software turns anyone into the "thought police".

I see no reason why the GNU project or the FSF should raise awareness of
solutions that go against their founding philosophy, except to raise
awareness to its dangers.

Further up the list, I read RMS suggesting mentioning MELPA with a
disclaimer and warning about its use. 

In fact, one of the most worrying aspects of this survey idea, as I see
it, is the suggested use of non-free Javascript to implement it.

> it is unfortunate how Emacs leadership does little to follow the 
> external, "unofficial" polls.

What do you mean by this?

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 16:45, Marcel Ventosa wrote:
> On Fri, 16 Oct 2020 15:17:16 +0300
> Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
>> Do you think the "mockery" is entirely without merit?
> 
> Yes, and I think mockery precludes understanding, particularly as we
> are dealing with a thoughtful man and a well developed and time tested
> philosophy, successfully put into practice against the odds.

Shutting our eyes to actual user behavior also precludes understanding.

>> ignore a project that has done a lot to popularize Emacs over the years.
> 
> I fail to understand the narrative that pushes popularity over all else.

Strawman.

> What would be the use of making Emacs the most popular editor if it
> discarded the philosophy that brought it about?

It doesn't.

And picking on 2-3 "ideologically impure" packages (out of several 
thousands!) that are distributed on MELPA is counter-productive.

>> I think it's both insulting to its developers, and stinks of thought
>> police. Far from the idea of user freedom I hope to expect from GNU and FSF.
> 
> You are conflating freedom, as in the freedom to do whatever you want,

I don't. But a certain freedom of thought, knowledge and discussion is 
necessary, as should be apparent to any educated individual.

> I don't understand how refusing to draw attention to a repository that
> recommends proprietary software turns anyone into the "thought police".

It's a *survey*! A survey is supposed to gather insight into what users 
do, and what they need. Not shape their behavior.

You can't be effective at affecting change anyway, if you don't know 
what's going on outside.

> Further up the list, I read RMS suggesting mentioning MELPA with a
> disclaimer and warning about its use.

That didn't seem to be the preferred option, in RMS's opinion.

> In fact, one of the most worrying aspects of this survey idea, as I see
> it, is the suggested use of non-free Javascript to implement it.

Didn't Philip show a prototype that didn't use JavaScript?

>> it is unfortunate how Emacs leadership does little to follow the
>> external, "unofficial" polls.
> 
> What do you mean by this?

I don't recall any single change in Emacs' behavior that resulted from 
an external poll or survey.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Fri, 16 Oct 2020 17:04:22 +0300
Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 16.10.2020 16:45, Marcel Ventosa wrote:
> > On Fri, 16 Oct 2020 15:17:16 +0300
> > Dmitry Gutov <dgutov@yandex.ru> wrote:
> >   
> >> Do you think the "mockery" is entirely without merit?  
> > 
> > Yes, and I think mockery precludes understanding, particularly as we
> > are dealing with a thoughtful man and a well developed and time
> > tested philosophy, successfully put into practice against the odds.
> >  
> 
> Shutting our eyes to actual user behavior also precludes
> understanding.

From what I just read from Thibaut, a free software compatible solution
to replace MELPA is underway. Refusing to draw attention to something is
not "shutting our eyes".

> >> ignore a project that has done a lot to popularize Emacs over the
> >> years.  
> > 
> > I fail to understand the narrative that pushes popularity over all
> > else.  
> 
> Strawman.

I thought your argument was popularity, something that keeps coming up
in these kinds of discussions. What was your argument?

> > What would be the use of making Emacs the most popular editor if it
> > discarded the philosophy that brought it about?  
> 
> It doesn't.
> 
> And picking on 2-3 "ideologically impure" packages (out of several 
> thousands!) that are distributed on MELPA is counter-productive.

We could turn this argument around and ask why the developers who
maintain MELPA don't remove `2-3' packages that promote non-free
software. What came first, the GNU Emacs or the MELPA?

> >> I think it's both insulting to its developers, and stinks of
> >> thought police. Far from the idea of user freedom I hope to expect
> >> from GNU and FSF.  
> > 
> > You are conflating freedom, as in the freedom to do whatever you
> > want,  
> 
> I don't. But a certain freedom of thought, knowledge and discussion
> is necessary, as should be apparent to any educated individual.
> 
> > I don't understand how refusing to draw attention to a repository
> > that recommends proprietary software turns anyone into the "thought
> > police".  
> 
> It's a *survey*! A survey is supposed to gather insight into what
> users do, and what they need. Not shape their behavior.
> 
> You can't be effective at affecting change anyway, if you don't know 
> what's going on outside.

Indeed. As I recall, RMS suggested open questions instead of multiple
choice questions that "shape their behavior". With open questions, there
is no need to mention MELPA at all in fact. With open questions, the
insights that could be derived would be much more interesting.

> > Further up the list, I read RMS suggesting mentioning MELPA with a
> > disclaimer and warning about its use.  
> 
> That didn't seem to be the preferred option, in RMS's opinion.
> 
> > In fact, one of the most worrying aspects of this survey idea, as I
> > see it, is the suggested use of non-free Javascript to implement
> > it.  
> 
> Didn't Philip show a prototype that didn't use JavaScript?

That's very good news if the issue has been settled.

> >> it is unfortunate how Emacs leadership does little to follow the
> >> external, "unofficial" polls.  
> > 
> > What do you mean by this?  
> 
> I don't recall any single change in Emacs' behavior that resulted
> from an external poll or survey.

Why should Emacs development be guided by (external) survey results? I
would think it should be guided, for the most part, by what the people
putting their time into it want to create, within the principles of the
philosophy of the project and its goals. Also, anyone can suggest
changes and convince the maintainers that these changes are in the best
interest of the project (and contribute the actual changes if they are
accepted).

If they are not, Emacs makes it quite simple to implement changes for
personal "improvements". I have written functions that serve me
personally and change the behavior of Emacs to suit my needs. There are
limits to what I can do, which could be pushed if I dedicated a greater
effort to do so. How is that unfair?

If I thought one of my changes could benefit the community at large, I
would approach the maintainers and suggest it. If they disagreed with my
view, I could publish the code and could still share it with anyone
willing to run it.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> > And picking on 2-3 "ideologically impure" packages (out of several
> > thousands!) that are distributed on MELPA is counter-productive.
>
> We could turn this argument around and ask why the developers who
> maintain MELPA don't remove `2-3' packages that promote non-free
> software.

I sincerely hope it doesn't happen. Those packages might rely on
non-free software, but they are still packages that some users find
valuable or even vital, and that they would want to find somewhere
else if not available on Melpa. Removing them from Melpa would only
move the "problem".

Also, is the goal of the GNU project to deny users the right to run
the software they want or need? Or is it to give them a choice of free
options to replace non-free software?

Having those packages removed would make it more difficult to use
non-free software, but it would do nothing to improve the power of
free software.

> What came first, the GNU Emacs or the MELPA?

What does it have to do with the question?

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   > > And picking on 2-3 "ideologically impure" packages (out of several
   > > thousands!) that are distributed on MELPA is counter-productive.
   >
   > We could turn this argument around and ask why the developers who
   > maintain MELPA don't remove `2-3' packages that promote non-free
   > software.

   I sincerely hope it doesn't happen. 

We should all hope that it does, since such software is bad, one way
that is being done is with the non-GNU ELPA archive.

   Also, is the goal of the GNU project to deny users the right to run
   the software they want or need? Or is it to give them a choice of
   free options to replace non-free software?

The goal of the GNU project is to replace all non-free sofware, since
any non-free software is unjust.  That means, we do not want to
recommend users to install any kind of non-free software.

This isn't the same thing as prohibiting users what they can do on
their own accord, but rather that the project goal is to eliminate an
injustice being done to all computer users.  And with that means that
we do not want to do anything that would work against our goals.

   Having those packages removed would make it more difficult to use
   non-free software, but it would do nothing to improve the power of
   free software.

That may be so, but recommending users to use non-free software would
be worse -- it would be immoral and unethical.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 17:51, Alfred M. Szmidt <ams@gnu.org> a écrit :
>
>    > > And picking on 2-3 "ideologically impure" packages (out of several
>    > > thousands!) that are distributed on MELPA is counter-productive.
>    >
>    > We could turn this argument around and ask why the developers who
>    > maintain MELPA don't remove `2-3' packages that promote non-free
>    > software.
>
>    I sincerely hope it doesn't happen.
>
> We should all hope that it does, since such software is bad, one way
> that is being done is with the non-GNU ELPA archive.

I hope that once the non-GNU Elpa archive exists, those non-free
packages will still live on Melpa, and will die their just death,
once, and not before, a free alternative exists.

>
>    Also, is the goal of the GNU project to deny users the right to run
>    the software they want or need? Or is it to give them a choice of
>    free options to replace non-free software?
>
> The goal of the GNU project is to replace all non-free sofware, since
> any non-free software is unjust.  That means, we do not want to
> recommend users to install any kind of non-free software.
>
> This isn't the same thing as prohibiting users what they can do on
> their own accord, but rather that the project goal is to eliminate an
> injustice being done to all computer users.  And with that means that
> we do not want to do anything that would work against our goals.

The message I was answering to suggested asking the maintainers of
Melpa to remove some packages.

This is not about what the FSF and the GNU project do or don't do, or
even recommend, this is about telling an Emacs user what to put in
their privately maintained repository.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

Asking to for non-free software to be removed is a laudable thing to
do and it is something I hope more people will do, but it is not
telling someone what they should or shouldn't do.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 18:16, Alfred M. Szmidt <ams@gnu.org> a écrit :
>
> Asking to for non-free software to be removed is a laudable thing to
> do and it is something I hope more people will do, but it is not
> telling someone what they should or shouldn't do.

Would you mind clarifying the difference between the two? As it is,
your message reads like doublethink to me.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   > Asking to for non-free software to be removed is a laudable thing to
   > do and it is something I hope more people will do, but it is not
   > telling someone what they should or shouldn't do.

   Would you mind clarifying the difference between the two? 

Could you please get me a cup of tea?

I want a cup of tea, get me one.

Is there anyone who wants to implement frobnitz in Emacs?

You must implement frobnitz in Emacs otherwise we will start using vi.

   As it is, your message reads like doublethink to me.

That is unfriendly to say the least.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 18:50, Alfred M. Szmidt <ams@gnu.org> a écrit :
>
>    > Asking to for non-free software to be removed is a laudable thing to
>    > do and it is something I hope more people will do, but it is not
>    > telling someone what they should or shouldn't do.
>
>    Would you mind clarifying the difference between the two?
>
> Could you please get me a cup of tea?
>
> I want a cup of tea, get me one.
>
> Is there anyone who wants to implement frobnitz in Emacs?
>
> You must implement frobnitz in Emacs otherwise we will start using vi.

So the difference is to say "please" and be polite when we ask them to
remove some packages?

What if they say no? Does it change our opinion of them?

> >    As it is, your message reads like doublethink to me.
> That is unfriendly to say the least.

I regret my choice of words, my intention was not to be offensive.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]


   Le ven. 16 oct. 2020 à 18:50, Alfred M. Szmidt <ams@gnu.org> a écrit :
   >
   >    > Asking to for non-free software to be removed is a laudable thing to
   >    > do and it is something I hope more people will do, but it is not
   >    > telling someone what they should or shouldn't do.
   >
   >    Would you mind clarifying the difference between the two?
   >
   > Could you please get me a cup of tea?
   >
   > I want a cup of tea, get me one.
   >
   > Is there anyone who wants to implement frobnitz in Emacs?
   >
   > You must implement frobnitz in Emacs otherwise we will start using vi.

   So the difference is to say "please" and be polite when we ask them to
   remove some packages?

That is the differences between asking someone if they can do
something for you, and giving them an order ... so yes?  

   What if they say no? Does it change our opinion of them?

If someone brings you a cup of tea, or does something for you, don't
you hold them to a slightly higher regard?

And if someone, or a project, removes something that we find
problematic, instead of maybe insisting on keeping it around, don't
you think that also changes the overal perception between the two
parties?

If they say no, then they said no.  Maybe one can convince them of
otherwise, but trying to force their hand would be unkind and unfair.
Just like it is unkind to try and force the GNU project to for example
recommend Melpa in Emacs since it would go against our principles.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 19:32, Alfred M. Szmidt <ams@gnu.org> a écrit :
>    So the difference is to say "please" and be polite when we ask them to
>    remove some packages?
>
> That is the differences between asking someone if they can do
> something for you, and giving them an order ... so yes?

Okay, with the slight difference that clearly they *can* do it:
removing a package is a trivial action. The question is really whether
they *want* to do it.

> If they say no, then they said no.  Maybe one can convince them of
> otherwise, but trying to force their hand would be unkind and unfair.

Okay, as long as their saying no would not immediately make them as
"bad" as the few packages they refuse to remove.

I still personally do not wish for them to remove those packages, and
I explained why earlier, but that's just me.

> Just like it is unkind to try and force the GNU project to for example
> recommend Melpa in Emacs since it would go against our principles.

As far as I can tell, nobody in this conversation is asking for that.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   >    So the difference is to say "please" and be polite when we ask them to
   >    remove some packages?
   >
   > That is the differences between asking someone if they can do
   > something for you, and giving them an order ... so yes?

   Okay, with the slight difference that clearly they *can* do it:
   removing a package is a trivial action. The question is really whether
   they *want* to do it.

Indeed.

   > If they say no, then they said no.  Maybe one can convince them of
   > otherwise, but trying to force their hand would be unkind and unfair.

   Okay, as long as their saying no would not immediately make them as
   "bad" as the few packages they refuse to remove.

There is are large difference between calling them, which I am
guessing you mean those who maintain a project, and the actions, or
the situation bad.

   > Just like it is unkind to try and force the GNU project to for example
   > recommend Melpa in Emacs since it would go against our principles.

   As far as I can tell, nobody in this conversation is asking for that.

When the discussion was raised before about a non-GNU ELPA, that was
exactly what being demanded.  And it is what is being demanded by
having an Emacs survey that mentions MELPA for example.  Maybe demand
is slightly strong, but several harsh words have been used in
describing why the GNU project does not wish to do that.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We could turn this argument around and ask why the developers who
  > > maintain MELPA don't remove `2-3' packages that promote non-free
  > > software.

  > I sincerely hope it doesn't happen. Those packages might rely on
  > non-free software, but they are still packages that some users find
  > valuable or even vital, and that they would want to find somewhere
  > else if not available on Melpa. Removing them from Melpa would only
  > move the "problem".

I think you have picked up your values (your basis of judging what is
good or bad) from what most people think, and that you took for granted
we have the same values.  But we don't.

The goal of the GNU Project is not simply "to help users."  It is to
help users _escape from nonfree software_ (des logiciels pas libres), and
ultimately to build a world where all software is free.  Nonfree
software is an injustice -- nonfree software subjugates users.
Our goal is to _eradicate it_.

See https://gnu.org/philosophy/free-software-even-more-important.html
and fsf.org/tedx for explanation of these ideas.

Suppose that a user finds some nonfree software because a Melpa
package refers to it.  Suppose the user considers that nonfree
software very useful.  What should we say about that?

You, thinking based on your values, seem to consider that a good
thing.  You draw the conclusion that it would be unfortunate if Melpa
deleted the Lisp package which refers to that nonfree program.

We, based on the values we have followed since the 1980s say that it
is bad that someone is using a nonfree program, that we wish the
nonfree program did not exist, and that we hope someone will liberate
its users soon by developing a free replacement for it.  We would
be very glad if Melpa deleted that Lisp package.

If user A says, "I really like nonfree program X," my guess is that
you would say, "I am glad you like it."  If A had found out about X
from you, you might feel proud to have "helped".

We would say, "How sad, one step backwards away from freedom."  If A
had found out about X from us, we would say, "Oops!  An own-goal!
What did we do wrong, and how can we make sure we don't do it again?"

We tend to expect that people joining in GNU Project discussion lists
are familiar with the basic ideas and values of the GNU Project.  But
this is not always the case.  It would be good if we could recognize
this sooner and educate people about the basic ideas of the GNU
Project sooner.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le sam. 17 oct. 2020 à 06:21, Richard Stallman <rms@gnu.org> a écrit :
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > We could turn this argument around and ask why the developers who
>   > > maintain MELPA don't remove `2-3' packages that promote non-free
>   > > software.
>
>   > I sincerely hope it doesn't happen. Those packages might rely on
>   > non-free software, but they are still packages that some users find
>   > valuable or even vital, and that they would want to find somewhere
>   > else if not available on Melpa. Removing them from Melpa would only
>   > move the "problem".
>
> I think you have picked up your values (your basis of judging what is
> good or bad) from what most people think, and that you took for granted
> we have the same values.  But we don't.

> The goal of the GNU Project is not simply "to help users."  It is to
> help users _escape from nonfree software_ (des logiciels pas libres), and
> ultimately to build a world where all software is free.

I gave my reasons above. It's not just about "helping users", it's
about helping them move more of their activities to the free world.
Those packages (helm-lastpass, lastpass) are helping users who already
use lastpass at the moment do exactly that.

> Nonfree
> software is an injustice -- nonfree software subjugates users.
> Our goal is to _eradicate it_.

Again, the same question: by arranging for links to such software to
be removed everywhere? Or by offering free alternatives?

Incidentally, I see a lot of effort so far discussing how evil
helm-lastpass and lastpass are, and how to get them moved to obscure
parts of the internet. What I don't see is efforts discussing free
alternatives.

As far as I know the only free competitor for lastpass is bitwarden,
why is nobody talking about developing an emacs interface to bitwarden
and educating lastpass users about it? Isn't _that_ what the GNU
project is about, rather than making it as hard as possible to find
the software we don't like?


> You, thinking based on your values, seem to consider that a good
> thing.  You draw the conclusion that it would be unfortunate if Melpa
> deleted the Lisp package which refers to that nonfree program.
>
> We, based on the values we have followed since the 1980s say that it
> is bad that someone is using a nonfree program, that we wish the
> nonfree program did not exist, and that we hope someone will liberate
> its users soon by developing a free replacement for it.  We would
> be very glad if Melpa deleted that Lisp package.

Me too I guess, once and not before the free replacement exists. As
long as it doesn't, it means that the aforementioned users will be
trapped in the non-free ecosystem because they can't use their
password manager in Emacs (or IceCat). How is that helping the cause
of free software?

> We tend to expect that people joining in GNU Project discussion lists
> are familiar with the basic ideas and values of the GNU Project.  But
> this is not always the case.  It would be good if we could recognize
> this sooner and educate people about the basic ideas of the GNU
> Project sooner.

I am familiar with the ideas and values of the GNU Project, and to a
large extent I agree with them. My points of disagreement lie in
implementation details and priorities.

Framing those disagreements as a lack of education is demeaning.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-17 07:50]:
> I gave my reasons above. It's not just about "helping users", it's
> about helping them move more of their activities to the free world.
> Those packages (helm-lastpass, lastpass) are helping users who already
> use lastpass at the moment do exactly that.
> 
> > Nonfree
> > software is an injustice -- nonfree software subjugates users.
> > Our goal is to _eradicate it_.
> 
> Again, the same question: by arranging for links to such software to
> be removed everywhere? Or by offering free alternatives?
> 
> Incidentally, I see a lot of effort so far discussing how evil
> helm-lastpass and lastpass are, and how to get them moved to obscure
> parts of the internet. What I don't see is efforts discussing free
> alternatives.

There are many password managers in any GNU/Linux system, including, I
am sure, and there are cross platform free software password managers
such as keepass, then there are packages that can manage passwords
with Emacs only, those may not be well integrated, then both KDE/Gnome
have their password managers, each browser has it password managers.

Especially when we are talking about subject of password management,
advising GNU Emacs users to keep their passwords online in a cloud,
managed by proprietary software is very wrong.

Thus there is no alternative to free software.

From Wikipedia:
https://en.wikipedia.org/wiki/LastPass

https://en.wikipedia.org/wiki/LastPass#2011_security_incident
https://en.wikipedia.org/wiki/LastPass#2015_security_breach
https://en.wikipedia.org/wiki/LastPass#2016_security_incidents
https://en.wikipedia.org/wiki/LastPass#2017_security_incidents
https://en.wikipedia.org/wiki/LastPass#2019_security_incidents

Those are only publicly announced security incidents. How many there
are not announced?

In that sense, knowing the background of the insecurities at the
company producing proprietary software, the package lastpass for Emacs
and helm-lastpass is only helping that company subjugates users to
keep their passwords online and sooner or later abuse Emacs users.

My system of keeping passwords is the file .passwords which is stored
on encrypted partition. It is appendable only file by using chattr +a,
and Emacs asks me for host name, username, email, etc. and it
generates password which is appeneded to a file. Other simple function
is grepping and finding list of passwords. It would be disaster to
keep my 4362 passwords online, unsaid of keeping it in some cloud with a
company known for security incidents.

At MELPA bug tracking, or Github issue tracker, the issue is closed,
there was no question if the package "lastpass" is driving users to
insecurities, issue was simply closed, without possibility to publish
this exact information.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le sam. 17 oct. 2020 à 07:44, Jean Louis <bugs@gnu.support> a écrit :
>
> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-17 07:50]:
> > I gave my reasons above. It's not just about "helping users", it's
> > about helping them move more of their activities to the free world.
> > Those packages (helm-lastpass, lastpass) are helping users who already
> > use lastpass at the moment do exactly that.
> >
> > > Nonfree
> > > software is an injustice -- nonfree software subjugates users.
> > > Our goal is to _eradicate it_.
> >
> > Again, the same question: by arranging for links to such software to
> > be removed everywhere? Or by offering free alternatives?
> >
> > Incidentally, I see a lot of effort so far discussing how evil
> > helm-lastpass and lastpass are, and how to get them moved to obscure
> > parts of the internet. What I don't see is efforts discussing free
> > alternatives.
>
> There are many password managers in any GNU/Linux system, including, I
> am sure, and there are cross platform free software password managers
> such as keepass, then there are packages that can manage passwords
> with Emacs only, those may not be well integrated, then both KDE/Gnome
> have their password managers, each browser has it password managers.

Can you use Keepass with Emacs? Can you use Keepass on a phone? Can
you use it on a computer without root access?

> both KDE/Gnome have their password managers

Can you use them with Emacs? Can you use them on a phone?

> each browser has it password managers.

I don't know what Edge does, and Chrome and Chromium use Google
services for their password manager.

Firefox offers Lockwise and Opera also has its in-house method, which
at least work on phones (afaik). But then they require storage in the
cloud in the same way as Lastpass. And can you use them with Emacs?

I mentioned it before, but as far as I know, the only free software
offering for a service similar to Lastpass is Bitwarden: free software
for both the client and the server with the possibility to self-host,
same features as Lastpass (including measuring the overall safety of
your passwords, which I don't think those other password managers do)
and same compatibility list.

Focusing efforts towards evaluating the freedom (freeness?) of
Bitwarden, and if applicable, extending the support for Bitwarden to
the level of that of emacs-lastpass would make it a lot easier to
convince users to abandon that bit of non-free software.

>
> Especially when we are talking about subject of password management,
> advising GNU Emacs users to keep their passwords online in a cloud,
> managed by proprietary software is very wrong.
>
> (...)
>
> From Wikipedia:
> https://en.wikipedia.org/wiki/LastPass
>
> https://en.wikipedia.org/wiki/LastPass#2011_security_incident
> https://en.wikipedia.org/wiki/LastPass#2015_security_breach
> https://en.wikipedia.org/wiki/LastPass#2016_security_incidents
> https://en.wikipedia.org/wiki/LastPass#2017_security_incidents
> https://en.wikipedia.org/wiki/LastPass#2019_security_incidents
>
> Those are only publicly announced security incidents. How many there
> are not announced?
>
> In that sense, knowing the background of the insecurities at the
> company producing proprietary software, the package lastpass for Emacs
> and helm-lastpass is only helping that company subjugates users to
> keep their passwords online and sooner or later abuse Emacs users.
>
> (...)
>
> At MELPA bug tracking, or Github issue tracker, the issue is closed,
> there was no question if the package "lastpass" is driving users to
> insecurities, issue was simply closed, without possibility to publish
> this exact information.

Yes yes, but that's still about the availability of and the problems
behind lastpass and the emacs packages. My question is about
alternatives.

Or, what would you tell users who currently use lastpass and
emacs-lastpass, after you tell them they should stop using lastpass?

Surely you don't want to convince them to use an inferior product just
for purity of software?

I would keep the issue of security incidents separate. Security flaws
are regularly found in both free and non-free software. Lastpass makes
it a policy to announce such breaches.

And 5 incidents in 9 years does not make Lastpass "known for security
incidents", not any more than OpenSSL would be known for security
incidents (even though in the same period, 6 flaws were found and
patched in OpenSSL).

> My system of keeping passwords is the file .passwords which is stored
> on encrypted partition. It is appendable only file by using chattr +a,
> and Emacs asks me for host name, username, email, etc. and it
> generates password which is appeneded to a file. Other simple function
> is grepping and finding list of passwords.

Do you use it across devices? On devices where you don't have root
access? On phones?

> It would be disaster to keep my 4362 passwords online

Assuming that sufficiently strong encryption is used, why exactly?

> Especially when we are talking about subject of password management,
> advising GNU Emacs users to keep their passwords online in a cloud,
> managed by proprietary software is very wrong.
>
> Thus there is no alternative to free software.

I don't see what it has to do with the question, but it is factually
wrong. There are plenty of alternatives to free software.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-17 09:42]:
> Can you use Keepass with Emacs? Can you use Keepass on a phone? Can
> you use it on a computer without root access?
> 
> > both KDE/Gnome have their password managers
> 
> Can you use them with Emacs? Can you use them on a phone?

Password by definition is a secret word or phrase known only to
restricted group or individual, I do not share opinion that passwords
should be placed with unknown group of people like lastpass, neither
at all held in any type of remote cloud.

In case of Keepass as free software, yes, it does work on many
operating systems, probably more than the proprietary lastpass, and it
seems that there is keepass-mode, at least to copy passwords to Emacs,
I did not reseach command line.

Overall it does not matter, those questions are not relevant, as for
GNU project, proprietary software or supporting proprietary software
by making Emacs packages that support such or interact with such, are
simply no go, as such are controlling the user, and do not give
freedom of computing to user.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

1. Looking for free password managers on GNU/Linux makes sense,
and perhaps developing another would be useful.

However, neither of these possibly useful activities
directly affects the ethical issue of leading users to use lastpass.

2. If and when the free password managers are miles above lastpass, it
could happen that no one has a rational use for lastpass except due to
habit.

If and when that happens, perhaps people could modify the Lisp package
for using lastpass so that it clearly informs users, that lastpass is
not worth even trying and they should instead try the free password
managers X, Y, Z.

3. Once that change is, perhaps that Lisp package would be so unlikely
to lead anyone to use lastpass that we would have no reason to worry
about informing users about its existence.  In particular, it would
not be a flaw in MELPA that MELPA contains it.

4. However, each of the other programs in MELPA that depends on a
nonfree program would be a separate issue.

5. If someday MELPA contains no programs that might lead users
to use a nonfree program, another one might be added at any time.
As long as the MELPA maintainers say it is ok to add such packages,
we have to suppose that more such packages may be added.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-18 07:17]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> 1. Looking for free password managers on GNU/Linux makes sense,
> and perhaps developing another would be useful.

There are many password managers as free software including those that
work on various operating system and mobile devices.

There is GNU password manager:
https://www.gnu.org/software/gnu-pw-mgr/

There are GUI password managers in Gnome and KDE, there is
closs-platform keepass and many others:

pass - lightweight directory-based password manager
passwordsafe - Simple & Secure Password Management
qtpass - GUI for password manager pass
gnome-pass-search-provider - GNOME Shell search provider for the pass
password manager
impass - Simple and secure password management and retrieval system
ylva - command line password manager

What the proprietary software lastpass do, they provide online cloud
service for people to store passwords so that they are accessible from
various devices. Maybe 60 million people use that software.

As I am not of opinion that passwords should be stored online, I will
not say that such software is necessary. Other people will argue it is
necessary.

Additionally there are password management packages for Emacs out
there that do not use proprietary software.

There is Syncthing free software package that can synchronize files
across devices, it could also synchronize passwords, it would not be
well integrated.

> 2. If and when the free password managers are miles above lastpass, it
> could happen that no one has a rational use for lastpass except due to
> habit.

There is no rational use to store passwords at centralized server by
using proprietary software, as one cannot know exactly how they
encrypt those passwords, neither is management of their server and
breahes of security transparent.

What can make sense in free software world is cross platform password
manager that users can host on their own server while knowing what
type of encryption is used there.

> If and when that happens, perhaps people could modify the Lisp package
> for using lastpass so that it clearly informs users, that lastpass is
> not worth even trying and they should instead try the free password
> managers X, Y, Z.

That is good idea, MELPA can be replicated and some packages could be
automated to be made unusable with such warning.

> 3. Once that change is, perhaps that Lisp package would be so unlikely
> to lead anyone to use lastpass that we would have no reason to worry
> about informing users about its existence.  In particular, it would
> not be a flaw in MELPA that MELPA contains it.

It would not be flaw if package author would agree with you. The fact
that package author made the package shows that there is opinion
difference, so it cannot work that way. It can only work by third
party that is not MELPA, to duplicate MELPA and offer it in safe way
to users.

> 5. If someday MELPA contains no programs that might lead users
> to use a nonfree program, another one might be added at any time.
> As long as the MELPA maintainers say it is ok to add such packages,
> we have to suppose that more such packages may be added.

That is right, exactly that is the problem, they consider if software
package is GNU GPL, that it is enough regardless if package is
subjugating user.

What I consider larger problem at hand is that MELPA is developed on
Github, Emacs package contributors have to use none free Javascript to
become developers and they all bind themselves to centralized
development system held by Microsoft.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I mentioned it before, but as far as I know, the only free software
  > offering for a service similar to Lastpass is Bitwarden: free software


Are there any that run locally, just a program with no service involved?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-18 07:19]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > I mentioned it before, but as far as I know, the only free software
>   > offering for a service similar to Lastpass is Bitwarden: free software
> 
> 
> Are there any that run locally, just a program with no service
> involved?

There are several on any GNU/Linux distribution.

Majority of them will not place passwords on cloud that data can be
exchanged.

Free software keepass is cross-platform, it already has free software
package for Android/Replicant system and can read data from computer
to other device and vice versa probably, I did not try it.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Surely you don't want to convince them to use an inferior product just
  > for purity of software?

Are you defining "superior" and "inferior" based on practical
considerations only?  Most people think that way; business and the
media inculcate that way of thinking.

However, the GNU Project follows the values of the free software
movement, which hold that most important characteristic of a program
is whether it respects the user's freedom or tramples it.  If you
value your freedom strongly, you will consider any free program better
than any nonfree program.

Based on these values, switching to a free program is always a step
up.

The people who use lastpass are probably not supporters of the free
software movement.  Probably they think that lastpass is better than
no program at all.  They might thing that lastpass is superior to some
free programs.

If we want to persuade them of something, we need to argue based on
premises they agree with.  We may need to make arguments that can
persuade even a person who thinks that a nonfree program is a good
thing.

But we must be careful not to endorse the idea that a nonfree program
is a good thing.
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

>   > Surely you don't want to convince them to use an inferior product just
>   > for purity of software?
>
> Are you defining "superior" and "inferior" based on practical
> considerations only?  Most people think that way; business and the
> media inculcate that way of thinking.
>
> However, the GNU Project follows the values of the free software
> movement, which hold that most important characteristic of a program
> is whether it respects the user's freedom or tramples it.  If you
> value your freedom strongly, you will consider any free program better
> than any nonfree program.
>
> Based on these values, switching to a free program is always a step
> up.
>
> The people who use lastpass are probably not supporters of the free
> software movement.  Probably they think that lastpass is better than
> no program at all.  They might thing that lastpass is superior to some
> free programs.
>
> If we want to persuade them of something, we need to argue based on
> premises they agree with.  We may need to make arguments that can
> persuade even a person who thinks that a nonfree program is a good
> thing.

I agree with all of that. But I'm an idealist: I think if the software
is free and has as many or more functionalities (e.g. MS office vs
Libreoffice, Sublime Text vs Emacs), the argument should convince
everybody, no matter how much they care about freedom.

As an aside, English is obviously not my native language, but in
French, there is a subtle difference being persuade (persuader) and
convince (convaincre): the latter appeals to reason and facts, the
former to emotions. I don't know if the same difference exists in
English, but if it does, I think that what we want is to convince
people to use free software.

> But we must be careful not to endorse the idea that a nonfree program
> is a good thing.

I don't think that I said that.

But a feature available in a non-free program can be a good feature
which we might want to add to free programs (assuming that it is
possible without compromising on freedom, of course).

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But a feature available in a non-free program can be a good feature
  > which we might want to add to free programs (assuming that it is
  > possible without compromising on freedom, of course).

The feature, in the abstract, might be good in the sense of
convenient, and we might well want to implement it.  However,
it is clearer to call it "convenient" rather than "good",
to delineate the contrast with the ethical meaning of "good".

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Yuri Khan]

On Sat, 17 Oct 2020 at 07:51, Thibaut Verron <thibaut.verron@gmail.com> wrote:

> Incidentally, I see a lot of effort so far discussing how evil
> helm-lastpass and lastpass are, and how to get them moved to obscure
> parts of the internet. What I don't see is efforts discussing free
> alternatives.

+1!

> As far as I know the only free competitor for lastpass is bitwarden,
> why is nobody talking about developing an emacs interface to bitwarden
> and educating lastpass users about it?

Have another one: https://www.passwordstore.org/

It is a convention for storing each password in a separate encrypted
text file and a shell script wrapper around GPG and optionally Git
implementing that convention.

The store directory just happens to be easily browsable in Emacs
without any additional interface packages — as you visit an encrypted
file, Emacs asks for the decryption key passphrase and displays a
buffer with decrypted password and any additional metadata associated
with it. (I did not check what happens if you modify and save that
buffer; whether it gets encrypted with all the keys it was originally
encrypted with.)

And it comes with a script that imports passwords from LastPass format.


(I now looked at Bitwarden and it looks like its server component,
while itself being Free, requires Microsoft SQL Server which is
free-to-use but not Free; and without the server component the user is
locked into using someone else’s server instance. So, ideologically,
Bitwarden is not much better than LastPass.)

Re: Proposal for an Emacs User Survey

[Ergus]

On Fri, Oct 16, 2020 at 09:33:12PM +0700, Marcel Ventosa wrote:
>On Fri, 16 Oct 2020 17:04:22 +0300
>Dmitry Gutov <dgutov@yandex.ru> wrote:
>
>
>From what I just read from Thibaut, a free software compatible solution
>to replace MELPA is underway. Refusing to draw attention to something is
>not "shutting our eyes".
>
Melpa will never die. Specially because it is a more open community; has
been around for long time and anyone can contribute almost
anonymously. It is the equivalent to the AUR in arch linux.

>
>I thought your argument was popularity, something that keeps coming up
>in these kinds of discussions. What was your argument?
>
Popularity, usability... there are many. Many packages in melpa are just
unique and has a lot of work invested on them that nobody is planning to
duplicate (ex: magit) 

>
>We could turn this argument around and ask why the developers who
>maintain MELPA don't remove `2-3' packages that promote non-free
>software. What came first, the GNU Emacs or the MELPA?
>
Because this 1) don't improve free software alternatives in any way 2)
will force the user to use free software instead of giving them the
option to choose it 3) Because it will restrict melpa in a way opposed
to its intention... Basically if a user is concerned then

1) will not add melpa to emacs
2) will read every package licenses...

>
>Indeed. As I recall, RMS suggested open questions instead of multiple
>choice questions that "shape their behavior". With open questions, there
>is no need to mention MELPA at all in fact. With open questions, the
>insights that could be derived would be much more interesting.
>
>
>That's very good news if the issue has been settled.
>
>
>Why should Emacs development be guided by (external) survey results? I
>would think it should be guided, for the most part, by what the people
>putting their time into it want to create, within the principles of the
>philosophy of the project and its goals. Also, anyone can suggest
>changes and convince the maintainers that these changes are in the best
>interest of the project (and contribute the actual changes if they are
>accepted).
>
The main problem we have in emacs is the limited number of active
developers. Half of the time the problem is not to convince the
developers to do the work; but having someone to do it... So in that
scenario, all the work invested in melpa packages is something we are
not in conditions to reject; because those developers provide support
and packages that we don't have the man power to do.

>If they are not, Emacs makes it quite simple to implement changes for
>personal "improvements". I have written functions that serve me
>personally and change the behavior of Emacs to suit my needs. There are
>limits to what I can do, which could be pushed if I dedicated a greater
>effort to do so. How is that unfair?
>
>If I thought one of my changes could benefit the community at large, I
>would approach the maintainers and suggest it. If they disagreed with my
>view, I could publish the code and could still share it with anyone
>willing to run it.
>
Exactly... and you can share those changes freely in melpa instead of
keeping them to you if you don't have a copyright, the changes are small
or just someone in your team don't want to register a copyright for any
reason, your package is in an initial state or for very specific uses;
your boss does don't give you a disclaim for your code, or you just want
to share your changes with the hope that someone in the future improve
what you did or find your idea useful in a more informal way.

That's the freedom that many users find in melpa over elpa. (like AUR in
arch linux). Share what they did freely without compromise and probably
get improves and feedback from the community without asking who are you
or if you have a copyright document or feeling the responsibility to
maintain the package in the future.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 17:33, Marcel Ventosa wrote:

>> Shutting our eyes to actual user behavior also precludes
>> understanding.
> 
>  From what I just read from Thibaut, a free software compatible solution
> to replace MELPA is underway. Refusing to draw attention to something is
> not "shutting our eyes".

To make a replacement for something, you need to know what something is, 
and why it exists.

>>>> ignore a project that has done a lot to popularize Emacs over the
>>>> years.
>>>
>>> I fail to understand the narrative that pushes popularity over all
>>> else.
>>
>> Strawman.
> 
> I thought your argument was popularity, something that keeps coming up
> in these kinds of discussions. What was your argument?

You made a strawman claiming that I somehow push popularity "over all else".

My argument is that they have done a lot for the community. Some of 
those effects reach the core too.

>> And picking on 2-3 "ideologically impure" packages (out of several
>> thousands!) that are distributed on MELPA is counter-productive.
> 
> We could turn this argument around and ask why the developers who
> maintain MELPA don't remove `2-3' packages that promote non-free
> software. What came first, the GNU Emacs or the MELPA?

No, we couldn't "turn this argument around". Those statements are not 
even remotely on the same level.

Some Emacs maintainers themselves are known to use proprietary software. 
And use Emacs on proprietary OSes. And yet, somehow, MELPA is the odd 
one out?

>> You can't be effective at affecting change anyway, if you don't know
>> what's going on outside.
> 
> Indeed. As I recall, RMS suggested open questions instead of multiple
> choice questions that "shape their behavior". With open questions, there
> is no need to mention MELPA at all in fact. With open questions, the
> insights that could be derived would be much more interesting.

So we won't suggest ELPA as an option either? What about the users who 
don't know the difference? MELPA is also an ELPA, after all (as in 
"Emacs Lisp Package Archive").

>> Didn't Philip show a prototype that didn't use JavaScript?
> 
> That's very good news if the issue has been settled.

It's solvable, let's put it this way.

>>>> it is unfortunate how Emacs leadership does little to follow the
>>>> external, "unofficial" polls.
>>>
>>> What do you mean by this?
>>
>> I don't recall any single change in Emacs' behavior that resulted
>> from an external poll or survey.
> 
> Why should Emacs development be guided by (external) survey results?

Why should it use the scientific method? Why should it pay attention to 
other people, or the world at large? Is that what you're asking?

> I
> would think it should be guided, for the most part, by what the people
> putting their time into it want to create, within the principles of the
> philosophy of the project and its goals.

It's not a painting or a novel. It's a software project, with certain 
expectations of practicality.

> Also, anyone can suggest
> changes and convince the maintainers that these changes are in the best
> interest of the project (and contribute the actual changes if they are
> accepted).

No, not "anyone can convince".

And like I said: no attention to external polls. Which is what you asked 
me to clarify.

> If they are not, Emacs makes it quite simple to implement changes for
> personal "improvements". I have written functions that serve me
> personally and change the behavior of Emacs to suit my needs. There are
> limits to what I can do, which could be pushed if I dedicated a greater
> effort to do so. How is that unfair?

You're veering the discussion off to the side for some reason.

But if we're talking of "unfair", releasing Emacs under GPL, enabling 
such excellent extensibility that multiple communities spring up over 
years, ones brimming with creativity and people dedicating years of 
their spare time to the extensions, and then badmouthing them from afar 
as though they violated some existing contract (social or legal), *that* 
is unfair.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Dmitry Gutov <dgutov@yandex.ru> [2020-10-16 22:08]:
> My argument is that they have done a lot for the community. Some of
> those effects reach the core too.

That is right, and that still does not mean it is good to endorse
MELPA as free software repository as it is for reasons of those
packages in question, dangerous for users, as proprietary software in
itself is dangerous.

It may sound not logical, but contributions are welcome from anybody,
and denunciation is good tool to point out to users that GNU and also
FSF is not endorsing the non-free software distributions, and also
non-free software repositories.

> Some Emacs maintainers themselves are known to use proprietary
> software. And use Emacs on proprietary OSes. And yet, somehow, MELPA
> is the odd one out?

Everybody is welcome to contribute to GNU.

GNU policies do not mind if developers use proprietary software or
operating systems, that is how free software entered proprietary
operating systems.

It does not matter if Emacs maintainers are themselves using
proprietary software as they do not necessarily push or set the
policies for the overall GNU project.

In any case, everybody is welcome to contribute in creation and
maintenance of free software operating systems and its parts.

Denunciation would not discriminate between Emacs developers or non
Emacs developers, but it would speaking out against publicly
distribution of software that purports to be free, but its sole
purpose was to run proprietary software or to interact with non-free
software or interacting with proprietary Services as Software
Substitutes: http://www.gnu.org/philosophy/who-does-that-server-really-serve.html

Add your useful voice to Github issue on MELPA, to remove such
software packages that wrap proprietary: https://github.com/melpa/melpa/issues/7185

> > Why should Emacs development be guided by (external) survey results?
> 
> Why should it use the scientific method?

I think it is far from being scientific. Those few people proposed it
who do not have any experience in making a survey. If one has
experience, one is not making any questions, one just do it. Finally,
it was not a professional opinion poll agency.

> > Also, anyone can suggest changes and convince the maintainers that
> > these changes are in the best interest of the project (and
> > contribute the actual changes if they are accepted).
> 
> No, not "anyone can convince".

Just {M-x report-emacs-bug} and they will not ask you for name, birth
day, developers will look into the bug report, or suggestion
improvement, it is a group decision.

If you have something to propose, you are free, and anybody is free,
it is so far I know, not even a subscription mailing list, people can
write to mailing list without being subscribed, great thing.

There are other mailing lists like help-gnu-emacs, also very friendly.

I am only sorry for #emacs IRC channel being unfriendly especially
towards free software movement at times, but that is due to few who do
their personal hate promotion, or have nothing better to do in life,
or simply work for proprietary software and hate themselves.

> > If they are not, Emacs makes it quite simple to implement changes for
> > personal "improvements". I have written functions that serve me
> > personally and change the behavior of Emacs to suit my needs. There are
> > limits to what I can do, which could be pushed if I dedicated a greater
> > effort to do so. How is that unfair?
> 
> You're veering the discussion off to the side for some reason.
> 
> But if we're talking of "unfair", releasing Emacs under GPL, enabling such
> excellent extensibility that multiple communities spring up over years, ones
> brimming with creativity and people dedicating years of their spare time to
> the extensions, and then badmouthing them from afar as though they violated
> some existing contract (social or legal), *that* is unfair.

Everybody is free to make proprietary software by using Emacs Lisp and
run it with Emacs Lisp. So people are also free to sell such
proprietary software. GNU is not supporting those communities.

Similarly, packages can be made solely for purpose of running
proprietary software or interacting with such, or interacting with
Service as a Software Substitute, or fetching some proprietary
information. When such software is promoted on supposedly free
software repository, and free software repository has no clear policy
on protecting users' freedom, then it is appropriate to speak out
against it.

There is maybe no written contract, yet there is unspoken
understanding that if person is releasing free software such as Emacs
package that such should not use proprietary software, as that defeats
the purpose.

- Hey brother... I made here one spectacular Emacs package.

- wow, great, I hope you have free software license, that I can use it
  too.

- sure it has free software license, fine GPL terms here inside

- alright, give me that I try

- sure, here it is, to run this software you only need to download
  the proprietary LastPass, then it will work

- oh, is that so? And what does LastPass do to my computer? Do they
  track me? Read my information? Read my passwords?

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 22:52, Jean Louis wrote:
> - sure, here it is, to run this software you only need to download
>    the proprietary LastPass, then it will work

I never said we should applaud individual packages like this.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

When recommending someone to use Emacs, one is recommending free
software, but when one recommends a package repository that can
contain non-free software or software that depends on non-free
software, then one could end up promoting non-free software -- at
which point it is simpler to simply not refer to such repositories.

(standards) References has a good section on it.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Fri, 16 Oct 2020 22:08:01 +0300
Dmitry Gutov <dgutov@yandex.ru> wrote:

> >> You can't be effective at affecting change anyway, if you don't
> >> know what's going on outside.  
> > 
> > Indeed. As I recall, RMS suggested open questions instead of
> > multiple choice questions that "shape their behavior". With open
> > questions, there is no need to mention MELPA at all in fact. With
> > open questions, the insights that could be derived would be much
> > more interesting.  
> 
> So we won't suggest ELPA as an option either? What about the users
> who don't know the difference? MELPA is also an ELPA, after all (as
> in "Emacs Lisp Package Archive").

Is it a survey then or is it an opportunity to "educate/advertise?"

> > I
> > would think it should be guided, for the most part, by what the people
> > putting their time into it want to create, within the principles of the
> > philosophy of the project and its goals.  

> It's not a painting or a novel. It's a software project, with certain 
> expectations of practicality.

Are you claiming Emacs is not practical?

> > If they are not, Emacs makes it quite simple to implement changes for
> > personal "improvements". I have written functions that serve me
> > personally and change the behavior of Emacs to suit my needs. There are
> > limits to what I can do, which could be pushed if I dedicated a greater
> > effort to do so. How is that unfair?  

> You're veering the discussion off to the side for some reason.

I'm explaining how easy it is to modify Emacs to suit particular needs,
and listing the possibilities that already exist for doing so.

> But if we're talking of "unfair", releasing Emacs under GPL, enabling 
> such excellent extensibility that multiple communities spring up over 
> years, ones brimming with creativity and people dedicating years of 
> their spare time to the extensions, and then badmouthing them from afar 
> as though they violated some existing contract (social or legal), *that* 
> is unfair.

It is GNU policy not to promote or encourage proprietary software. To
the extent that any community does so, GNU must not promote or encourage
that community. You mentioned it's a matter of 2 or 3 packages that
recommend proprietary software, so the current impasse should be very
easy to fix. I fail to see what injustice has been perpetrated on the
MELPA maintainers here, or how they have been badmouthed.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 17.10.2020 14:40, Marcel Ventosa wrote:

>> So we won't suggest ELPA as an option either? What about the users
>> who don't know the difference? MELPA is also an ELPA, after all (as
>> in "Emacs Lisp Package Archive").
> 
> Is it a survey then or is it an opportunity to "educate/advertise?"

Somebody else already explained that a poll where only one of the 
options is written and others are free-form will be biased anyway.

>>> I
>>> would think it should be guided, for the most part, by what the people
>>> putting their time into it want to create, within the principles of the
>>> philosophy of the project and its goals.
> 
>> It's not a painting or a novel. It's a software project, with certain
>> expectations of practicality.
> 
> Are you claiming Emacs is not practical?

It's less practical than it could be.

And you seem to be claiming it doesn't need to.

>>> If they are not, Emacs makes it quite simple to implement changes for
>>> personal "improvements". I have written functions that serve me
>>> personally and change the behavior of Emacs to suit my needs. There are
>>> limits to what I can do, which could be pushed if I dedicated a greater
>>> effort to do so. How is that unfair?
> 
>> You're veering the discussion off to the side for some reason.
> 
> I'm explaining how easy it is to modify Emacs to suit particular needs,
> and listing the possibilities that already exist for doing so.

Why are you explaining that to an Emacs developer?

>> But if we're talking of "unfair", releasing Emacs under GPL, enabling
>> such excellent extensibility that multiple communities spring up over
>> years, ones brimming with creativity and people dedicating years of
>> their spare time to the extensions, and then badmouthing them from afar
>> as though they violated some existing contract (social or legal), *that*
>> is unfair.
> 
> It is GNU policy not to promote or encourage proprietary software.

Those are not objective words, but a matter of opinion, to say the least.

> To
> the extent that any community does so, GNU must not promote or encourage
> that community.

You are saying that, by mentioning MELPA is the poll, GNU would promote 
MELPA. Right?

Are you promoting proprietary software by saying the phrase "proprietary 
software", over and over, on this mailing list? And even giving examples 
sometimes.

> I fail to see what injustice has been perpetrated on the
> MELPA maintainers here, or how they have been badmouthed.

Saying "MELPA promotes proprietary software" is like saying that GNU 
project promotes, I don't know... late-stage capitalism. Just by the 
virtue of being mostly developed in a capitalist country, and by being 
useful to many organizations participating in capitalist economies with 
high degree of inequality.

Or that it promotes proprietary software by providing support for some 
of it.

Let's disregard its mission, and all virtues and accomplishments, shall we?

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   > It is GNU policy not to promote or encourage proprietary software.

   Those are not objective words, but a matter of opinion, to say the least.

Most things in life are opinions, but those are the opinions of the
GNU project, Emacs, and this list.

   > To
   > the extent that any community does so, GNU must not promote or encourage
   > that community.

   You are saying that, by mentioning MELPA is the poll, GNU would promote 
   MELPA. Right?

Reading the GNU Coding Standards (see (standards) References) would be
a good start to understanding what the policies are; the short answer
is yes.

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

The original discussion involved the following:

- Richard suggested a survey with open questions (There is no need to
  mention any repository in an open question). 

- MELPA contains packages that promote proprietary software.

- If MELPA *had* to be mentioned in the survey, Richard suggested
  mentioning it with a disclaimer to abide by GNU policy and warn
  unsuspecting survey takers. 

With respect, I don't see the point of discussing off-topic tangents or
responding to apagogical arguments. 

P.S. For the record, I never provided examples of proprietary software.

On Sat, 17 Oct 2020 22:51:54 +0300
Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 17.10.2020 14:40, Marcel Ventosa wrote:
> 
> >> So we won't suggest ELPA as an option either? What about the users
> >> who don't know the difference? MELPA is also an ELPA, after all (as
> >> in "Emacs Lisp Package Archive").  
> > 
> > Is it a survey then or is it an opportunity to "educate/advertise?"
> >  
> 
> Somebody else already explained that a poll where only one of the 
> options is written and others are free-form will be biased anyway.
> 
> >>> I
> >>> would think it should be guided, for the most part, by what the
> >>> people putting their time into it want to create, within the
> >>> principles of the philosophy of the project and its goals.  
> >   
> >> It's not a painting or a novel. It's a software project, with
> >> certain expectations of practicality.  
> > 
> > Are you claiming Emacs is not practical?  
> 
> It's less practical than it could be.
> 
> And you seem to be claiming it doesn't need to.
> 
> >>> If they are not, Emacs makes it quite simple to implement changes
> >>> for personal "improvements". I have written functions that serve
> >>> me personally and change the behavior of Emacs to suit my needs.
> >>> There are limits to what I can do, which could be pushed if I
> >>> dedicated a greater effort to do so. How is that unfair?  
> >   
> >> You're veering the discussion off to the side for some reason.  
> > 
> > I'm explaining how easy it is to modify Emacs to suit particular
> > needs, and listing the possibilities that already exist for doing
> > so.  
> 
> Why are you explaining that to an Emacs developer?
> 
> >> But if we're talking of "unfair", releasing Emacs under GPL,
> >> enabling such excellent extensibility that multiple communities
> >> spring up over years, ones brimming with creativity and people
> >> dedicating years of their spare time to the extensions, and then
> >> badmouthing them from afar as though they violated some existing
> >> contract (social or legal), *that* is unfair.  
> > 
> > It is GNU policy not to promote or encourage proprietary software.  
> 
> Those are not objective words, but a matter of opinion, to say the
> least.
> 
> > To
> > the extent that any community does so, GNU must not promote or
> > encourage that community.  
> 
> You are saying that, by mentioning MELPA is the poll, GNU would
> promote MELPA. Right?
> 
> Are you promoting proprietary software by saying the phrase
> "proprietary software", over and over, on this mailing list? And even
> giving examples sometimes.
> 
> > I fail to see what injustice has been perpetrated on the
> > MELPA maintainers here, or how they have been badmouthed.  
> 
> Saying "MELPA promotes proprietary software" is like saying that GNU 
> project promotes, I don't know... late-stage capitalism. Just by the 
> virtue of being mostly developed in a capitalist country, and by
> being useful to many organizations participating in capitalist
> economies with high degree of inequality.
> 
> Or that it promotes proprietary software by providing support for
> some of it.
> 
> Let's disregard its mission, and all virtues and accomplishments,
> shall we?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-16 17:33]:
> > I don't recall any single change in Emacs' behavior that resulted
> > from an external poll or survey.
> 
> Why should Emacs development be guided by (external) survey results? I
> would think it should be guided, for the most part, by what the people
> putting their time into it want to create, within the principles of the
> philosophy of the project and its goals. Also, anyone can suggest
> changes and convince the maintainers that these changes are in the best
> interest of the project (and contribute the actual changes if they are
> accepted).

There is nothing wrong for Emacs developers to look into external
survey results, to look into comments people are making around various
websites, or any outside data, it is finally their creation, there is
nothing wrong into looking into survey results, especially if
something useful can be created out of it.

There shall be even a checklist to look for bugs in various GNU/Linux
distributions where users report various bugs, as such could be maybe
overseen if not reported upstream, then to look into whatever data is
available externally, that is all fine.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Dmitry Gutov <dgutov@yandex.ru> [2020-10-16 17:04]:
> And picking on 2-3 "ideologically impure" packages (out of several
> thousands!) that are distributed on MELPA is counter-productive.

There are few only until letter C, and I did not finish letter C, so I
guess there are many, number of downloads in thousands, that is pretty
devastating for a repository that is supposed to push free software,
in other words, repository is hypocritical.

Would it be only hypocritical without influencing users, fine, but it
does influence thousands of users and is promoting the download number
of software that guides users to non-free software, so that is how it
becomes not ethical.

You know, in many countries, government is working for people, but
from time to time, they will simply kill someone. You get the idea? Is
it counter productive to complain on few killed?

> > I don't understand how refusing to draw attention to a repository that
> > recommends proprietary software turns anyone into the "thought police".
> 
> It's a *survey*! A survey is supposed to gather insight into what users do,
> and what they need. Not shape their behavior.

Every professional public survey is there to shape their behavior. You
do the survey, you find out what public wants, then you make changes
in such manner to influence public to gather more around you.

All professional surveys are like that. There is nothing wrong with it.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 22:17, Jean Louis wrote:
> * Dmitry Gutov <dgutov@yandex.ru> [2020-10-16 17:04]:
>> And picking on 2-3 "ideologically impure" packages (out of several
>> thousands!) that are distributed on MELPA is counter-productive.
> 
> There are few only until letter C, and I did not finish letter C, so I
> guess there are many, number of downloads in thousands, that is pretty

Not too long ago I went through the first 100 or so most popular 
packages on MELPA (at the request of RMS), and like 2-3 of them 
encouraged or required proprietary software to be installed. Ones near 
the tail of that 100, when sorted by popularity. It's a minor fraction.

> devastating for a repository that is supposed to push free software,
> in other words, repository is hypocritical.

No, it's supposed to _publish_ free software.

> Would it be only hypocritical without influencing users, fine, but it
> does influence thousands of users and is promoting the download number
> of software that guides users to non-free software, so that is how it
> becomes not ethical.

Some time ago I read an essay about infighting in the political left 
community. How a lot of members choose to find and attack progressively 
personal character faults in their fellow activists, instead of working 
together and presenting a united front against the capitalist oppressors.

You can draw an easy parallel to the software community and the struggle 
vs proprietary software.

> You know, in many countries, government is working for people, but
> from time to time, they will simply kill someone. You get the idea? Is
> it counter productive to complain on few killed?

It's counter-productive to quarrel with your neighbor about unmowed 
lawns or whatever when the government is doing the killing.

>>> I don't understand how refusing to draw attention to a repository that
>>> recommends proprietary software turns anyone into the "thought police".
>>
>> It's a *survey*! A survey is supposed to gather insight into what users do,
>> and what they need. Not shape their behavior.
> 
> Every professional public survey is there to shape their behavior. You
> do the survey, you find out what public wants, then you make changes
> in such manner to influence public to gather more around you.

That's what I said.

But it doesn't work as well if the survey is biased from the outset.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

Sorry,

On 16.10.2020 22:36, Dmitry Gutov wrote:
> Not too long ago I went through the first 100 or so most popular 
                                             ^ 1000
> packages on MELPA

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Dmitry Gutov <dgutov@yandex.ru> [2020-10-16 22:37]:
> > devastating for a repository that is supposed to push free software,
> > in other words, repository is hypocritical.
> 
> No, it's supposed to _publish_ free software.

Alright, I just think of policies.

Imagine if free software repository would publish only software that
wraps around proprietary software, would that be free? So it is matter
of policy.

For same reaso Debian GNU/Linux cannot be said to be free, at many
pages they guide users to include non-free software, unspoken from
Archlinux or other distributions.

So it is just matter of endorsement.

> > Would it be only hypocritical without influencing users, fine, but it
> > does influence thousands of users and is promoting the download number
> > of software that guides users to non-free software, so that is how it
> > becomes not ethical.
> 
> Some time ago I read an essay about infighting in the political left
> community. How a lot of members choose to find and attack progressively
> personal character faults in their fellow activists, instead of working
> together and presenting a united front against the capitalist oppressors.
> 
> You can draw an easy parallel to the software community and the struggle vs
> proprietary software.

That question is best answered here:
https://github.com/melpa/melpa/issues/7185

So that is request to remove those packages and to make a policy not
to include such, that is best way to raise the issue and see if we are
truly together in free software promotion.

> But it doesn't work as well if the survey is biased from the outset.

But survey is "biased" (opinionated) from beginning, it was not published on
emacs-devel initially, it was published on Reddit initially and I am
sure there are reasons for initiator for the survey, those reasons may
be that whay you call biased, I call it opinions, but not necessarily
unjust opinions (biased).

It is better to say opinionated. Again I see nothing wrong against
opinionated surveys as what Emacs teaches its users it to become
opinionated, they constantly try to improve their configurations,
themese, etc.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> Alright, I just think of policies.
>
> Imagine if free software repository would publish only software that
> wraps around proprietary software, would that be free? So it is matter
> of policy.

Incidentally, if Melpa agrees to your request to remove all packages
wrapping around proprietary software, it will be a matter of days
before such a repository appears.

And yes, it would be free, in my opinion. Because users are free to
use Emacs in whichever way they want, including in workflows involving
non-free software.

> For same reaso Debian GNU/Linux cannot be said to be free, at many
> pages they guide users to include non-free software, unspoken from
> Archlinux or other distributions.

This is, again, a matter of opinion.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 23:30]:
> > For same reaso Debian GNU/Linux cannot be said to be free, at many
> > pages they guide users to include non-free software, unspoken from
> > Archlinux or other distributions.
> 
> This is, again, a matter of opinion.

It may be "user repository" but it is still advertised through
Archlinux:
https://aur.archlinux.org/packages/?K=proprietary&SB=

It is hypocritical to say that one repository link or directory is
official and other one is not official, of both of them are
distributed from the same official domain "archlinux.org"

See here:
https://bbs.archlinux.org/viewtopic.php?pid=1924794#p1924794

> Arch does believe in free software -- we provide a free software operating system which doesn't depend on proprietary software to run. Any (well, most) Linux operating system could do that.
> We *also* believe people have the right to use proprietary software addons if they want to, and in some cases we make it exceedingly easy for them to opt in.

> All we care about is that we have written permission from the proprietary owners to do so, and we distribute these in /usr/share/licenses/ along with marking the package as possessing a "custom" license:
> https://github.com/archlinux/svntogit-c … ISSION.eml
> https://github.com/archlinux/svntogit-c … ibute.mbox

Now for Debian:

https://duckduckgo.com/?q=site%3Adebian.org+enable+non-free&t=ffab&ia=web

Quote from some of links:

> "Make sure to enable contrib and non-free in the /etc/apt/sources.list line"

> "In some cases the installer detects the need for non-free firmware
> and prompts the user to make the firmware available to the installer
> to complete the installation. This can happen, for example, with
> wireless network cards which often require non-free firmware to
> function"

> "First you have to enable the non-free repository in APT's
> sources.list file: see Section 6.1, "Filling in the sources.list File"
> for details about this file. Many firmware are proprietary and are
> thus located in this repository."

Debian directly instructs users to enable non-free repositories.

So that opinion of mine is quite factual.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Imagine if free software repository would publish only software that
  > > wraps around proprietary software, would that be free? So it is matter
  > > of policy.

  > Incidentally, if Melpa agrees to your request to remove all packages
  > wrapping around proprietary software, it will be a matter of days
  > before such a repository appears.

To move all the recommendations of nonfree software
OUT of Melpa, and into some other obscure new repository,
would be a great step forward!  It would eliminate one
of the reasons why currently we must not inform people about Melpa.

If Melpa then were to eliminates the other reason, reported here
today, that it requires use of nonfree software to contribute
packages, and commits to stay on this path, that could make it
possible for us to recommend Melpa!  That would be a big
change for the better.

  > And yes, it would be free, in my opinion. Because users are free to
  > use Emacs in whichever way they want, including in workflows involving
  > non-free software.

Indeed they are, but you've changed to a different question.


  > > For same reaso Debian GNU/Linux cannot be said to be free, at many
  > > pages they guide users to include non-free software, unspoken from
  > > Archlinux or other distributions.

  > This is, again, a matter of opinion.

You're entitled to your opinion, but this list is for discussing what
to do in the GNU Project.  In the GNU Project, this question is a
matter of stated policy.

In https://gnu.org/distros/free-system-distribution-guidelines.html,
our criteria say this is not acceptable.  That is why we classify
Debian as nonfree in https://gnu.org/distros/common-distros.html.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le sam. 17 oct. 2020 à 06:19, Richard Stallman <rms@gnu.org> a écrit :
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Imagine if free software repository would publish only software that
>   > > wraps around proprietary software, would that be free? So it is matter
>   > > of policy.
>
>   > Incidentally, if Melpa agrees to your request to remove all packages
>   > wrapping around proprietary software, it will be a matter of days
>   > before such a repository appears.
>
> To move all the recommendations of nonfree software
> OUT of Melpa, and into some other obscure new repository,
> would be a great step forward!  It would eliminate one
> of the reasons why currently we must not inform people about Melpa.

It would not be obscure for very long, because most of the Emacs blogs
would not refrain from advertising it.

Moving the free packages to a GNU-managed non-GNU Elpa is imo a better
way to achieve such separation between free and non-free community
packages.

>
>   > And yes, it would be free, in my opinion. Because users are free to
>   > use Emacs in whichever way they want, including in workflows involving
>   > non-free software.
>
> Indeed they are, but you've changed to a different question.
>

I don't think I did. Making it as hard as possible for users to find
packages necessary for their workflow is acting against that freedom,
in my opinion.

>
>   > > For same reaso Debian GNU/Linux cannot be said to be free, at many
>   > > pages they guide users to include non-free software, unspoken from
>   > > Archlinux or other distributions.
>
>   > This is, again, a matter of opinion.
>
> You're entitled to your opinion, but this list is for discussing what
> to do in the GNU Project.  In the GNU Project, this question is a
> matter of stated policy.
>
> In https://gnu.org/distros/free-system-distribution-guidelines.html,
> our criteria say this is not acceptable.  That is why we classify
> Debian as nonfree in https://gnu.org/distros/common-distros.html.

As well as, apparently, all other major GNU/Linux distributions. I
wonder how many users would know about the GNU project if not for
those non-free distributions.

And there seems to have been no problem in publishing that list on
gnu.org, even though it names a number of non-free options. Why can't
the same be done with Melpa ?

Re: Proposal for an Emacs User Survey

[Stefan Monnier]

>> To move all the recommendations of nonfree software
>> OUT of Melpa, and into some other obscure new repository,
>> would be a great step forward!  It would eliminate one
>> of the reasons why currently we must not inform people about Melpa.
> It would not be obscure for very long, because most of the Emacs blogs
> would not refrain from advertising it.

AFAIK they make up a very small fraction of the MELPA archive, both in
terms of number of packages and in terms of popularity of those packages.

Of course, those people who want those packages would still find a way
to get to them, but that's OK, we're not in the business of disallowing
people from shackling themselves to proprietary software.


        Stefan

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-17 07:19]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > Imagine if free software repository would publish only software that
>   > > wraps around proprietary software, would that be free? So it is matter
>   > > of policy.
> 
>   > Incidentally, if Melpa agrees to your request to remove all packages
>   > wrapping around proprietary software, it will be a matter of days
>   > before such a repository appears.
> 
> To move all the recommendations of nonfree software
> OUT of Melpa, and into some other obscure new repository,
> would be a great step forward!  It would eliminate one
> of the reasons why currently we must not inform people about Melpa.

Author said yesterday or was it today, they are not interested to
impose a policy to remove questionable packages, issue is closed,
there is no discussion about that at MELPA.

Issue was:

To remove packages that guide users to non-free software or wrap
around proprietary software #7185

Answer by purcell maintainer was:

The MELPA maintainers already take some care to host only Emacs Lisp
packages with GPL or GPL-compatible licences. We are not aware of any
packages which are in legal violation of this compatibility, including
the packages you mention.

Beyond that, we respectfully decline to institute a policy as
proposed, and wish you all the best.

> If Melpa then were to eliminates the other reason, reported here
> today, that it requires use of nonfree software to contribute
> packages, and commits to stay on this path, that could make it
> possible for us to recommend Melpa!  That would be a big
> change for the better.

I don't think so, the answer is clear, they will allow GPLG or
GPL-compatible licenses without looking into purpose of the package,
such open policy allows inclusion of new packages that are meant only
to use, control proprietary software, or insecure remote websites, or
similar.

Re: Proposal for an Emacs User Survey

[Andy Moreton]

On Sat 17 Oct 2020, Richard Stallman wrote:
> You're entitled to your opinion, but this list is for discussing what
> to do in the GNU Project.  In the GNU Project, this question is a
> matter of stated policy.

No. This list is for discussion of emacs development. GNU policy is
related but largely off topic.

Please use a more appropriate advocacy list for policy discussion.

    AndyM

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   On Sat 17 Oct 2020, Richard Stallman wrote:
   > You're entitled to your opinion, but this list is for discussing what
   > to do in the GNU Project.  In the GNU Project, this question is a
   > matter of stated policy.

   No. This list is for discussion of emacs development. GNU policy is
   related but largely off topic.

When it comes to clarifying misconceptioons, or trying to clearing up
policies that are relevant for the GNU project or a GNU project, then
it is certainly on-topic.  This list is after all part of the GNU
project, and as such it follows the policies of the GNU project.

Re: Proposal for an Emacs User Survey

[Eli Zaretskii]

> From: ams@gnu.org (Alfred M. Szmidt)
> Date: Sat, 17 Oct 2020 11:56:26 -0400
> Cc: emacs-devel@gnu.org
> 
>    On Sat 17 Oct 2020, Richard Stallman wrote:
>    > You're entitled to your opinion, but this list is for discussing what
>    > to do in the GNU Project.  In the GNU Project, this question is a
>    > matter of stated policy.
> 
>    No. This list is for discussion of emacs development. GNU policy is
>    related but largely off topic.
> 
> When it comes to clarifying misconceptioons, or trying to clearing up
> policies that are relevant for the GNU project or a GNU project, then
> it is certainly on-topic.  This list is after all part of the GNU
> project, and as such it follows the policies of the GNU project.

Be it as it may, the signal-to-noise ratio on this list is lately very
low, which is a pity, and IME a clear sign that too many discussions
are only very marginally on-topic, at best.

Since these discussions never lead to any useful or even practical
results, I wish people could stop trying convince one another on such
"on-topic" subjects.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   >    > You're entitled to your opinion, but this list is for discussing what
   >    > to do in the GNU Project.  In the GNU Project, this question is a
   >    > matter of stated policy.
   > 
   >    No. This list is for discussion of emacs development. GNU policy is
   >    related but largely off topic.
   > 
   > When it comes to clarifying misconceptioons, or trying to clearing up
   > policies that are relevant for the GNU project or a GNU project, then
   > it is certainly on-topic.  This list is after all part of the GNU
   > project, and as such it follows the policies of the GNU project.

   Be it as it may, the signal-to-noise ratio on this list is lately very
   low, which is a pity, and IME a clear sign that too many discussions
   are only very marginally on-topic, at best.

That is true, but there are better ways to steer discussions to more
Emacs related topics than to claim that GNU policies are off-topic for
a GNU project.  In this particular case, Richard was fully on-topic
and worth noting..

   Since these discussions never lead to any useful or even practical
   results, I wish people could stop trying convince one another on such
   "on-topic" subjects.

I am not sure what discussion you are refering to, the policies for
GNU are written down and are quite practical.  There is little to
discuss about them as well.

Re: Proposal for an Emacs User Survey

[Eli Zaretskii]

> From: ams@gnu.org (Alfred M. Szmidt)
> CC: andrewjmoreton@gmail.com, emacs-devel@gnu.org
> Date: Sat, 17 Oct 2020 17:38:54 -0400
> 
>    Since these discussions never lead to any useful or even practical
>    results, I wish people could stop trying convince one another on such
>    "on-topic" subjects.
> 
> I am not sure what discussion you are refering to, the policies for
> GNU are written down and are quite practical.  There is little to
> discuss about them as well.

Exactly.  So why waste everyone''s time by discussing them?

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since these discussions never lead to any useful or even practical
  > results, I wish people could stop trying convince one another on such
  > "on-topic" subjects.

People shouldn't bring up abstract questions here which are not
relevant to Emacs development decisions.  Please, everyone, don't
discuss such topics here.

But often nontechnical questions of our values and goals come up
precisely because they are pertinent to the paqctical decision at
hand.  Then these questions are on-topic for the list.  But they are
not questions of personal opinion if the GNU Project has a position on
them.

When people state assertions which conflict with GNU Project policies
or principles, we need to present an authoritative response.  I will
do that eventually, but I am typically more than a day behind.  Alfred
knows our policies and principles, and he often gets there long before
me.  That is useful.

But often his messages don't state that they are authoritative.  So
people may think he is just stating a personal opinion, and argue.

Alfred, when you state GNU Project policies and principles, perhaps
you should say explicitly that "This is the GNU Project's position?"
Also, could you refer to published statements of same?  That could
make your statements more effective.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   Alfred, when you state GNU Project policies and principles, perhaps
   you should say explicitly that "This is the GNU Project's position?"
   Also, could you refer to published statements of same?  That could
   make your statements more effective.

Those are good points, I'll try to keep them in mind -- I'm just
slightly lazy at times.  Thank you.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 23:03, Jean Louis wrote:
> * Dmitry Gutov <dgutov@yandex.ru> [2020-10-16 22:37]:
>>> devastating for a repository that is supposed to push free software,
>>> in other words, repository is hypocritical.
>>
>> No, it's supposed to _publish_ free software.
> 
> Alright, I just think of policies.

As do I.

> Imagine if free software repository would publish only software that
> wraps around proprietary software, would that be free? So it is matter
> of policy.

If it had a policy to only publish wrappers for proprietary software, it 
would be a "repository for wrappers of proprietary software".

But a repository which has a policy to only publish free software, is a 
repository for free software.

> For same reaso Debian GNU/Linux cannot be said to be free, at many
> pages they guide users to include non-free software, unspoken from
> Archlinux or other distributions.

I don't think that's how it works. If the software is free, we call it free.

>>> Would it be only hypocritical without influencing users, fine, but it
>>> does influence thousands of users and is promoting the download number
>>> of software that guides users to non-free software, so that is how it
>>> becomes not ethical.
>>
>> Some time ago I read an essay about infighting in the political left
>> community. How a lot of members choose to find and attack progressively
>> personal character faults in their fellow activists, instead of working
>> together and presenting a united front against the capitalist oppressors.
>>
>> You can draw an easy parallel to the software community and the struggle vs
>> proprietary software.
> 
> That question is best answered here:
> https://github.com/melpa/melpa/issues/7185

Good example of infighting. Not an answer.

It would be a lot more polite to include your actual name in the account 
description, by the way. Calling yourself "GNU Support" looks like an 
overreach.

> So that is request to remove those packages and to make a policy not
> to include such, that is best way to raise the issue and see if we are
> truly together in free software promotion.

"No true Scotsman"

>> But it doesn't work as well if the survey is biased from the outset.
> 
> But survey is "biased" (opinionated) from beginning, it was not published on
> emacs-devel initially, it was published on Reddit initially and I am

That doesn't necessarily make it biased. That argument is weak.

> sure there are reasons for initiator for the survey, those reasons may
> be that whay you call biased, I call it opinions, but not necessarily
> unjust opinions (biased).

Removing a known popular option from the answers makes a survey biased.

> It is better to say opinionated.

Are we reinventing English words now?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Dmitry Gutov <dgutov@yandex.ru> [2020-10-17 00:10]:
> If it had a policy to only publish wrappers for proprietary software, it
> would be a "repository for wrappers of proprietary software".
> 
> But a repository which has a policy to only publish free software, is a
> repository for free software.
> 
> > For same reaso Debian GNU/Linux cannot be said to be free, at many
> > pages they guide users to include non-free software, unspoken from
> > Archlinux or other distributions.
> 
> I don't think that's how it works. If the software is free, we call
> it free.

Maybe I have expressed myself wrongly.

It is not a free software repository if it publishes non-free
software, it is mixture of the two.

So MELPA is also mixture of software that is truly free software and
that other group of software which has the only purpose to interact
with non-free software, more or less promotional tool in free software
community to capture some users for proprietary software, or pathetic
way to promote proprietary software.

> > That question is best answered here:
> > https://github.com/melpa/melpa/issues/7185
> 
> Good example of infighting. Not an answer.

Question about MELPA will be answered there. It is best answered
there, we will see if MELPA authors do like idea or maybe like
Archlinux do not mind if some packages are steering users to access
proprietary software.

> It would be a lot more polite to include your actual name in the account
> description, by the way. Calling yourself "GNU Support" looks like an
> overreach.

I could not register the really intended domain support.gnu, so it is
other way around.

> > sure there are reasons for initiator for the survey, those reasons may
> > be that whay you call biased, I call it opinions, but not necessarily
> > unjust opinions (biased).
> 
> Removing a known popular option from the answers makes a survey biased.
> 
> > It is better to say opinionated.
> 
> Are we reinventing English words now?

Maybe you are native English speaker, I am not. So I am consulting
dictionaries for fine differences, like Wordnut.

biased

* Overview of verb bias

The verb bias has 2 senses (no senses from tagged texts)
1. bias -- (influence in an unfair way; "you are biasing my choice by telling me yours")
2. bias, predetermine -- (cause to be biased)

while opinionated:

* Overview of adj opinionated

The adj opinionated has 1 sense (first 1 from tagged texts)
1. (1) opinionated, opinionative, self-opinionated -- (obstinate in
your opinions)

* Overview of verb obstinate

The verb obstinate has 1 sense (no senses from tagged texts)
1. obstinate -- (persist stubbornly; "he obstinates himself against all rational arguments")


GNU project is obstinated stubbornly not to promote proprietary
software and cannot endorse software repositories that do so.

It is not biased because the stubborn opinion is not equal to being
influenced in unfair way, quite opposite it is being influenced in
just way.

Free software is good because it helps the user to control its own
computing.

That package that is made for Emacs can be issued under GNU GPL, yet
with the only reason to circumvent the purpose of GPL so that
proprietary software makers can control such user, then such software
repository is not ethical and questions about such repository are not
relevant for GNU Emacs development, as it is already known that MELPA
is not endorsed for the discussed reasons.

In fact, how much we have written here, instead why you did not write
to MELPA on that issue and ask them to remove those few packages?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-16 16:45]:
> I've noticed a trend of speaking about Emacs and other free software
> projects as if they were "commodities" and "products," but as I see
> them, it is precisely because they are community driven projects that
> they are not "commodities" or "products".

My opinion is that Emacs is a product, and it is software commodity as
well. Definitions from Wordnet here below like 1. maybe 2, and 5 do
apply, it is product in few definitions. It has been sold for money in
past through GNU project, which is totally alright, and is probably
sold even today, I just don't know where.

If product is community driven, it still remains product. There is
nothing wrong with that word or calling it a product, because it is,
major and significant free software product made by Emacs developers.

* Overview of noun product

The noun product has 6 senses (first 4 from tagged texts)
1. (52) merchandise, ware, product -- (commodities offered for sale; "good business depends on having good merchandise"; "that store offers a variety of products")
2. (25) product, production -- (an artifact that has been created by someone or some process; "they improve their product every year"; "they export most of their agricultural production")
3. (8) product, mathematical product -- (a quantity obtained by multiplication; "the product of 2 and 3 is 6")
4. (2) product -- (a chemical substance formed as a result of a chemical reaction; "a product of lime and nitric acid")
5. product -- (a consequence of someone's efforts or of a particular set of circumstances; "skill is the product of hours of practice"; "his reaction was the product of hunger and fatigue")
6. intersection, product, Cartesian product -- (the set of elements common to two or more sets; "the set of red hats is the intersection of the set of hats and the set of red things")

> > I think it's both insulting to its developers, and stinks of thought 
> > police. Far from the idea of user freedom I hope to expect from
> > GNU and FSF.

It is not important if somebody cals it a product, what is important
is if the distributor or seller provides license with it and gives
same rights to its buyers or users.

> I don't understand how refusing to draw attention to a repository that
> recommends proprietary software turns anyone into the "thought
> police".

It is "proprietary thought police" and there is nothing wrong with
it. Even the thought police will not say there is anything wrong with
thought police. :-p

So when we distribute free software, we tend to speak out against
proprietary software being distributed or promoted together with free
software.

In that sense we are policing various free software repositories and
speaking out publicly against, or denouncing, the inclusion, usage,
and dangers of proprietary software.

So, next time somebody thinks of proprietary software, just remember,
we are watching... slap on fingers.

> In fact, one of the most worrying aspects of this survey idea, as I see
> it, is the suggested use of non-free Javascript to implement it.

Idea about Emacs survey is alright, only that awareness of free
software has yet to arrive to those who initiated the survey, as they
proposed using Google Spreadsheet and similar, which would in fact put
Emacs users at direct risk.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> My opinion is that Emacs is a product,

Certainly.  It is produced, by labor.  It doesn't
fall from the sky.

> and it is software commodity as well. 

No, not really.  A commodity is something that's
produced for sale.

And a common connation of "commodity" is something
that is run-of-the-mill, as opposed to specially
produced (even if for sale).

> It has been sold for money in past through GNU
> project, which is totally alright, and is probably
> sold even today, I just don't know where.

GNU Emacs is not produced for sale, and it's not,
in general, sold.  Generally, it's not a commodity.

If someone sells GNU Emacs then yes, in that case
it could be said to be a commodity.

But as a general characterization that's not true:
Emacs is not produced for sale.  That's not why
it's produced, and selling it is not typical.

In particular, because in its usual form it's also
provided for free (no sale).  If someone sells it,
they likely sell some specialized version of it,
or they sell something else along with it, such as
support services.

Citing synonyms for "product" doesn't suffice, as
a product can be, and most often is (in a market
society), a commodity.

Synonyms describe usage, and they're often given
liberally, intended to include more than draw
distinctions.  Why?  Because people often look to
a thesaurus for a word that means something similar.
Often they're looking for a given word that they
can't quite think of.  Whether the word they're
looking for really means the same thing, or very
close to it, doesn't always matter.  The point is
to help them find that related word.

One shouldn't look to a thesaurus imagining that
a list of synonyms it provides are really exact
synonyms, or even very close.

(And no two words are ever precisely synonymous.
Every word has its own connotations.)

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-16 22:48]:
> > and it is software commodity as well. 
> 
> No, not really.  A commodity is something that's
> produced for sale.

Emacs have been sold and can be sold today, thus it is commodity by
definition.

> And a common connation of "commodity" is something
> that is run-of-the-mill, as opposed to specially
> produced (even if for sale).
> 
> > It has been sold for money in past through GNU
> > project, which is totally alright, and is probably
> > sold even today, I just don't know where.
> 
> GNU Emacs is not produced for sale, and it's not,
> in general, sold.  Generally, it's not a commodity.
> 
> If someone sells GNU Emacs then yes, in that case
> it could be said to be a commodity.
> 
> But as a general characterization that's not true:
> Emacs is not produced for sale.  That's not why
> it's produced, and selling it is not typical.

Read:
https://www.gnu.org/bulletins/bull24.html

If it is typical or not, it was sold and can be sold any time now,
there is no problem selling it, so it is still commodity.

> In particular, because in its usual form it's also provided for free
> (no sale).  If someone sells it, they likely sell some specialized
> version of it, or they sell something else along with it, such as
> support services.

Commodity is if traded, so you say, you cannot possibly know who sold
it and how many times, and how. So it is impossible to say "it is not
commodity" as "it is not sold" as you cannot oversee the planet.

In fact it is sold on many GNU/Linux distributions, it is bundled in
the GNU/Linux operating systems and sold through commercial
distributions, including sold as part of pre-installed systems.

When a customer purchases notebook with Windows, customer is in the
same time indirectly getting the commodity named Windows, even though
did not get any receipt for Windows, but for the notebook (depending
of the shop and country). That does not make Windows less commodity in
its term.

GNU Emacs is pre-installed on many computers, and such installation
require some time, effort, money spent for employees, DVDs, Internet,
very similar to installation of proprietary software. Customer need
not pay directly for the GNU/Linux distribution, but is paying
indirectly.

RE: Proposal for an Emacs User Survey

[Drew Adams]

Maybe reread what I wrote.  I said essentially what
you're saying.  Perhaps we don't disagree at all;
dunno.

But I distinguished the fact that something that is
typically not a commodity (it's offered freely, not
sold) _can_ be sold, and so might in that instance
be said to be a commodity, is nevertheless not
generally a commodity.

GNU Emacs is not generally a commodity.  (It's always
a product.  Not every product is a commodity, even in
what is generally a "market" economy.)

If I paint my car red and green stripes it becomes a
red-and-green car.  But generally cars aren't red and
green.
___

"This division of a product into a useful thing and a value
 becomes practically important, only when exchange has acquired
 such an extension that useful articles are produced for the
 purpose of being exchanged, and their character as values has
 therefore to be taken into account, beforehand, during production."

Commodity production: Products are produced with
an eye to being sold.  Selling and sales value
are taken into account during production - they
are part of the character of the product, as
commodity.  This is _production for exchange_.

GNU Emacs is, in general, not that.

The notes you write to yourself (in Org mode or
whatever) are useful to you, and they weren't
written with an eye to being sold.  If someone
sells those notes, say 100 years from now, they
_could_ be considered a commodity as a result
of that exchange transaction.  But that would
be useless hair-splitting.  The notes weren't
written with an eye to being sold (presumably).
That wasn't the purpose for which they were
produced.

https://web.stanford.edu/~davies/Symbsys100-Spring0708/Marx-Commodity-Fetishism.pdf

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-17 00:02]:
> GNU Emacs is not generally a commodity.  (It's always
> a product.  Not every product is a commodity, even in
> what is generally a "market" economy.)

> Commodity production: Products are produced with an eye to being
> sold.  Selling and sales value are taken into account during
> production - they are part of the character of the product, as
> commodity.  This is _production for exchange_.

Not necessarily, products can be natural, produced by nature,
collected and sold, it can be still a product and commodity. It need
not have a vision to be sold. Example are minerals, example are
natural products collected by people and sold, let us say forest
mushrooms, it is product and commodity.

GNU Emacs was sold by GNU and is (probably) sold today by various
Emacs promoters, I guess with books or DVD or similar. If it is sold
or not currently does not make it less of a commodity, it could be
sold, just as GNU/Linux full OSes can be sold, as nobody forbids it.

It is exchangeable for money.

Software as a community product and community commodity is
exchangeable too, we can see that because it is free software, it
moves people to create more free software, share and help others. That
is the exchange that Emacs is creating itself.

If you consider commodity only that what is exchanged for dollars,
consider that before paper currencies there was maybe gold or shelves,
or something else, like goats maybe, so at that time you could
exchange GNU Emacs for a GNU Goat for example, and it would be just
fine commodity without currency.

> The notes you write to yourself (in Org mode or whatever) are useful
> to you, and they weren't written with an eye to being sold.

In particular on my side, I write instruction manuals that I do sell
to my clients. Yet I get your point, but see above, products need not
have in their creation the purpose to be sold, even though Emacs was
sold by GNU project and FSF in past, maybe not today, but it was sold
on CDs as source software or bundled with GNU software together, and
is sold today by various individuals and companies, sometimes on
CD/DVD with the book or instructions, more often in commercial
GNU/Linux distributions, it need not be in English language, it can be
Japanese language or other countries. GNU/Linux with GNU Emacs is sold
here in Uganda, that is what I can say that I have seen practically,
you come and you can buy it on DVD all together. So it is part of OS
and can be sold with the OS.

You probably want to say that it is not common for you to find GNU
Emacs being sold, you do not consider it commodity.

I am giving you few examples that you see it is indirectly sold,
sometimes directly on DVD with books about Emacs, and as part of
GNU/Linux distributions, probably other distributions as well. It is
probably bundled with many BSD-based OSes as well which are sold as
DVD, like shoplinuxonline.com

It may not be common to you, as you download it, yet there are people
who cannot and who need DVD, and they will buy it.

The last big exchange that Emacs is creating is donations to Free
Software Foundation and supporting GNU infrastructure and its
management.

There is nothing wrong calling it a product or commodity.

Surely I understand your viewpoint, it is not a common for Emacs to be
sold, but it is sold, I gave you few references.

It is special commodity of its own kind.

Note that there are countries without good Internet access, Internet
may be expensive, where DVDs are quite popular in those countries,
where software is sold on DVD and not accessible online. It is cheaper
to purchase DVD with software then download it in those places.

Free software movement have already made OS sellers like Red Hat and
so many others to change into different direction, it also introduced
it to Microsoft and other proprietary companies, but that all does not
make it less of a product, as it is product of human effort, so it is
exchangeable, if not for money then for donations, goodwill,
contributions to the code, it is exchange of effort, so free software
is currency within the community of free software.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is not, strictly speaking, false to think about Emacs in ordinary
economic terms.  But it is not very illuminating to do so, since for the
most part what we do is not driven by economic motives.

That discussion is not pertinent to developing Emacs;
would you please take it off this list?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > No, not really.  A commodity is something that's
  > > produced for sale.

  > Emacs have been sold and can be sold today, thus it is commodity by
  > definition.

Would people please take this discussion off the list emacs-devel?

This list is for discussing practical questions about what to do in
Emacs development.  That discussion of semantics doesn't contribute,
it is a tangent about semantics.

Having disputes like this on the list makes the list unpleasant,
so please move it.

https://gnu.org/philosophy/kind-communication.html urges people to
resist the temptation to discuss tangets.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Ergus]

On Fri, Oct 16, 2020 at 03:17:16PM +0300, Dmitry Gutov wrote:
>On 16.10.2020 11:25, Marcel Ventosa wrote:
>
>>Perhaps. Transitional solutions go both ways though. A cursory glance at
>>Reddit's Emacs group is enough to notice not only ignorance of the
>>philosophy behind GNU, but quite recurrent mockery of what it stands
>>for. Usually in the form of deriding RMS. For the most recent example,
>>one user comments under abrochard's survey post: "So, will you be
>>censoring the survey to maintain ideological purity, like rms
>>insisted?", to which abrochard responds: "I agree with you. The
>>discussion around Melpa is a big factor as to why the survey is
>>happening in parallel to the gnu project."
>
>Do you think the "mockery" is entirely without merit?
>
>It's not even so much real mockery as probably the only way the user 
>could describe the conflict.
>
>When a survey purposefully omits a well-known and popular option, it 
>is going to discount a sizable portion of our community, and ignore a 
>project that has done a lot to popularize Emacs over the years.
>
>I think it's both insulting to its developers, and stinks of thought 
>police. Far from the idea of user freedom I hope to expect from GNU 
>and FSF.
>
>>I'm not saying the obfuscation is purposeful either in the case of
>>`MELPA' imitating the name of the existing `GNU ELPA', or of Adrien
>>calling his survey *The* Emacs User survey", but what I do think is that
>>all non-GNU initiatives that affect perception of GNU, particularly the
>>ones that clearly do not share the GNU philosophy (the survey referred
>>to `GNU/Linux' as `Linux', for example), would seem much more
>>transparent if they were very clearly and visibly labeled as unofficial.
>>My hope, of course, would be that these initiatives could respect the
>>GNU philosophy, even if they did not share it.
>
>Calling it "THE Emacs User survey" is perhaps unfortunate. But 
>likewise it is unfortunate how Emacs leadership does little to follow 
>the external, "unofficial" polls.
>
>A survey doesn't have to be "the official GNU survey" to be useful.
>
I totally agree. I have contributed (and almost maintain) some melpa
packages that have been abandoned by their creators. It is not possible
to add them to elpa because most of the contributors are not active
anymore or are hard to contact, or just don't care at all. This doesn't
mean that they are "bad" or "dangerous" packages just because they can't
follow the paperwork. And that doesn't mean that I should rewrite all
the code just to avoid including the other authors to add them to
elpa. That would be unethical. 

Some melpa packages like LSP, magit, elpy are basically the only
alternative the users have to make emacs useful for their
purposes. While others like helm, use-package, evil are essential for a
huge part of the users.

At the end anyone can use, read the code, distribute and improve that
code... so it is free independently of what a piece of paper says (or
don't say) and closing the code will just kill the packages.

Reject using melpa for ideological reasons means that the users won't
have alternatives for their work and emacs will become like a useless
tool for a big part of our community; whitout melpa many users will go
for more "useful" alternatives that really steal their freedom like
VSCode or Atom... because they need to have the work done any way.

Re: Proposal for an Emacs User Survey

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 15:57:16 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Marcel Ventosa <mve1@runbox.com>,
>  Thibaut Verron <thibaut.verron@gmail.com>,
>  Richard Stallman <rms@gnu.org>, Jean Louis <bugs@gnu.support>,
>  emacs-devel <emacs-devel@gnu.org>
> 
> Reject using melpa for ideological reasons means that the users won't
> have alternatives for their work and emacs will become like a useless
> tool for a big part of our community; whitout melpa many users will go
> for more "useful" alternatives that really steal their freedom like
> VSCode or Atom... because they need to have the work done any way.

I believe the efforts to start nonGNU ELPA are intended to address
that.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Ergus <spacibba@aol.com> [2020-10-16 16:58]:
> Reject using melpa for ideological reasons means that the users won't
> have alternatives for their work and emacs will become like a useless
> tool for a big part of our community; whitout melpa many users will go
> for more "useful" alternatives that really steal their freedom like
> VSCode or Atom... because they need to have the work done any way.

Not true, it is very easy to replicate MELPA, and publish on a
website, while removing any proprietary wrappers or accessors.

In fact MELPA is free software project, so everybody is free to do it,
all what you need is a standard server software, git, Emacs, web
server, it is quite easy.

Just git clone melpa, make, you will see what is happening, then you
can modify it little and have all proprietary stuff removed.

Re: Proposal for an Emacs User Survey

[Alfred M. Szmidt]

   When a survey purposefully omits a well-known and popular option, it is 
   going to discount a sizable portion of our community, and ignore a 
   project that has done a lot to popularize Emacs over the years.

Only that wasn't what was being suggested, rather there where two
solution purposed to solve that issue (either freeform text, or a
disclaimer about why MELPA isn't recommended).

And just because a different project popularizes a GNU project,
doesn't mean that it must automatically be supported -- specially if
there are ethical and moral reasons to not.

   I think it's both insulting to its developers, and stinks of thought 
   police. Far from the idea of user freedom I hope to expect from GNU and FSF.

What users do, and what the GNU or the FSF does are quite different
things -- the GNU project and the FSF try to protect that users have
the freedom, with that comes a responsibility to users to not point
them towards software that is non-free.  What users do, is up to them.

That is neither insulting, or being thought police -- the GNU project
has a specific goal, somethings are simply not related to it, or
against it.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-16 11:25]:
> I understand your point, and transitional solutions may indeed be a good
> thing (though they can lead both ways). However, it's a long-standing
> position of GNU not to be seen to endorse these compromises, whether or
> not their existence is a good thing. For me, the name GNU has always
> signified a free software safe haven. The idea that one might be misled
> into installing proprietary software because of well earned trust in GNU
> should be avoided at all costs.

That is right.

> I would posit the same if I was a member of a vegan organization that
> was relaxing their views on eating animals to ease the transition. While
> there is little doubt transitions can be beneficial, the vegan
> organization should not confuse it's tenets.

That is right, some organization lose their purpose or deviate it.

GNU project never deviates its purpose and is in that sense without compromises.

> Perhaps. Transitional solutions go both ways though. A cursory
> glance at Reddit's Emacs group is enough to notice not only
> ignorance of the philosophy behind GNU, but quite recurrent mockery
> of what it stands for. Usually in the form of deriding RMS. For the
> most recent example, one user comments under abrochard's survey
> post: "So, will you be censoring the survey to maintain ideological
> purity, like rms insisted?", to which abrochard responds: "I agree
> with you. The discussion around Melpa is a big factor as to why the
> survey is happening in parallel to the gnu project."

That is right, there is some mockery of free software philosophy,
especially on IRC #emacs channel, that is very sad state that
intelligent people cannot recognize they are being very rude, it is
type of opression taking place on official GNU communication lines, as
long as #emacs channel on IRC is considered official, that I do not
know. Reddit is not.

> Yes, I see it as a problem when an unofficial offshoot of a project does
> not make it crystal clear that it is so.

It is very easy to clone MELPA, to build packages, to exclude list of
packages and to have your own ELPA locally installed, or to offer such
on Internet for others, in a fixed mode, not continually built
mode. It requires review, and collaboration of many developers to make
it safe and secure for others.

> In fact, and on the same topic, a post was made about the survey
> yesterday on Reddit by Abrochard with the title "The Emacs User
> Survey 2020 will open on Oct 19th," which makes it sound as though
> it were an official survey. Further down the thread, a user
> criticized RMS of trying to censor the survey, mentioning the MELPA
> discussion in particular, to which Abrochard responded: "I agree
> with you. The discussion around Melpa is a big factor as to why the
> survey is happening in parallel to the gnu project."

The Reddit page is hidden propaganda, they said "they contacted
emacs-devel but got no response" which is not true. There were
positive feedbacks overall, and acknowledgment including
acknowledgment by RMS. 

> I'm not saying the obfuscation is purposeful either in the case of
> `MELPA' imitating the name of the existing `GNU ELPA', or of Adrien
> calling his survey *The* Emacs User survey", but what I do think is that
> all non-GNU initiatives that affect perception of GNU, particularly the
> ones that clearly do not share the GNU philosophy (the survey referred
> to `GNU/Linux' as `Linux', for example), would seem much more
> transparent if they were very clearly and visibly labeled as
> unofficial.

If it would be some belief or only philosophy, fine, but it is not. It
is technical term.

Obviously people do not call "Linux" for Android operating system,
neither they call Android GNU because it is not GNU. Yet it runs on
Linux kernel. Also Replicant operating system is not called
GNU/Replicant as it is not GNU, so it is Replicant, operating system
running in Linux kernel.

Linux when published by Linus Torvalds in beginning, there is where he
said that you need GNU for operating system, Linux is only kernel. How
more clearer than that can it be?

Now when somebody says: Archlinux that is derived name, it can be
named of operating system. By the way, Archlinux is promoting
proprietary software without differences, so I cannot recommend it.

Jean

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> The Reddit page is hidden propaganda, they said "they contacted
> emacs-devel but got no response" which is not true.

Where exactly? The exact statement I find is this:

> From there I got a lot of feedback around the set of questions itself but also how to distribute it. You can find the whole thread on the archives under "Proposal for an Emacs User Survey".

That's quite far from "I got no response", so I'm curious as to which
message you refer to.

>
> > I'm not saying the obfuscation is purposeful either in the case of
> > `MELPA' imitating the name of the existing `GNU ELPA', or of Adrien
> > calling his survey *The* Emacs User survey", but what I do think is that
> > all non-GNU initiatives that affect perception of GNU, particularly the
> > ones that clearly do not share the GNU philosophy (the survey referred
> > to `GNU/Linux' as `Linux', for example), would seem much more
> > transparent if they were very clearly and visibly labeled as
> > unofficial.
>
> If it would be some belief or only philosophy, fine, but it is not. It
> is technical term.
>
> Obviously people do not call "Linux" for Android operating system,
> neither they call Android GNU because it is not GNU. Yet it runs on
> Linux kernel. Also Replicant operating system is not called
> GNU/Replicant as it is not GNU, so it is Replicant, operating system
> running in Linux kernel.
>
> Linux when published by Linus Torvalds in beginning, there is where he
> said that you need GNU for operating system, Linux is only kernel. How
> more clearer than that can it be?
>
> Now when somebody says: Archlinux that is derived name, it can be
> named of operating system. By the way, Archlinux is promoting
> proprietary software without differences, so I cannot recommend it.
>
> Jean
>

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 22:12]:
> > The Reddit page is hidden propaganda, they said "they contacted
> > emacs-devel but got no response" which is not true.
> 
> Where exactly? The exact statement I find is this:
> 
> > From there I got a lot of feedback around the set of questions itself but also how to distribute it. You can find the whole thread on the archives under "Proposal for an Emacs User Survey".
> 
> That's quite far from "I got no response", so I'm curious as to which
> message you refer to.

https://www.reddit.com/r/emacs/comments/j6x7ad/proposal_for_an_emacs_user_survey/g820ywv/?utm_source=reddit&utm_medium=web2x&context=3

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 21:54, Jean Louis <bugs@gnu.support> a écrit :
>
> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 22:12]:
> > > The Reddit page is hidden propaganda, they said "they contacted
> > > emacs-devel but got no response" which is not true.
> >
> > Where exactly? The exact statement I find is this:
> >
> > > From there I got a lot of feedback around the set of questions itself but also how to distribute it. You can find the whole thread on the archives under "Proposal for an Emacs User Survey".
> >
> > That's quite far from "I got no response", so I'm curious as to which
> > message you refer to.
>
> https://www.reddit.com/r/emacs/comments/j6x7ad/proposal_for_an_emacs_user_survey/g820ywv/?utm_source=reddit&utm_medium=web2x&context=3

I see, thanks. I was not looking for the correct thread or the correct
user. In any case, I believe that it refers to the suggestion he
posted below the remark, and not about the present survey suggestion:

> :How about creating a new function that generates a report of how a user has configured emacs. Various options, global-minor-modes, faces, etc, similar to what report-emacs-bug generates. Setup a separate email address to receive reports and automatically tabulate what the popular settings are."

I have not seen that thread, so I don't know can't comment about
whether they got responses or not.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Perhaps. Transitional solutions go both ways though. A cursory glance at
  > Reddit's Emacs group is enough to notice not only ignorance of the
  > philosophy behind GNU, but quite recurrent mockery of what it stands
  > for.

Here is an ironic conflict.  It would be good for someone to explain
the GNU pilosophy there so that they understand that we have reasons
for what we do.  But I can't ask you to run nonfree software to
post on Reddit.

Is there any API that someone could use to make a free program
that can post on Reddit?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 730 bytes --]

> 
> Here is an ironic conflict.  It would be good for someone to explain
> the GNU pilosophy there so that they understand that we have reasons
> for what we do.  But I can't ask you to run nonfree software to
> post on Reddit.
> 
> Is there any API that someone could use to make a free program
> that can post on Reddit?
> 
> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

It seems that https://i.reddit.com/ <https://i.reddit.com/> works pretty well
with LibreJS enabled. I assume it’s intended to
be a JavaScript free version. Browsing and posting both works.

[-- Attachment #1.2: Type: text/html, Size: 1369 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Proposal for an Emacs User Survey

[Robert Pluim]

>>>>> On Sat, 17 Oct 2020 04:31:09 +0000, Qiantan Hong <qhong@mit.edu> said:

    Qiantan> It seems that https://i.reddit.com/ <https://i.reddit.com/> works pretty well
    Qiantan> with LibreJS enabled. I assume it’s intended to
    Qiantan> be a JavaScript free version. Browsing and posting both works.

Does that site also work for registering an account on reddit with
LibreJS enabled?

Robert
-- 

Re: Proposal for an Emacs User Survey

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 736 bytes --]

The funny icon on the top right doesn’t work,
But try
https://www.reddit.com/register.compact <https://www.reddit.com/register.compact>
and
https://www.reddit.com/login.compact <https://www.reddit.com/login.compact>

> On Oct 19, 2020, at 6:12 AM, Robert Pluim <rpluim@gmail.com> wrote:
> 
>>>>>> On Sat, 17 Oct 2020 04:31:09 +0000, Qiantan Hong <qhong@mit.edu> said:
> 
>    Qiantan> It seems that https://i.reddit.com/ <https://i.reddit.com/> works pretty well
>    Qiantan> with LibreJS enabled. I assume it’s intended to
>    Qiantan> be a JavaScript free version. Browsing and posting both works.
> 
> Does that site also work for registering an account on reddit with
> LibreJS enabled?
> 
> Robert
> -- 


[-- Attachment #1.2: Type: text/html, Size: 1820 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 19.10.2020 19:15, Qiantan Hong wrote:
> The funny icon on the top right doesn’t work,
> But try
> https://www.reddit.com/register.compact
> and
> https://www.reddit.com/login.compact

Have you managed to register with it?

It seems to want me to use a captcha anyway (looking at the network 
console), but fails to show said captcha.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 10:54]:
> I personally don't think many users install non-free software because
> they saw it wrapped in a Melpa package.

- helm-lastpass was downloaded 777 times
- lastpass was downloaded 987 times

1 user guided to use non-free software is already many.

- chatwork package was downloaded 1093 times

> Taking the example of emacs-lastpass given above, I don't see how
> anyone would even find this package without searching for it with the
> keyword "lastpass".

They can find it in the list, for majority of packages I did not
search by keyword, I was just downloading, inspecting code for short
time, and trying to use it.

> The audience, rather, is users who are currently using Lastpass in
> their browsers but are interested in bringing some of their online
> activities to Emacs, but rely on their password manager to do so.

That is not based on data, unless you have made opinion poll for that
specific package.

I am really thinking that some of users will download lastpass when
they see there is Emacs package for lastpass

I have been downloading like espeak or festival speech packages, which
are free software, when I have seen there are Emacs packages for
speech, in the same way users will be guided to proprietary software.

MELPA is to ELPA what Archlinux and Debian is to Guix and other free
software distributions. They do not explicitly warn users about
proprietary software, even though I do not think MELPA is letting
non-free software being distributed.

> I absolutely support the fact that Melpa is not activated by default,
> and that there should be a warning about the existence of those
> packages everywhere possible. But I still consider that the value of
> those packages outweigh their dangers, just like the win32 build of
> Emacs.

Proprietary software wrapped is security issue, and largest danger is
lack of security as MELPA is not reviewing software updates, any time
malicious code can be inserted which could affect thousands of users.

Here is example when Gentoo Linux was cracked on Github:
https://nakedsecurity.sophos.com/2018/06/29/linux-distro-hacked-on-github-all-code-considered-compromised/

Here is example when Linux Mint distribution was cracked:
https://www.techrepublic.com/article/why-the-linux-mint-hack-is-an-indicator-of-a-larger-problem/

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-16 10:24]:
> I understand, thanks for the explanation. In that case, I think I'm
> well informed enough to have avoived the dangers. I wonder how many
> people are not. AFAIK, both Trisquel and Parabola keep out packages
> that recommend or encourage proprietary software, which seems essential
> to protect users' freedom.

Which packages?

Please report the issues in their issue trackers. Both of them are
removing anything like that, that is what I know.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 09:53]:
> Le ven. 16 oct. 2020 à 08:03, Marcel Ventosa <mve1@runbox.com> a écrit :
> >
> > On Thu, 15 Oct 2020 23:59:07 -0400
> > Richard Stallman <rms@gnu.org> wrote:
> >
> > > I hope that only a minority of Emacs users know about MELPA, and I'd
> > > rather not inform the rest about it.  But if something is going to
> > > inform them anyway, it is better to do it with a denunciation.
> >
> >
> > I've been using Emacs (and MELPA) for the best part of a decade and
> > knew nothing about this! I'm concerned to use only free software and
> > actively avoid proprietary software, so this is a bit of a shock.
> 
> As I understand it, Melpa packages cannot *be* or *install* non-free
> software. But some will not work without such software, which can in
> theory encourage users to install it.

MELPA as such is definitely free software project with few freedom
issues with some pakages and lax attitude on usage of proprietary
information through Emacs. For example, I like that when I find
definition in a dictionary, that I can freely include it in the
instruction book, and not that I am chased with licenses not allowing
me to include such information.

MELPA does have a checklist for packages:

Checklist

Please confirm with x:

    The package is released under a GPL-Compatible Free Software License.
    [x ] I've read CONTRIBUTING.org
    [ x] I've used the latest version of package-lint to check for packaging issues, and addressed its feedback
    [ x] My elisp byte-compiles cleanly
    [ x] M-x checkdoc is happy with my docstrings
    [ x] I've built and installed the package using the instructions in CONTRIBUTING.org
    I have confirmed some of these without doing them

Example:
https://github.com/melpa/melpa/pull/6387

People and MELPA maintainer are verifying packages, but they do not
possibly verify it each time. So it is prone to security issues at any
time. Once package is accepted, they are not automatically verifying
the package, so far I understand, packages are built in real time and
offered to users in real time.

Any account can be cracked and malicious code introduced at any
time.

Github is in general unsafe place for development as it is held by
major company providing proprietary software, one never knows what are
they up to.

Re: Proposal for an Emacs User Survey

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

> Any account can be cracked and malicious code introduced at any
> time.
> 
> Github is in general unsafe place for development as it is held by
> major company providing proprietary software, one never knows what are
> they up to.

And how is that different for, say, Savannah?  In fact, I'd expect
Github to have a dedicated security team, bug bounties to entice
security researchers into discovering and reporting security issues and
overall better security features for locking down your account, such as
2FA.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As I understand it, Melpa packages cannot *be* or *install* non-free
  > software. But some will not work without such software, which can in
  > theory encourage users to install it.

Yes, that's exactly the issue.  Listing a program for possible use can
lead people to use it.  If using that program requires running some
nonfree program, the listing can lead people to run that nonfree
program.

  > ELPA means Emacs Lisp Package Archive, so both Melpa and GNU Elpa are ELPA's.
  > I think that commonly referring to GNU Elpa as simply Elpa (which I am
  > also guilty of) is a bigger source of confusion than Melpa and GNU
  > Elpa sharing the same suffix.

I agree.  I sent another message about that issue.

I think that the basic mistake was implementing support in GNU Emacs
for other package archives.  It may have seemed like the general thing
to do, at the time; but it led to a bad situation.  If I had thought
more about this, maybe I would have realized it beforehand.

  > GNU Elpa and the future Non-GNU Elpa are (will be) activated by
  > default as package archives, Melpa is not.
  > The hope is that once 99% of the packages by the community are
  > available in an archive activated by default, users will not rush to
  > install Melpa in the same proportions as today.

Yes, that's the aim.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

MELPA issues - Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-16 09:03]:
> On Thu, 15 Oct 2020 23:59:07 -0400
> Richard Stallman <rms@gnu.org> wrote:
> 
> > I hope that only a minority of Emacs users know about MELPA, and I'd
> > rather not inform the rest about it.  But if something is going to
> > inform them anyway, it is better to do it with a denunciation.
> 
> 
> I've been using Emacs (and MELPA) for the best part of a decade and
> knew nothing about this! I'm concerned to use only free software and
> actively avoid proprietary software, so this is a bit of a shock.
> 
> Is there anywhere I can read more about this issue?

I have not checked all the software on MELPA, but due to Github
policies that free (of charge)repositories should have only free (as
in liberty)software licenses, I am assuming that probably none of
those software is non-free. But there can be MELPA software that is
vague because maybe maintainers have not put the proper license, which
is often the case.

The software provided by MELPA may lead users to non-free software or
may control non-free software or be made exclusively for usage of free
software.

Example that I have found is ChatWork package, it works with ChatWork
chat software, for which I only assume it is proprietary, I have not
checked it very good, it seemed to be so from verification of their
website.

Corporations can very easily sponsor somebody to provide software for
Emacs to provide features that control or interact with their
proprietary software.

It is also method of advertising.

Then there is software to access various websites, let us say software
that provides quotes from specific website, it could be funny quote or
smart one, but maybe the purpose is simply advertising. Finally,
fetching something from other website I consider dangerous, package
itself need not be, but other packages following, could be easily
dangerous.

More danger from MELPA comes from the fact that MELPA is not verifying
the packages, not that I know, I have read they said they are not
doing it.

There is plethora of insecurities on MELPA. It is far from harmless.

So far I understood, the packages arriving to GNU ELPA are assigned
with copyright to FSF, I am also assuming as user that such packages
are somehow reviewed by developers, not just one developer, and that
they are placed into ELPA as duplicate or copy from the upstream. I
may be wrong in all that assumption, but I think that GNU ELPA
packages are verified for freedom and mostly for security and safety
of users. We are speaking of loading true programming language code
and executing such on users' computers.

It is not equivalent to Javascript, it is far more dangerous than
Javascript which tend to execute in safe environment, which tends to
execute in such way as not to abuse users' computers and data, yet
people have found ways to crack browsers and to crack and enter into
users' file systems, there are many ways how Javascript can be
malicious.

The more packages there are that are not verified, but simple offered
for download through MELPA, the more and more insecurities are coming
in future.

MELPA is allowing Google to track users by using Google Analytics on
their website, that speaks already about the webmaster's lack of
skills in managing the website. There are so many free software
programs for web statistics, and there is no need for third party
tracking.

Now, the real insecurity comes from program that are sourced from
Github. If there are 4000+ packages, there can be 1000+ authors, maybe
even 2000+ authors.

Each of those authors represent insecurity to computing, as their
packages are not verified each time they are pulled, they are blindly
trusted.

The blind trust to MELPA packages is what is making it highly insecure
for computer users.

It requires just 1 author for their accounts to be cracked and for
malicious code to be inserted, thousands of computer users can be
affected that way.

Finally, author can go nut himself, and can become psychotic, there
are programmers who became so, they can introduce malicious code
themselves, or can do it by claiming it was somebody else.

Packages that I think do not belong in free software repository for
reason they are using proprietary information or wrapping proprietary
software, or use known spying networks:

babel - that uses non-free Babelfish translations (if I am mistaken
tell me)

chatwork - that uses non-free ChatWork proprietary chat software

bing-dict - that uses Microsoft Bing proprietary dictionary

calfw-gcal - to edit Google calendar

Obviously I came to letter C, I could browse more and find more
troublesome packages.

Yet major insecurity is number of packages where they are not verified
by human to be safe and blind offering and blind acceptance by users
thinking they are safe.

Jean

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 16.10.2020 19:33, Jean Louis wrote:
> But there can be MELPA software that is
> vague because maybe maintainers have not put the proper license, which
> is often the case.

"GPL compatible license" is among the requirements for inclusion.

It's in their CONTRIBUTING.org, take a look: 
https://raw.githubusercontent.com/melpa/melpa/master/CONTRIBUTING.org

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 208 bytes --]

As a package writer, my main concern is that
it seems that the package submission process
require proprietary software (particularly non-free
JavaScript by m$ Github).

Do Melpa people provide any workaround?

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 17.10.2020 01:00, Qiantan Hong wrote:
> As a package writer, my main concern is that
> it seems that the package submission process
> require proprietary software (particularly non-free
> JavaScript by m$ Github).
> 
> Do Melpa people provide any workaround?

You can ask someone else to do that for you. Email one of the MELPA 
maintainers, for example.

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > As a package writer, my main concern is that
  > it seems that the package submission process
  > require proprietary software (particularly non-free
  > JavaScript by m$ Github).

What I would do, in that situation, is not submit my package to Melpa.

Meanwhile, this aspect of Melpa is an additional reason not to do
anything to lead people to use Melpa.

I hope the developers of Melpa will fix this.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[chad]

[-- Attachment #1: Type: text/plain, Size: 1193 bytes --]

On Fri, Oct 16, 2020 at 9:20 PM Richard Stallman <rms@gnu.org> wrote:

>   > As a package writer, my main concern is that
>   > it seems that the package submission process
>   > require proprietary software (particularly non-free
>   > JavaScript by m$ Github).
>
> What I would do, in that situation, is not submit my package to Melpa.
>
> Meanwhile, this aspect of Melpa is an additional reason not to do
> anything to lead people to use Melpa.
>

This seems to be based on the incorrect assumption that package authors
must use GitHub to use MELPA. According to the MELPA docs:

 MELPA supports git <http://git-scm.com/>, github <https://github.com/>,
> gitlab <https://gitlab.com/>, and hg <https://www.mercurial-scm.org/> (Mercurial)
> [...]
> :url specifies the URL of the version control repository.
> *required for the git, and hg fetchers. [...]*:repo specifies the github
> or gitlab repository and is of the form user/repo-name. *required for
> the github and gitlab fetchers*.


Digging in a bit further, MELPA used to also support EmacsWiki, which it
deprecated for security concerns, and Bazaar, CVS, Darcs, Fossil and
Subversion, which it deprecated due to lack of use.

~Chad

[-- Attachment #2: Type: text/html, Size: 8159 bytes --]

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 1510 bytes --]

However, I do need to use GitHub to open a Pull Request to add
the receipt. 

As someone pointed out, one workaround might be just email the MELPA
maintainers. I’ve never tried that myself.

> On Oct 17, 2020, at 12:59 AM, chad <yandros@gmail.com> wrote:
> 
> On Fri, Oct 16, 2020 at 9:20 PM Richard Stallman <rms@gnu.org <mailto:rms@gnu.org>> wrote:
>   > As a package writer, my main concern is that
>   > it seems that the package submission process
>   > require proprietary software (particularly non-free
>   > JavaScript by m$ Github).
> 
> What I would do, in that situation, is not submit my package to Melpa.
> 
> Meanwhile, this aspect of Melpa is an additional reason not to do
> anything to lead people to use Melpa.
> 
> This seems to be based on the incorrect assumption that package authors must use GitHub to use MELPA. According to the MELPA docs:
> 
>  MELPA supports git <http://git-scm.com/>, github <https://github.com/>, gitlab <https://gitlab.com/>, and hg <https://www.mercurial-scm.org/> (Mercurial) [...]
> :url specifies the URL of the version control repository. required for the git, and hg fetchers. [...]
> :repo specifies the github or gitlab repository and is of the form user/repo-name. required for the github and gitlab fetchers.
>  
> Digging in a bit further, MELPA used to also support EmacsWiki, which it deprecated for security concerns, and Bazaar, CVS, Darcs, Fossil and Subversion, which it deprecated due to lack of use.
> 
> ~Chad


[-- Attachment #1.2: Type: text/html, Size: 9616 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le sam. 17 oct. 2020 à 06:59, chad <yandros@gmail.com> a écrit :
>
> On Fri, Oct 16, 2020 at 9:20 PM Richard Stallman <rms@gnu.org> wrote:
>>
>>   > As a package writer, my main concern is that
>>   > it seems that the package submission process
>>   > require proprietary software (particularly non-free
>>   > JavaScript by m$ Github).
>>
>> What I would do, in that situation, is not submit my package to Melpa.
>>
>> Meanwhile, this aspect of Melpa is an additional reason not to do
>> anything to lead people to use Melpa.
>
>
> This seems to be based on the incorrect assumption that package authors must use GitHub to use MELPA. According to the MELPA docs:

But can a user submit a new package without filing a pull request
against Melpa's github repository?

This requires using github to create an account, create the fork and
file the pull request.

There is a github cli program which might allow do the latter two, but
creating the account will still require github's javascript.

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

Thank you for your explanation Jean Louis.

On Fri, 16 Oct 2020 19:33:45 +0300
Jean Louis <bugs@gnu.support> wrote:

> * Marcel Ventosa <mve1@runbox.com> [2020-10-16 09:03]:
> > On Thu, 15 Oct 2020 23:59:07 -0400
> > Richard Stallman <rms@gnu.org> wrote:
> >   
> > > I hope that only a minority of Emacs users know about MELPA, and
> > > I'd rather not inform the rest about it.  But if something is
> > > going to inform them anyway, it is better to do it with a
> > > denunciation.  
> > 
> > 
> > I've been using Emacs (and MELPA) for the best part of a decade and
> > knew nothing about this! I'm concerned to use only free software and
> > actively avoid proprietary software, so this is a bit of a shock.
> > 
> > Is there anywhere I can read more about this issue?  
> 
> I have not checked all the software on MELPA, but due to Github
> policies that free (of charge)repositories should have only free (as
> in liberty)software licenses, I am assuming that probably none of
> those software is non-free. But there can be MELPA software that is
> vague because maybe maintainers have not put the proper license, which
> is often the case.
> 
> The software provided by MELPA may lead users to non-free software or
> may control non-free software or be made exclusively for usage of free
> software.
> 
> Example that I have found is ChatWork package, it works with ChatWork
> chat software, for which I only assume it is proprietary, I have not
> checked it very good, it seemed to be so from verification of their
> website.
> 
> Corporations can very easily sponsor somebody to provide software for
> Emacs to provide features that control or interact with their
> proprietary software.
> 
> It is also method of advertising.
> 
> Then there is software to access various websites, let us say software
> that provides quotes from specific website, it could be funny quote or
> smart one, but maybe the purpose is simply advertising. Finally,
> fetching something from other website I consider dangerous, package
> itself need not be, but other packages following, could be easily
> dangerous.
> 
> More danger from MELPA comes from the fact that MELPA is not verifying
> the packages, not that I know, I have read they said they are not
> doing it.
> 
> There is plethora of insecurities on MELPA. It is far from harmless.
> 
> So far I understood, the packages arriving to GNU ELPA are assigned
> with copyright to FSF, I am also assuming as user that such packages
> are somehow reviewed by developers, not just one developer, and that
> they are placed into ELPA as duplicate or copy from the upstream. I
> may be wrong in all that assumption, but I think that GNU ELPA
> packages are verified for freedom and mostly for security and safety
> of users. We are speaking of loading true programming language code
> and executing such on users' computers.
> 
> It is not equivalent to Javascript, it is far more dangerous than
> Javascript which tend to execute in safe environment, which tends to
> execute in such way as not to abuse users' computers and data, yet
> people have found ways to crack browsers and to crack and enter into
> users' file systems, there are many ways how Javascript can be
> malicious.
> 
> The more packages there are that are not verified, but simple offered
> for download through MELPA, the more and more insecurities are coming
> in future.
> 
> MELPA is allowing Google to track users by using Google Analytics on
> their website, that speaks already about the webmaster's lack of
> skills in managing the website. There are so many free software
> programs for web statistics, and there is no need for third party
> tracking.
> 
> Now, the real insecurity comes from program that are sourced from
> Github. If there are 4000+ packages, there can be 1000+ authors, maybe
> even 2000+ authors.
> 
> Each of those authors represent insecurity to computing, as their
> packages are not verified each time they are pulled, they are blindly
> trusted.
> 
> The blind trust to MELPA packages is what is making it highly insecure
> for computer users.
> 
> It requires just 1 author for their accounts to be cracked and for
> malicious code to be inserted, thousands of computer users can be
> affected that way.
> 
> Finally, author can go nut himself, and can become psychotic, there
> are programmers who became so, they can introduce malicious code
> themselves, or can do it by claiming it was somebody else.
> 
> Packages that I think do not belong in free software repository for
> reason they are using proprietary information or wrapping proprietary
> software, or use known spying networks:
> 
> babel - that uses non-free Babelfish translations (if I am mistaken
> tell me)
> 
> chatwork - that uses non-free ChatWork proprietary chat software
> 
> bing-dict - that uses Microsoft Bing proprietary dictionary
> 
> calfw-gcal - to edit Google calendar
> 
> Obviously I came to letter C, I could browse more and find more
> troublesome packages.
> 
> Yet major insecurity is number of packages where they are not verified
> by human to be safe and blind offering and blind acceptance by users
> thinking they are safe.
> 
> Jean
> 
> 
> 
> 

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > babel - that uses non-free Babelfish translations (if I am mistaken
  > > tell me)
  > > 
  > > calfw-gcal - to edit Google calendar

When you say that those things are non-free, what precisely does that
mean?  It might be a misunderstanding, because a service is neither
free nor proprietary.

Do these packages run any nonfree software or use some nonfree
collection of data?  Or are you referring to the fact that they send
commands to a service?

See https://www.gnu.org/philosophy/network-services-arent-free-or-nonfree.html.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: MELPA issues - Re: Proposal for an Emacs User Survey

[Jean Louis]

* Richard Stallman <rms@gnu.org> [2020-10-18 07:12]:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > > babel - that uses non-free Babelfish translations (if I am mistaken
>   > > tell me)
>   > > 
>   > > calfw-gcal - to edit Google calendar
> 
> When you say that those things are non-free, what precisely does that
> mean?  It might be a misunderstanding, because a service is neither
> free nor proprietary.

Service is neither free nor proprietary. That is right.

It is my opinion that it does not give freedom to user:

- the package "babel" is accessing non-free dictionary, their terms
  are that one shall not replicate, sell or reporduce content in any
  way. This warning may not be given to user by the package, and
  practically, I am using free dictionaries, which information I can
  embed, including sell as index or glossary. When using babel
  package, I would as user assume that information is free and would
  maybe incorporate it in other works, finally I am using Emacs which
  is meant for text editing.

  Using Wiktionary on the other hand gives me those freedoms.

- for package calfw-gcal, it may allow me to access Google Calendar
  services. It also implies that me as user must register Google
  account, meaning I must submit to being tracked by Google, I have
  access non-free Javascript. Those actions that are necessary before
  I can use the package calfw-gcal. 

> Do these packages run any nonfree software or use some nonfree
> collection of data?

For Babelfish, I doubt that package maintainer complained legally to
their terms, for Google calendar package is clear that one has to
register Google account by using non-free Javascript in order to use
the package.

> Or are you referring to the fact that they send commands to a
> service?

No.

I refer to practical facts above.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In fact, I would go the extra mile and say Emacs should expressly warn
  > users over the dangers of installing proprietary software from
  > unofficial repositories

That could be a good idea.  What would be good occasions on which to warn?

Perhaps in list-packages when it sees a non-GNU repo, or when it sees MELPA?

Perhaps in describe-package and packageinstall, when the package comes
from a non-GNU repo, or specifically from MELPA?

Any other ideas?

                            (by the way, I always just assumed MELPA was
  > somehow official and related to ELPA, because its name is so similar to
  > ELPA).

Yes, this is a source of confusion.

Perhaps we should renamme GNU ELPA to a name that will avoid this confusion.
Maybe GNU EP (GNU Emacs Packages)?

EP is not meaningful to those who don't know what it means.  But
neither is ELPA.  People understand it only if they have been told.
So EP is no worse than ELPA.

WDYT?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Marcel Ventosa]

On Sun, 18 Oct 2020 00:10:02 -0400
Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > In fact, I would go the extra mile and say Emacs should expressly
>   > warn users over the dangers of installing proprietary software
>   > from unofficial repositories  
> 
> That could be a good idea.  What would be good occasions on which to
> warn?
> 
> Perhaps in list-packages when it sees a non-GNU repo, or when it sees
> MELPA?

Any non-GNU repo that has not made an express commitment to uphold free
sofware values?

> Perhaps in describe-package and packageinstall, when the package comes
> from a non-GNU repo, or specifically from MELPA?
> 
> Any other ideas?

How about something similar to Parabola's `your-freedom' package
approach? It doesn't necessarily have to prevent installation of nonfree
programs like `your-freedom' does, but could present a warning where the
user must expressly agree to run the nonfree package the first time it's
loaded (something similar to the `load-theme' warning).

I'm not sure what the technical difficulties implementing this would be.
It would also require an interested party and an ongoing discussion to
keep an up to date list of such packages. If it amounts to 2-3 packages
as I was told, it might be a very simple task.

This would have the added advantage that individual packages that are
not distributed through repositories could be added as they are
discovered, and that the responsibility for warning users about the
dangers of nonfree programs rests with GNU itself.

Who would have the final say for individual candidate packages? In the
case of Parabola, I know there have been long discussions about whether
certain components of certain programs (such as web browser engines) are
unfree. I find these discussions to be a feature rather than an
inconvenience though.

>   > (by the way, I always just assumed MELPA was
>   > somehow official and related to ELPA, because its name is so
>   > similar to ELPA).  
> 
> Yes, this is a source of confusion.
> 
> Perhaps we should renamme GNU ELPA to a name that will avoid this
> confusion. Maybe GNU EP (GNU Emacs Packages)?
> 
> EP is not meaningful to those who don't know what it means.  But
> neither is ELPA.  People understand it only if they have been told.
> So EP is no worse than ELPA.
> 
> WDYT?

This solution would have saved me from the name related confusion.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 18.10.2020 10:51, Marcel Ventosa wrote:
> How about something similar to Parabola's `your-freedom' package
> approach? It doesn't necessarily have to prevent installation of nonfree
> programs like `your-freedom' does, but could present a warning where the
> user must expressly agree to run the nonfree package the first time it's
> loaded (something similar to the `load-theme' warning).

You could propose something like that to MELPA guys, if that would lead 
to MELPA being endorsed.

Could be an acceptable middle ground.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Marcel Ventosa <mve1@runbox.com> [2020-10-18 10:51]:
> How about something similar to Parabola's `your-freedom' package
> approach? It doesn't necessarily have to prevent installation of nonfree
> programs like `your-freedom' does, but could present a warning where the
> user must expressly agree to run the nonfree package the first time it's
> loaded (something similar to the `load-theme' warning).

That is good idea to make package for GNU ELPA that will recognize
those packages using proprietary software and warn users about freedom
issues.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le dim. 18 oct. 2020 à 06:10, Richard Stallman <rms@gnu.org> a écrit :
>   > In fact, I would go the extra mile and say Emacs should expressly warn
>   > users over the dangers of installing proprietary software from
>   > unofficial repositories
>
> That could be a good idea.  What would be good occasions on which to warn?
>
> Perhaps in list-packages when it sees a non-GNU repo, or when it sees MELPA?

With the subtlety that non-GNU Elpa would not be a non-GNU repo?

> Perhaps in describe-package and packageinstall, when the package comes
> from a non-GNU repo, or specifically from MELPA?

Not all users use package.el, some still use git (either manually or
through a package manager).

If such a warning becomes implemented, would that automatically make
package managers which do not (or cannot) display the warning as
dangerous as Melpa is currently?

---

Fwiw I still believe that information about Melpa on the webpage
should be available, with appropriate warnings.

When researching options, I found this webpage:
https://www.gnu.org/software/gnuzilla/addons.html
about IceCat extensions.

It does give the URL for the Mozilla store, with a warning that some
extensions may not be compatible with the goals of the GNU project,
and then it gives a list of vetted extensions.

Those add-ons are vetted for their licence and their free character.
Only the latter would be necessary for Melpa.

A problem with that list is that a lot of those extensions might have
since disappeared from the Mozilla store, but that is not something
that happens frequently with Melpa, especially when it comes to
extremely popular packages.

(Another problem, obviously, is that this page is obsolete, but as I
understand it that's due to technical changes in Firefox, not to the
page being incompatible with GNU policy.)

So, would a similar approach be viable for Melpa in the Emacs manual
and/or webpage?

The work of vetting packages would not be wasted, since the same list
could be immediately reused to populate non-GNU Elpa.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Perhaps in list-packages when it sees a non-GNU repo, or when it sees MELPA?

  > With the subtlety that non-GNU Elpa would not be a non-GNU repo?

Sorry that my words were confusing, but no, there would not be a
warning about Non-GNU ELPA.  We would make sure all the packages in
that repo would be ok to recommend, and indeed we would recommend it.


  > It does give the URL for the Mozilla store, with a warning that some
  > extensions may not be compatible with the goals of the GNU project,
  > and then it gives a list of vetted extensions.

As you've noticed, that issue and the MELPA issue are instances
of the same general class of situations.  They are affected by
the same factors.

However, the details are different, and what is right for one
is not necessarily right for the other.

The node References discusses these factors.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Philip K.]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > In fact, I would go the extra mile and say Emacs should expressly warn
>   > users over the dangers of installing proprietary software from
>   > unofficial repositories
>
> That could be a good idea.  What would be good occasions on which to warn?
>
> Perhaps in list-packages when it sees a non-GNU repo, or when it sees MELPA?
>
> Perhaps in describe-package and packageinstall, when the package comes
> from a non-GNU repo, or specifically from MELPA?
>
> Any other ideas?

This is a weaker suggestion: Maybe packages could have a tag to signal
that they are related to proprietary software, that could be then
displayed both via list-packages and describe-package? I'm fairly sure
that MELPA would encourage or even require people to add this tag.
Most software on MELPA is "ok", and people (like me) choose to use it
because is offers more packages than ELPA.

>                             (by the way, I always just assumed MELPA was
>   > somehow official and related to ELPA, because its name is so similar to
>   > ELPA).
>
> Yes, this is a source of confusion.
>
> Perhaps we should renamme GNU ELPA to a name that will avoid this confusion.
> Maybe GNU EP (GNU Emacs Packages)?
>
> EP is not meaningful to those who don't know what it means.  But
> neither is ELPA.  People understand it only if they have been told.
> So EP is no worse than ELPA.
>
> WDYT?

There shouldn't be any reason for ELPA to rename itself, it should be
clear that MELPA is not official, as it is not pre-configured.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is a weaker suggestion: Maybe packages could have a tag to signal
  > that they are related to proprietary software, that could be then
  > displayed both via list-packages and describe-package?

This approach depends on any number of people that we don't know
to include a special marker whenever they release packages that lead
the user to nonfree software.  Since these people eapparently
don't agree with our basic ideas about such packages, I don't think
we could count on them to do that.

  > > EP is not meaningful to those who don't know what it means.  But
  > > neither is ELPA.  People understand it only if they have been told.
  > > So EP is no worse than ELPA.
  > >
  > > WDYT?

  > There shouldn't be any reason for ELPA to rename itself, it should be
  > clear that MELPA is not official, as it is not pre-configured.

I have a feeling we are miscommunicating somehow here.  The point of
the renaming is simply to avoid a natural confusion in the minds of
people.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 19.10.2020 06:48, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > This is a weaker suggestion: Maybe packages could have a tag to signal
>    > that they are related to proprietary software, that could be then
>    > displayed both via list-packages and describe-package?
> 
> This approach depends on any number of people that we don't know
> to include a special marker whenever they release packages that lead
> the user to nonfree software.  Since these people eapparently
> don't agree with our basic ideas about such packages, I don't think
> we could count on them to do that.

MELPA to a large part consists of a database of "recipes", one for each 
package that it builds and distributes. This tag can be put inside 
recipes, and thus be controlled by MELPA maintainers, and not by the 
packages authors themselves.

If we provide some well-defined criteria for such tags, and pick a 
neutral-enough name, I don't see why the MELPA maintainers (who are 
quite reasonable people IME) wouldn't go for it.

Question is, would you agree to add MELPA to the default list of package 
archives if they do?

We'll also have to discuss how the user interface would look when 
dealing with those tags.

Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

This is related to MELPA subject and future Emacs repositories.

PROPRIETARY JAVASCRIPT ON MICROSOFT GITHUB ISSUE:
=================================================

I just guess that Github.com cannot be used with LibreJS, as I have
tried it, LibreJS report problems, this means all developers and
contributors to MELPA are automatically subjugated by Microsoft
Github, which is and was not the point of Emacs as free software part
of free operating system.

Additionally Github.com does not support many browsers, they publicly
say so, which in turn means that many users of free software browsers
from free software distributions are unable to access Github.com,
users of Github are supposed to use Firefox or Chrome or Microsoft
Browsers, while I am using Icecat and Iceweaseal-UXP from Hyperbola
GNU/Linux-libre -- so my experience with Github.com is narrowed, I
cannot access large parts of the website.

Developers using MELPA are misguided by Microsoft Github, and MELPA is
only following the popularity trend. Thus developers or old and new
contributors for reasons of getting included in MELPA are forced to
use non-free Javascript and browsers that are mostly not available on
free software distributions.

MELPA is telling users on its website that it is extensible, but
please "contribute your recipes via github, and we'll build the
packages", they are tageting Github, and thus lead Emacs users to use
non-free Javascript on Github. This is contradictory to purposes why
free software systems have been created.

There is https://savannah.nongnu.org then there is Trisquel hosting,
they would easily add Emacs packages without problem
https://devel.trisquel.info/users/sign_in then there are other
non-free Javascript and friendly free software hosting providers.

Could you maybe ask MELPA to switch to some free software hosting
sites for code, that way website would be accessible without using
proprietary Javascript.

That would be right direction to go for MELPA. 

It is good if we make incentive to those people who contribute to
MELPA to switch their hosting from Microsoft Github with proprietary
Javascript and their policies to some of free software hosting
providers, but what is really best is that MELPA switches to free
software hosting code providers.

AVOIDING EMACS PACKAGES WRAPPING PROPRIETARY SOFTWARE:
======================================================

This is other issue, it could be solved (as already somebody
mentioned) with one package to be developed in GNU ELPA, named similar
to "freedom-check" or even better as a built-in package that would
warn Emacs users about usage of non-free software or any other freedom
issues, it could ask user to disable those packages discovered on
system that are wrapping non-free software.

The package "freedom-check" would be for Emacs what is LibreJS on
browsers and what is "your-freedom" on Hyperbola GNU/Linux-libre and
other free software distributions.

It could disable or remove the packages that were installed to wrap
proprietary software. Additionally, it could disable repositories that
do not care about users' freedom and would tell users why is it so.

It would be good direction if such package is developed and then
beside being distributed on GNU ELPA, also injected in MELPA, as it
would be innovative package in users' interest, thus fully complying
with MELPA principles.

Let us not forget that inclusion of packages wrapping proprietary
software represents large security risk as well, not only that it
subjugates users or bring about a moral dillema.

OTHER SECURITY ISSUES ON MELPA:
===============================

For me largest security problem on MELPA is that Github.com is run by
Microsoft, we have no idea what principles they share beyond
commercial principles, and the more Emacs packages are hosted on
Microsoft Github, the more probability is there for mischievous
conduct or security breaches.

Same can be said for hosting providers that value free sofware,
security breaches are possible, but then we would know that people
maintaining the server do care of users enough, to prevent mischievous
conduct.

A package can be thus accepted in MELPA and become approved, then
later continuously updated, meaning without any control or
supervision, and then automatically offered to users. Backdoor can be
injected into users' Emacs and run on their computers. If I am wrong
on how MELPA works, you may tell me, that is what I understood from
their website.

SO FAR I KNOW there is no system of signing packages and verification
of such. I have verified MELPA git, and I cannot find that they are
using GnuPG or gpg or other system that packages were really made by
the author they claim to be made. This may be true at GNU ELPA too, I
did not verify it, but I know at that GNU ELPA is not automatically
built and offered to users blindly without verification (I at least
believe they are verified).

1. Github.com may not even know if somebody cracks some developer's
   account and changes few packages to misbehave, if they would be
   alerted, they would not maybe act in the best interest of the free
   software project, they would act in their own best interest.

2. MELPA managers do approve only GNU GPL or GPL compatible software
   to be included, they do not however continually verify or control
   the software, in fact they pull it by using recipes and build it
   automatically and offer automatically to users. Users in turn
   accept blindly packages, even though there is no mechanism to make
   sure that package was made by the author that was approved by
   MELPA. Let us say MELPA could maintain the list of public keys of
   authors, and then accept only packages that are signed by those
   same authors, before having them built, this would increase
   security this security issue, but not solve it, if packages are not
   verified by human.

3. Other issue is that users cannot trust MELPA only, as packages
   should be distributable from MELPA to other repositories, so each
   single package should be verifiable by user as well.

It is written in the Emacs manual: 41.4 Creating and Maintaining
Package Archives, that:

   Maintaining a public package archive entails a degree of
responsibility.  When Emacs users install packages from your archive,
those packages can cause Emacs to run arbitrary code with the
permissions of the installing user.  (This is true for Emacs code in
general, not just for packages.)  So you should ensure that your archive
is well-maintained and keep the hosting system secure.

   One way to increase the security of your packages is to “sign” them
using a cryptographic key.  If you have generated a private/public gpg
key pair, you can use gpg to sign the package like this:

     gpg -ba -o FILE.sig FILE

But it is not implemented into Emacs to verify packages being signed,
so my proposal is that Emacs get obligatory verification of a package
if such package is arriving from any repository and to warn user if
package was not signed. This would give initiative to MELPA to start
thinking about security issues.

That is one of reasons why Hyperbola GNU/Linux-libre and other
GNU/Linux distributions package some major Emacs packages, as that way
the package maintainers verify the package before it is included in
the free software distribution.

In the same manner Emacs should have a built-in package installation
procedure (that can be circumvented by users' configuration) to verify
all packages being installed by default.

Jean

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Kangas]

Jean Louis <bugs@gnu.support> writes:

>    One way to increase the security of your packages is to “sign” them
> using a cryptographic key.  If you have generated a private/public gpg
> key pair, you can use gpg to sign the package like this:
>
>      gpg -ba -o FILE.sig FILE
>
> But it is not implemented into Emacs to verify packages being signed,
> so my proposal is that Emacs get obligatory verification of a package
> if such package is arriving from any repository and to warn user if
> package was not signed. This would give initiative to MELPA to start
> thinking about security issues.
>
> That is one of reasons why Hyperbola GNU/Linux-libre and other
> GNU/Linux distributions package some major Emacs packages, as that way
> the package maintainers verify the package before it is included in
> the free software distribution.
>
> In the same manner Emacs should have a built-in package installation
> procedure (that can be circumvented by users' configuration) to verify
> all packages being installed by default.

We have signing of packages on the package archive side that is verified
by default when it exists.  See `package-check-signature'.  (If I'm not
mistaken, GNU ELPA signs packages but MELPA doesn't.  Please correct me
if I'm wrong.)

Note that package signatures still leaves us open to replay attacks.
See Bug#19479 and the branch scratch/package-security for an attempt to
improve the situation.

I think it would be useful if package archives could implement a
requirement for signed commits before building a new package.  This
could be optional or mandatory, and would buy us an additional layer of
protection against compromised developer credentials.

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Kangas <stefankangas@gmail.com> [2020-10-19 18:55]:
> We have signing of packages on the package archive side that is verified
> by default when it exists.  See `package-check-signature'.  (If I'm not
> mistaken, GNU ELPA signs packages but MELPA doesn't.  Please correct me
> if I'm wrong.)

Now I know about that. It was allow-unsigned as default, correct me if
mistaken. The more packages there are around, the more this becomes
potential problem, it is security hole, as warnings about potential
problems are too few.

Now when I turned it on, I cannot see or feel that some package was
verified, I tried installing from ELPA, but did not see any
difference, and cannot find any .sig files.

It would be good for user to get those verifications, as verification
should be doable personally.

Package signing is not ultimate security, it is just one level making
packages more secure.

> Note that package signatures still leaves us open to replay attacks.
> See Bug#19479 and the branch scratch/package-security for an attempt to
> improve the situation.

> I think it would be useful if package archives could implement a
> requirement for signed commits before building a new package.  This
> could be optional or mandatory, and would buy us an additional layer of
> protection against compromised developer credentials.

I have seen there apparently good recommendation for improvement of
package security.

But we do not have it.

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

> verified, I tried installing from ELPA, but did not see any

Side note: "ELPA" is the name of the protocol/infrastructure.
So now I don't know if you installed from Mermelada, GNU ELPA, MELPA, or
yet some other ELPA archive.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-19 20:31]:
> > verified, I tried installing from ELPA, but did not see any
> 
> Side note: "ELPA" is the name of the protocol/infrastructure.
> So now I don't know if you installed from Mermelada, GNU ELPA, MELPA, or
> yet some other ELPA archive.

Alright I understood it so from info document.

When enabled, that variable we spoke about that should be T to verify
packages, it is I assume general for any package archive. So I have
tried installing some from ELPA and did not see any difference.

By the way, Marmelade is not eatable any more.

There are just few well known, ELPA, Org and MELPA.

It would be good that package authors start publishing their own
repositories to decentralize and ensure better of security. I would
trust some authors more if I am getting their packages from their
domain, signed by their keys.

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

> tried installing some from ELPA and did not see any difference.

What difference did you expect to see?

> There are just few well known, ELPA, Org and MELPA.

Again, there is no ELPA archive named "ELPA".
Please try and avoid spreading the confusion between ELPA and GNU ELPA.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-19 21:02]:
> > tried installing some from ELPA and did not see any difference.
> 
> What difference did you expect to see?

I really expected to see that it is verified, like some message,
feedback in the Message buffer, but it just goes very silent, thus
user has to rely that package vas checked.

The name of function package-check-signature gives me more impression
that if I am the one who is setting the variable, that I would be in
some kind of control to know that signature of the package is checked.

Without message I see nothing, so Emacs is checking, but user does not
know that package was checked for signature.

I would rather expect message shown, just as it is not shown for
unsigned packages.

> > There are just few well known, ELPA, Org and MELPA.
> 
> Again, there is no ELPA archive named "ELPA".
> Please try and avoid spreading the confusion between ELPA and GNU ELPA.

That is right, I understand, so instead of ELPA which is meant for
general protocol, is better to say GNU ELPA.

Regarding packages in GNU ELPA, can I now assume they are all signed?

Is there a policy that GNU ELPA packages should be signed? That policy
would be good to have.

What I expect is a method for user to easily verify and know by which
key was which package signed, such function should exist.

I also expect that such verification should be by default, but default
was to accept unsigned, which is security issue in Emacs. Today is
common from package managers to expect at least automatic verification
of signatures.

It would be better if Emacs is delivered with the variable:
package-check-signature to be by default at least T or ALL to verify
all signatures.

Right now default is to accept unsigned. I am very surprised.

What I would like to see is:

  If the package-check-signature variable is for all types from
  packages, those downloaded any how, or from file system, and for
  those from archives, that ne wvariable is implemented so that
  packages downloaded from various centralized archives shall be
  checked for signature by DEFAULT.

- new variable: package-from-archive-check-signature to be by default
  T to check signature, with the warning in the description of the
  variable. 

- current variable: package-check-signature, for this one, I still
  study if it is valid for any packages, I can see I could install
  package from file without being checked for signature. So that is
  not my expectation, it probably works only for packages from
  archive, but not for any packages. My expectation is when this
  variable is set that it verifies all packages that user is
  installing.

  If there is only intended use of this variable for archive packages,
  then it is fine, but then documentation should not just say: Non-nil
  means to check package signatures when installing, but it should
  refer to package installing from archives

  I am installing often packages from file system by using
  package-install-file

- in that case that variable does not affect other sources but
  archives, users shall be warned in documentation strings or info
  that installing packages is risky activity.

What I wish to say, which may not be liked by many, I would like that
variable ensures that packages delivered through any current or future
onlinen archives are by default verified to be signed. This would
increase general security for Emacs users.

This would make necessity for MELPA to secure the packages by signing
them, thus giving little more safety to users.

As the situation how we have it now, those users of MELPA who value
the number of 4700+ packages being offered are more and more exposed
to potential risks.

There will be other repositories, anybody can then duplicate the
packages, but they cannot change signatures, they would need to make
new signatures, at least there would be some trace which key signed
which package.

Is making archive packages checkable by default for signatures
difficult for some practical reasons other than forcing MELPA to offer
new security level?

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

> I would rather expect message shown, just as it is not shown for
> unsigned packages.

`package.el` should emit a message when installing a package without any
signature, since that's the odd and undesirable case.  I find it
perfectly normal not to say anything when the signature check succeeded.

> Regarding packages in GNU ELPA, can I now assume they are all signed?

Of course.  It's been that way since Emacs-24.4, IIRC.

> Is there a policy that GNU ELPA packages should be signed?

Not sure what that would mean: *we* sign it, so there's no policy to
enforce.  At most there are bugs to fix if the sigs are missing
or incorrect.

> What I expect is a method for user to easily verify and know by which
> key was which package signed, such function should exist.

What does Debian do in this respect?

> I also expect that such verification should be by default, but default
> was to accept unsigned, which is security issue in Emacs.

2 reasons:
- the sig-checking code (i.e. PGP) might not be installed and we did
  not want to add it as a prerequisite.
- the signature system was introduced relatively shortly before it was
  deployed for Emacs-24.4, so we did not want to break it for the other
  ELPA archives.

Regarding the second point, AFAICT Melpa still doesn't sign its
packages, so its users presumably rely on `https` as their only line
of defense.  One of the main reasons might be that there is/was no easy
way to add other trusted keys to Emacs's keyring (tho the
`gnu-elpa-keyring-update` shows it can be done) so even if they signed
their packages their users would have to take some extra step to add
their key to the trusted keys.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-19 23:23]:
> > I would rather expect message shown, just as it is not shown for
> > unsigned packages.
> 
> `package.el` should emit a message when installing a package without any
> signature, since that's the odd and undesirable case.  I find it
> perfectly normal not to say anything when the signature check succeeded.
> 
> > Regarding packages in GNU ELPA, can I now assume they are all signed?
> 
> Of course.  It's been that way since Emacs-24.4, IIRC.
> 
> > Is there a policy that GNU ELPA packages should be signed?
> 
> Not sure what that would mean: *we* sign it, so there's no policy to
> enforce.  At most there are bugs to fix if the sigs are missing
> or incorrect.

It would be good to implement the policy.

> > What I expect is a method for user to easily verify and know by which
> > key was which package signed, such function should exist.
> 
> What does Debian do in this respect?

There are ways to verify package authenticity, so it is automated and
there is way to verify it package by package, I am on Hyperbola
GNU/Linux-libre, derivative of Archlinux, there is way to use pacman
package manager to verify authenticity.

Vasilij pointed out how it should be done. Verifications in Debian or
Archlinux how I see it, happen in real time during installation and
that is by default.

> > I also expect that such verification should be by default, but default
> > was to accept unsigned, which is security issue in Emacs.
> 
> 2 reasons:
> - the sig-checking code (i.e. PGP) might not be installed and we did
>   not want to add it as a prerequisite.

You know it better, maybe gnutls can be used as it is how I see it,
part of GNU Emacs here, but may not be part on every OS, I do not
know. It has OpenPGP API:
https://www.gnutls.org/manual/html_node/OpenPGP-API.html

So instead of using external gpg program, maybe you as developers
could use gnutls library and that API to create signatures for
packages in case that PGP/GnuPG cannot work.

> - the signature system was introduced relatively shortly before it was
>   deployed for Emacs-24.4, so we did not want to break it for the other
>   ELPA archives.

I understand and I find it unfortunate, and still suggest that it
becomes enabled now, and not years there after.

> Regarding the second point, AFAICT Melpa still doesn't sign its
> packages, so its users presumably rely on `https` as their only line
> of defense.  One of the main reasons might be that there is/was no easy
> way to add other trusted keys to Emacs's keyring (tho the
> `gnu-elpa-keyring-update` shows it can be done) so even if they signed
> their packages their users would have to take some extra step to add
> their key to the trusted keys.

And that is in best interest of users.

I think that it sounds tedious, yet it is in best interest to users.

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-20 00:53]:
> >> > Is there a policy that GNU ELPA packages should be signed?
> >> Not sure what that would mean: *we* sign it, so there's no policy to
> >> enforce.  At most there are bugs to fix if the sigs are missing
> >> or incorrect.
> > It would be good to implement the policy.
> 
> I don't know what that means (neither "the policy" nor "implement").

Rules of maintenance simply said:

- that every request to any ELPA goes over SSL connection, to totally
  disable non-SSL connections to archives. Many countries spy on their
  citizens, and in many of those countries citizens are using
  encryption features, even it could be illegal to use encryption. By
  using non-SSL connection or allowing such, possibility is there that
  user get in danger of life. This is one very real example, it will
  look unreal to many who are in normal countries. I have a friend in
  such country.

- that all packages are signed by default and that Emacs expects such
  by default

There is set of principles for Emacs Lisp packaging in the info
manual, those changes are only beneficial for future.

Read on this link that Vasilij have presented to me yesterday:
https://medium.com/hackernoon/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5

That does happen. Research the report on this site:
https://snyk.io/blog/javascript-frameworks-security-report-2019/

Compare insecurities on similar software package repositories with
other languages to Emacs, and implement policies to prevent
insecurities in future.

To implement means in this context to follow through, follow up,
follow out, carry out, implement, put through, go through -- pursue
to a conclusion or bring to a successful issue; (Wordnet) there may be
definitions in othe context. I speak of carrying out.

Policy means a plan of action adopted by an individual or social
group; "it was a policy of retribution"; "a politician keeps changing
his policies" (Wordnet) -- there may be other definitons in other
context, I speak of adopting plan of action for Emacs development.

> >> > What I expect is a method for user to easily verify and know by which
> >> > key was which package signed, such function should exist.
> >> What does Debian do in this respect?
> > There are ways to verify package authenticity,
> 
> How?  What does "package authenticity" mean?
> Do you get to see which key signed which package?

I skip this, I am sure you know it.

> > Vasilij pointed out how it should be done.  Verifications in Debian or
> > Archlinux how I see it, happen in real time during installation and
> > that is by default.
> 
> Right, just as we do with GNU ELPA, AFAICT.

It is not by default surprisingly to me. I had to turn on the option
to have packages verified for signatures.

> > So instead of using external gpg program, maybe you as developers
> > could use gnutls library and that API to create signatures for
> > packages in case that PGP/GnuPG cannot work.
> 
> The problem is not to create signatures (which we do on our own machines
> where we can easily make sure PGP is installed) but to verify them.

Maybe gnutls offers that API, I cannot know technically, I could see
the API is there. 

> >> - the signature system was introduced relatively shortly before it was
> >>   deployed for Emacs-24.4, so we did not want to break it for the other
> >>   ELPA archives.
> > I understand and I find it unfortunate, and still suggest that it
> > becomes enabled now, and not years there after.
> 
> The current default made sense then.  Maybe it should be changed
> now, indeed.

Thank you,

Think about the growing number of:

- users
- developers
- packages
- fascism and varieties of oppression in the world

Jean

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

>> >> > Is there a policy that GNU ELPA packages should be signed?
>> >> Not sure what that would mean: *we* sign it, so there's no policy to
>> >> enforce.  At most there are bugs to fix if the sigs are missing
>> >> or incorrect.
>> > It would be good to implement the policy.
>> I don't know what that means (neither "the policy" nor "implement").
> Rules of maintenance simply said:

So by "implement" you mean: write it in the doc that describes the ELPA protocol?

> - that every request to any ELPA goes over SSL connection, to totally
>   disable non-SSL connections to archives.  Many countries spy on their
>   citizens, and in many of those countries citizens are using
>   encryption features, even it could be illegal to use encryption.  By
>   using non-SSL connection or allowing such, possibility is there that
>   user get in danger of life.

The part I don't understand here is "or allowing such".  I see the
danger of using a non-encrypted connection but not the danger of
allowing such.

>> >> > What I expect is a method for user to easily verify and know by which
>> >> > key was which package signed, such function should exist.
>> >> What does Debian do in this respect?
>> > There are ways to verify package authenticity,
>> How?  What does "package authenticity" mean?
>> Do you get to see which key signed which package?
> I skip this, I am sure you know it.

No, I don't, that's why I asked.  More specifically, from where I sit,
I don't see much difference between the way Debian does it and the way
GNU ELPA does it.  And as a Debian user I don't know how to "easily
verify" nor "know by which key".

>> > Vasilij pointed out how it should be done.  Verifications in Debian or
>> > Archlinux how I see it, happen in real time during installation and
>> > that is by default.
>> Right, just as we do with GNU ELPA, AFAICT.
> It is not by default surprisingly to me.

It is by default in my book.

> I had to turn on the option to have packages verified for signatures.

I think those users who posted questions about signature verification
failures back when we changed to a new key are evidence to the contrary.

>> The problem is not to create signatures (which we do on our own machines
>> where we can easily make sure PGP is installed) but to verify them.
> Maybe gnutls offers that API, I cannot know technically, I could see
> the API is there.

Patch welcome (as long as it doesn't end up reimplementing part of GPG).


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-23 00:25]:
> >> >> > Is there a policy that GNU ELPA packages should be signed?
> >> >> Not sure what that would mean: *we* sign it, so there's no policy to
> >> >> enforce.  At most there are bugs to fix if the sigs are missing
> >> >> or incorrect.
> >> > It would be good to implement the policy.
> >> I don't know what that means (neither "the policy" nor "implement").
> > Rules of maintenance simply said:
> 
> So by "implement" you mean: write it in the doc that describes the
> ELPA protocol?

Thank you.

I meant to make it as a rule to sign packages, and that is should be
default in Emacs to accept only sign packages, that increases level of
security rather than leaving it acceptable for users to get unsigned
packages. It is definitely now everything about security, yet it is
one level.

Conditionally, if such feature is accepted in Emacs, than in Packaging
section would be described that packages have to be signed, and not
only that it is one way to increase the security.

> > - that every request to any ELPA goes over SSL connection, to totally
> >   disable non-SSL connections to archives.  Many countries spy on their
> >   citizens, and in many of those countries citizens are using
> >   encryption features, even it could be illegal to use encryption.  By
> >   using non-SSL connection or allowing such, possibility is there that
> >   user get in danger of life.
> 
> The part I don't understand here is "or allowing such".  I see the
> danger of using a non-encrypted connection but not the danger of
> allowing such.

My purpose was to tell you that if Emacs developers allow non-SSL by
default that users are automatically put at certain risks and that is
better to ask for SSL by default.

Users who maybe do not have SSL, they can turn it off for themselves.

> >> >> > What I expect is a method for user to easily verify and know by which
> >> >> > key was which package signed, such function should exist.
> >> >> What does Debian do in this respect?
> >> > There are ways to verify package authenticity,
> >> How?  What does "package authenticity" mean?
> >> Do you get to see which key signed which package?
> > I skip this, I am sure you know it.
> 
> No, I don't, that's why I asked.  More specifically, from where I sit,
> I don't see much difference between the way Debian does it and the way
> GNU ELPA does it.  And as a Debian user I don't know how to "easily
> verify" nor "know by which key".

You are right, it is vague to say easy.By easily I meant that there is
option to verify the package.

Let us see this as reference to some model design:
https://debian-handbook.info/browse/stable/sect.package-authentication.html

> When a third-party package source is added to the sources.list file,
> APT needs to be told to trust the corresponding GPG authentication key

So GNU Emacs users should maybe trust blindly packages from ELPA, but
not all packages by default. To trust other packages they should be
able transparently to import PGP keys and be able to see the
fingerprints.

By seeing fingerprints, users can at least see the same fingerprint on
the website. For better verification it should be necessary to contact
developers and make sure that fingerprints are same and valid.

Packages are meant to be distributable as well, if they are signed,
signature should be also fetched, but that is probably not original
design of Emacs. In my opinion, it should be. Signatures should be
inside of the package directory,
~/emcas.d/elpa/package-0.0/file.el.gpg

As if packages are distributable, one user from club A could simply
distribute packages on CD/USB/SD-card to other, or could transfer by
network. The user receiving packages beyond GNU ELPA archive, even
those without Internet, but having the trusted keys, should be able to
verify that packages have been signed by trusted keys, by central
authority in this case Emacs ELPA and trusted people maintaining
such.

-- 
Jean Louis

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

> I meant to make it as a rule to sign packages, and that is should be
> default in Emacs to accept only sign packages, that increases level of
> security rather than leaving it acceptable for users to get unsigned
> packages. It is definitely now everything about security, yet it is
> one level.

IOW, you're just restating in other words your request to change
`package-check-signature` to t?

> My purpose was to tell you that if Emacs developers allow non-SSL by
> default that users are automatically put at certain risks and that is
> better to ask for SSL by default.

And here you're suggesting that the default value of `package-archives`
should always use `https` regardless of the `gnutls-available-p`?

> So GNU Emacs users should maybe trust blindly packages from ELPA, but
> not all packages by default.  To trust other packages they should be
> able transparently to import GPG keys and be able to see the
> fingerprints.

IIUC I basically said the same earlier when I explained that maybe one
reason Melpa still doesn't sign its files is the lack of a clean/good
way for the user to add Melpa's keys to `package.el`s keyring.

Patches welcome in this area.

> Packages are meant to be distributable as well, if they are signed,
> signature should be also fetched, but that is probably not original
> design of Emacs. In my opinion, it should be. Signatures should be
> inside of the package directory,
> ~/emcas.d/elpa/package-0.0/file.el.gpg

This makes way too many assumptions to be worth discussing, IMO.
For the case of "single file ELPA package" (i.e. those files
distributed as a single .el file) maybe that can work without too much
trouble (tho there's still the issue of trusting the accompanying .elc
file), but for the more common packages distributed as tarballs, I think
this is completely impractical.

A saner approach might be to keep a "cache" of the packages in their
original (not-installed) form and make that available as a "local ELPA
archive" from which you can redistribute those packages to
other machines.  My impression is that this would be better served by
a separate package than by trying to add the feature directly to
`package.el`, especially since I suspect it would remain a fairly
unusual scenario.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-23 17:52]:
> > I meant to make it as a rule to sign packages, and that is should be
> > default in Emacs to accept only sign packages, that increases level of
> > security rather than leaving it acceptable for users to get unsigned
> > packages. It is definitely now everything about security, yet it is
> > one level.
> 
> IOW, you're just restating in other words your request to change
> `package-check-signature` to t?

Yes.

> > My purpose was to tell you that if Emacs developers allow non-SSL by
> > default that users are automatically put at certain risks and that is
> > better to ask for SSL by default.
> 
> And here you're suggesting that the default value of `package-archives`
> should always use `https` regardless of the `gnutls-available-p`?

I understand from that statement that probably not every platform will
have gnutls or whatever other solution. Let me mention that in some
countries governments forbid usage of various networks and software,
also encryption software, in particular I have a friend in
Iran and I know what can happen to a person. And networks are spied
over. This means in particular that misunderstanding or usage of
encryption tools could lead to unjust arrests and broken
families. Loading some packages from Emacs could automatically trigger
spying governments to abuse their citizens.

Reference:
https://security.stackexchange.com/questions/10992/encryption-laws-in-iran

There are other similar cases easy to find on search engines.

Now if that cannot be made default, then every non-SSL connection
should give serious warning to a user and should even ask user if one
wants to connect or not, because it is non-SSL. Such warning should
give good reference that data is visible on network and prone to Big
Brother's eyes.

> > Packages are meant to be distributable as well, if they are signed,
> > signature should be also fetched, but that is probably not original
> > design of Emacs. In my opinion, it should be. Signatures should be
> > inside of the package directory,
> > ~/emcas.d/elpa/package-0.0/file.el.gpg
> 
> This makes way too many assumptions to be worth discussing, IMO.
> For the case of "single file ELPA package" (i.e. those files
> distributed as a single .el file) maybe that can work without too much
> trouble (tho there's still the issue of trusting the accompanying .elc
> file), but for the more common packages distributed as tarballs, I think
> this is completely impractical.

Maybe tar can be signed as such?

> A saner approach might be to keep a "cache" of the packages in their
> original (not-installed) form and make that available as a "local ELPA
> archive" from which you can redistribute those packages to
> other machines.

Yes. For me is no problem. I speak for wide user base. Ability for
each ELPA to download full set of packages and keep it as local ELPA
would be convenient for many users who do not have stable Internet.

-- 
Jean Louis

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

>> IOW, you're just restating in other words your request to change
>> `package-check-signature` to t?
> Yes.

I'm find with it, but it's not up to me and I think it will require more
code changes so that errors are reported in a way that's more friendly
to the user (e.g. so the users can figure out whether the problem is
with the signature, the lack of key, or the lack of GPG, for example).

I suggest you `M-x report-emacs-bug` with this specific request and
accompany it with a patch that improves the error handling.

>> > My purpose was to tell you that if Emacs developers allow non-SSL by
>> > default that users are automatically put at certain risks and that is
>> > better to ask for SSL by default.
>> And here you're suggesting that the default value of `package-archives`
>> should always use `https` regardless of the `gnutls-available-p`?
> I understand from that statement that probably not every platform will
> have gnutls or whatever other solution.

I believe it is available on all the platforms we support, but you can
build Emacs without support for it (and under Windows, IIUC, you can
build it with support for gnutls and later run it without libgnutls in
which case it'll behave pretty much as if it had been built without
support for gnutls).

> Let me mention that ...

Yes, you already said so, and I believe it's been common knowledge on
this mailing-list for a while.

>> This makes way too many assumptions to be worth discussing, IMO.
>> For the case of "single file ELPA package" (i.e. those files
>> distributed as a single .el file) maybe that can work without too much
>> trouble (tho there's still the issue of trusting the accompanying .elc
>> file), but for the more common packages distributed as tarballs, I think
>> this is completely impractical.
> Maybe tar can be signed as such?

It is.  But the installed files are not the tarball, and it's difficult
to reproduce the tarball from the installed files in order to check that
they still "match the signature".

>> A saner approach might be to keep a "cache" of the packages in their
>> original (not-installed) form and make that available as a "local ELPA
>> archive" from which you can redistribute those packages to
>> other machines.
> Yes.  For me is no problem.  I speak for wide user base.  Ability for
> each ELPA to download full set of packages and keep it as local ELPA
> would be convenient for many users who do not have stable Internet.

You can mirror GNU ELPA via rsync.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-23 21:26]:
> You can mirror GNU ELPA via rsync.

Great. I cannot find instructions for that on website, do you have a
reference?

-- 
Jean Louis

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

>> You can mirror GNU ELPA via rsync.
> Great. I cannot find instructions for that on website, do you have a
> reference?

searching for "elpa rsync" lead me (after a few steps) to:

    https://lists.gnu.org/archive/html/emacs-devel/2016-07/msg00709.html


-- Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Side note: "ELPA" is the name of the protocol/infrastructure.
> So now I don't know if you installed from Mermelada, GNU ELPA, MELPA, or
> yet some other ELPA archive.

Is Mermelada the Hispanic version of Marmalade?

PS. In other news, I get a 502 Bad Gateway error now when visiting
     https://marmalade-repo.org/

Re: Proposal to include obligatory PGP verification of packages from any repository

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 298 bytes --]

> PS. In other news, I get a 502 Bad Gateway error now when visiting
>      https://marmalade-repo.org/

I've poked other people to poke Nic Ferrier who then promised on Twitter
to look into it soonish.  His website seems to be affected as well, so I
hope it's just a misconfiguration of the host.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

>> Side note: "ELPA" is the name of the protocol/infrastructure.
>> So now I don't know if you installed from Mermelada, GNU ELPA, MELPA, or
>> yet some other ELPA archive.
> Is Mermelada the Hispanic version of Marmalade?

Interesting mistake, indeed ;-)


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 15712 bytes --]

As someone working in information security, I feel compelled to reply to
this in detail as your email contains several claims I am not
comfortable with being taken at face value.

> I just guess that Github.com cannot be used with LibreJS, as I have
> tried it, LibreJS report problems, this means all developers and
> contributors to MELPA are automatically subjugated by Microsoft
> Github, which is and was not the point of Emacs as free software part
> of free operating system.

There is a difference between trying (enabling LibreJS and looking at
how many scripts it blocked) and trying (enabling LibreJS and actually
using the website while paying attention to how functional it is).  Out
of curiosity I've attempted the latter as it's a more meaningful test,
using the latest version of Firefox Developer Edition on Arch:

- Repositories can be created
- Issues can be created
- Forks can be created
- Pull requests can be created, with the limitation that the merge
  preview is broken.  This doesn't matter though as it's up to MELPA
  maintainers to review the pull request.
- Account creation is broken.  I can fill out the first screen
  (username, email address, password), then get stuck on the second
  screen as it relies on an external CAPTCHA service.  I suspect I can
  complete the process by temporarily allowing the JS script and
  continue contributing to repositories, but haven't verified that
  part.

My verdict so far: Github almost passes the test.  If one does have an
account already (from what I've seen you do), there is no excuse for you
to not contribute.  If one doesn't, they'll have to ponder whether
whitelisting the CAPTCHA is worth it.

> Additionally Github.com does not support many browsers, they publicly
> say so, which in turn means that many users of free software browsers
> from free software distributions are unable to access Github.com,
> users of Github are supposed to use Firefox or Chrome or Microsoft
> Browsers, while I am using Icecat and Iceweaseal-UXP from Hyperbola
> GNU/Linux-libre -- so my experience with Github.com is narrowed, I
> cannot access large parts of the website.

From a security perspective it's a bad idea to use any browser outside
of the main stream as only the largest browser vendors manage keeping up
to speed with security issues.  While one can use a browser based on
them, there will be issues.  As a side point, Debian settled its
trademark dispute with Mozilla and no longer offers rebranded versions
of Firefox.  Therefore Debian users will not face browser
incompatibility problems and avoid such security issues.

Something else worth trying is hub [1].  It seems to allow performing
the expected contribution workflow from the command line.

> Developers using MELPA are misguided by Microsoft Github, and MELPA is
> only following the popularity trend. Thus developers or old and new
> contributors for reasons of getting included in MELPA are forced to
> use non-free Javascript and browsers that are mostly not available on
> free software distributions.
> 
> MELPA is telling users on its website that it is extensible, but
> please "contribute your recipes via github, and we'll build the
> packages", they are tageting Github, and thus lead Emacs users to use
> non-free Javascript on Github. This is contradictory to purposes why
> free software systems have been created.

This is a regrettable aspect of MELPA (although I disagree with non-free
JavaScript or security being a problem), I don't have expectations for
it to change as MELPA's philosophy revolves around the central principle
of going with the most convenient mode of operation for developers to
publish their packages:

- Use Github, the most popular gratis Git hosting service
- Use Github's contribution workflow (forking and "pull requests")
- Automatically fetch packages from developers (as opposed to having
  them submit/upload new versions)
- Automatically rebuild packages after each detected commit
- Remove support for unpopular alternative workflows to reduce
  maintenance effort (Emacswiki, Bazaar, CVS, Darcs, Fossil, Subversion)

One does not share a package on MELPA to further the goals of the free
software movement, one does so to reach as many Emacs users as possible
with relatively little effort (which excludes GNU ELPA, we'll have to
see whether the same holds true for non-GNU ELPA).

> There is https://savannah.nongnu.org then there is Trisquel hosting,
> they would easily add Emacs packages without problem
> https://devel.trisquel.info/users/sign_in then there are other
> non-free Javascript and friendly free software hosting providers.

Savannah does not strike me as an adequate alternative for people used
to Github.  For starters it requires approval of each project before
it's publicly visible.  I'll have to do a proper evaluation of it though
before I can comment further on this aspect.  Gitlab is the closest
equivalent, though not perfect either.  For minimalism and integration
of an email-centric workflow sourcehut looks ideal, once it leaves its
alpha status.

> Could you maybe ask MELPA to switch to some free software hosting
> sites for code, that way website would be accessible without using
> proprietary Javascript.
> 
> That would be right direction to go for MELPA. 
> 
> It is good if we make incentive to those people who contribute to
> MELPA to switch their hosting from Microsoft Github with proprietary
> Javascript and their policies to some of free software hosting
> providers, but what is really best is that MELPA switches to free
> software hosting code providers.

As elaborated above, it is unlikely to happen.  Adherence to GNU
principles for non-GNU projects is not a priority.  See also the MELPA
maintainers' response to a thread you've created [2].

> This is other issue, it could be solved (as already somebody
> mentioned) with one package to be developed in GNU ELPA, named similar
> to "freedom-check" or even better as a built-in package that would
> warn Emacs users about usage of non-free software or any other freedom
> issues, it could ask user to disable those packages discovered on
> system that are wrapping non-free software.
> 
> The package "freedom-check" would be for Emacs what is LibreJS on
> browsers and what is "your-freedom" on Hyperbola GNU/Linux-libre and
> other free software distributions.
> 
> It could disable or remove the packages that were installed to wrap
> proprietary software. Additionally, it could disable repositories that
> do not care about users' freedom and would tell users why is it so.
> 
> It would be good direction if such package is developed and then
> beside being distributed on GNU ELPA, also injected in MELPA, as it
> would be innovative package in users' interest, thus fully complying
> with MELPA principles.

This is an interesting idea, though it will require continuous updates
for best effect.  Even if it doesn't prove to be effective in practice,
having a detailed list of what exactly is a problem to one's freedoms
would be a useful side effect for users caring about this.

> For me largest security problem on MELPA is that Github.com is run by
> Microsoft, we have no idea what principles they share beyond
> commercial principles, and the more Emacs packages are hosted on
> Microsoft Github, the more probability is there for mischievous
> conduct or security breaches.
> 
> Same can be said for hosting providers that value free sofware,
> security breaches are possible, but then we would know that people
> maintaining the server do care of users enough, to prevent mischievous
> conduct.

This argument does not follow.  A code forge can have security issues,
no matter whether its code can be freely audited or the political
alignment of its owners.  Merely being concerned about free software
doesn't mean that security issues will be investigated and acted upon,
neither does ownership by an entity pursuing different goals mean it
will neglect security issues.

I've briefly looked into Savannah's source code and found enough
evidence of it not being securely designed and requiring an in-depth
source code review.  While talking to a security-related contact person
I've been told that there was indeed a little-known security breach long
time ago.  Linus' law [3] seems a more appropriate hypothesis to explain
this behavior, depending on the visibility and amount of people actively
looking into a code forge, there will be more publicized security
incidents and attempts to improve overall security of the project.
Freedom and political alignment of the project are at best tangentially
related.

> A package can be thus accepted in MELPA and become approved, then
> later continuously updated, meaning without any control or
> supervision, and then automatically offered to users. Backdoor can be
> injected into users' Emacs and run on their computers. If I am wrong
> on how MELPA works, you may tell me, that is what I understood from
> their website.

That is indeed correct, there is no further quality assurance after the
initial submission.  It takes a significant amount of time for that
submission to be verified due to the few reviewers and given the sheer
amount of packages (4750 packages at the time of writing), I do not see
this working out with their current team.  For comparison, Arch has
"Trusted Users" responsible for their official packages, with anywhere
between 5 and 90 packages assigned to each of them.  Rest assured that
they don't perform security audits, only basic quality assurance as
packages are updated and bugs trickle in.  Having the kind of work force
a GNU/Linux distribution has is a luxury, not something to take for
granted.

That aside, how does this quality assurance look like for GNU ELPA?  Is
it users subscribing to a commit mailing list and looking out for
anything suspect?  Fewer releases overall?  A basic level of vetting by
asking for copyright assignment?  I doubt that code is audited for every
release either and that security issues are instead dealt with in a
reactive instead of proactive manner (meaning, as soon as one is known
of, appropriate measures such as a security release are taken).  There
is a lot more work to be done here, such as defining a comprehensive
threat model to work with.  Another important aspect: Say that non-GNU
ELPA manages to catch up with MELPA in terms of package count, does the
existing review system still work?  Will new measures be necessary to
guarantee security (for example by designing packages in terms of a more
restrictive approach with permissions, sandboxing and whatnot)?  I
strongly suspect that they will be and merely mirroring MELPA packages
won't suffice.

> SO FAR I KNOW there is no system of signing packages and verification
> of such. I have verified MELPA git, and I cannot find that they are
> using GnuPG or gpg or other system that packages were really made by
> the author they claim to be made. This may be true at GNU ELPA too, I
> did not verify it, but I know at that GNU ELPA is not automatically
> built and offered to users blindly without verification (I at least
> believe they are verified).

As mentioned above, MELPA's principle of going with the most convenient
option makes this a non-starter.  Vetting package authors for their
identity, requiring a GPG key and signed releases is something I'd
rather expect for GNU ELPA than them.

> 1. Github.com may not even know if somebody cracks some developer's
>    account and changes few packages to misbehave, if they would be
>    alerted, they would not maybe act in the best interest of the free
>    software project, they would act in their own best interest.

Github offers 2FA to protect accounts and has a dedicated security team
reacting to incidents.  I don't see signs of either for Savannah.  The
only point in favor of Savannah is it being relatively unknown and
therefore of less interest to attackers, but I wouldn't rely on that as
my only level of defense.

That aside, it's not even necessary to hack an account to distribute
malicious software.  Popular package archives for other programming
languages have seen attacks such as typo squatting [4]creating
(uploading a package with a misleading name) and hostile maintainers [5]
(where a package is transfered to a new maintainer who then abuses the
trust created by the package by adding malicious code to it).  I have to
confess I've pondered creating a malicious package for demonstrating how
easy it would be to sneak something in without anyone noticing, but I'm
afraid it will only lead to more emails sowing Fear, Uncertainty and
Doubt.  Or people weaponizing the code, not sure is likelier.

> 2. MELPA managers do approve only GNU GPL or GPL compatible software
>    to be included, they do not however continually verify or control
>    the software, in fact they pull it by using recipes and build it
>    automatically and offer automatically to users. Users in turn
>    accept blindly packages, even though there is no mechanism to make
>    sure that package was made by the author that was approved by
>    MELPA. Let us say MELPA could maintain the list of public keys of
>    authors, and then accept only packages that are signed by those
>    same authors, before having them built, this would increase
>    security this security issue, but not solve it, if packages are not
>    verified by human.

For this to be effective, one needs to build upon trust.  I doubt the
average Emacs user has thought in depth what package developers to trust
and whether they should verify packages installed to come from them.
And again, with MELPA's philosophy being one of convenience, I don't see
that happening.

>    One way to increase the security of your packages is to “sign” them
> using a cryptographic key.  If you have generated a private/public gpg
> key pair, you can use gpg to sign the package like this:
> 
>      gpg -ba -o FILE.sig FILE
> 
> But it is not implemented into Emacs to verify packages being signed,
> so my proposal is that Emacs get obligatory verification of a package
> if such package is arriving from any repository and to warn user if
> package was not signed. This would give initiative to MELPA to start
> thinking about security issues.

Emacs does have basic verification already, but it is far from ideal.
The failure case is not handled well at all, users are not told what to
do when it fails verifying a package and are inclined to instead disable
the mechanism once and for all.  If an Emacs is too old to contain the
appropriate GPG keys, the keyring will be needed to be updated outside
of Emacs, a task that can fail for opaque reasons due to key servers
being unreachable.  There is initial work in form of the
gnu-elpa-keyring-update package distributed via GNU ELPA, but it suffers
from a bootstrapping problem and needs to be installed *before* the keys
become outdated.  Until this issue is solved, I cannot imagine making
this opt-out or even mandatory.  For comparison, Arch ships its own
package manager specific GPG keyring and updates it during each package
update operation.

[1]: https://github.com/github/hub
[2]: https://github.com/melpa/melpa/issues/7185
[3]: "Given enough eyeballs, all bugs are shallow."
[4]: https://incolumitas.com/2016/06/08/typosquatting-package-managers/
[5]: https://snyk.io/blog/a-post-mortem-of-the-malicious-event-stream-backdoor/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

ELPA security (was: Proposal to include obligatory PGP verification of packages from any repository)

[Stefan Monnier]

> That aside, how does this quality assurance look like for GNU ELPA?
> Is it users subscribing to a commit mailing list and looking out for
> anything suspect?

Kind of, tho there's no such statement of intent.

There's a mailing-list where most(!) commits get sent, and some people
subscribe to it, but there's no reason to assume that they subscribe to
it specifically to review commits, let alone keeping an eye out for
quality or security issues.

I'm pretty sure several commits sent to the list aren't (re)viewed by
anyone at all.

I do subscribe to that mailing-list and perform a basic amount of
reviewing, but it's more geared towards catching mishaps and helping
contributors improve their code.  It wouldn't take much effort for an
attacker to make sure I don't look at his backdoor, I think.

> Fewer releases overall?

Definitely.

> I doubt that code is audited for every release either and that
> security issues are instead dealt with in a reactive instead of
> proactive manner (meaning, as soon as one is known of, appropriate
> measures such as a security release are taken).

Definitely.  The main source of "proactive" security is the fact that
the code is fully visible to anyone who wants to look, and the presumed
low benefits of introducing a backdoor in an ELisp package.

> Another important aspect: Say that non-GNU ELPA manages to catch up
> with MELPA in terms of package count, does the existing review system
> still work?

I think it'd largely end up similar to MELPA.

> Will new measures be necessary to guarantee security (for example by
> designing packages in terms of a more restrictive approach with
> permissions, sandboxing and whatnot)?  I strongly suspect that they
> will be and merely mirroring MELPA packages won't suffice.

The nature of ELisp makes sandboxing/permissions pretty difficult to
implement, so I think we'll just hope for the best, like MELPA does.

An alternative might be a system where we force/require some minimal
amount of code review before publishing a package, which may work if the
ELPA archive is popular enough that the annoyance of delayed releases
motivates people to contribute to the review effort rather than to
move to a more permissive archive.


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Jean Louis]

* Vasilij Schneidermann <mail@vasilij.de> [2020-10-19 21:29]:
> There is a difference between trying (enabling LibreJS and looking at
> how many scripts it blocked) and trying (enabling LibreJS and actually
> using the website while paying attention to how functional it is).  Out
> of curiosity I've attempted the latter as it's a more meaningful test,
> using the latest version of Firefox Developer Edition on Arch:

I am not using Arch for reason that they allow any kind of proprietary
software in GNU/Linux distribution, there are some freedom issues with
Firefox, so I am using IceCat derivative or Iceweasel-uxp and I could
not access activities.

Further, I am faced with the warning by Github that my browser is not
supported.

And I do know that with LibreJS enabled, many functions work, some
functions do not work. I did not test extensively.

> My verdict so far: Github almost passes the test.  If one does have an
> account already (from what I've seen you do), there is no excuse for you
> to not contribute.

I would also say if developers can use Github they can as well use
various libre online hosting systems.

> > Additionally Github.com does not support many browsers, they publicly
> > say so, which in turn means that many users of free software browsers
> > from free software distributions are unable to access Github.com,
> > users of Github are supposed to use Firefox or Chrome or Microsoft
> > Browsers, while I am using Icecat and Iceweaseal-UXP from Hyperbola
> > GNU/Linux-libre -- so my experience with Github.com is narrowed, I
> > cannot access large parts of the website.
> 
> From a security perspective it's a bad idea to use any browser outside
> of the main stream as only the largest browser vendors manage keeping up
> to speed with security issues. While one can use a browser based on
> them, there will be issues.  As a side point, Debian settled its
> trademark dispute with Mozilla and no longer offers rebranded versions
> of Firefox.  Therefore Debian users will not face browser
> incompatibility problems and avoid such security issues.

Since 2016, I am not using Debian GNU/Linux on our internal computers
for reason that they guide users to non-free firmware and help users
enable non-free repositories. I am using exclusively free software
operating systems, see: http://www.gnu.org/distros/free-distros.html

I am not sure for all of them, but I do not use what you use.

And I am not following the main stream, and I am behind bad networks,
my case is particular and personal, I am using eww, and even by using
eww I can find github.com link to clone repository, in that sense
github.com is usable, that still does not make their javascript free
software, and users of free software should not be guided to non-free
software, be it javascript.

Trisquel project, and Savannah, many others, like Gitlab, they also
offer free code hosting without proprietary software. It means all
those developers could switch.

> One does not share a package on MELPA to further the goals of the free
> software movement, one does so to reach as many Emacs users as possible
> with relatively little effort (which excludes GNU ELPA, we'll have to
> see whether the same holds true for non-GNU ELPA).

That is exactly what we are both talking about, thanks.

Packages from MELPA are not signed, can you maybe recommend from
security perspective that each author start signing their packages?

> > There is https://savannah.nongnu.org then there is Trisquel hosting,
> > they would easily add Emacs packages without problem
> > https://devel.trisquel.info/users/sign_in then there are other
> > non-free Javascript and friendly free software hosting providers.
> 
> Savannah does not strike me as an adequate alternative for people used
> to Github.  For starters it requires approval of each project before
> it's publicly visible.  I'll have to do a proper evaluation of it though
> before I can comment further on this aspect.  Gitlab is the closest
> equivalent, though not perfect either.  For minimalism and integration
> of an email-centric workflow sourcehut looks ideal, once it leaves its
> alpha status.

Maybe also the money is problem, as maintainers would by hosting
themselves, be paying themselves.

But price of freedom we all pay.

I make extensive travels to find proper computers that I can liberate
from Intel spying engine, and install libreboot, so the price I am
paying personally, so the price for hosting can be also somehow paid.

Various projects such as Trisquel would be welcoming new Emacs package
developers on their Gitlab instances.

> > For me largest security problem on MELPA is that Github.com is run by
> > Microsoft, we have no idea what principles they share beyond
> > commercial principles, and the more Emacs packages are hosted on
> > Microsoft Github, the more probability is there for mischievous
> > conduct or security breaches.
> > 
> > Same can be said for hosting providers that value free sofware,
> > security breaches are possible, but then we would know that people
> > maintaining the server do care of users enough, to prevent mischievous
> > conduct.
> 
> This argument does not follow.  A code forge can have security issues,
> no matter whether its code can be freely audited or the political
> alignment of its owners.  Merely being concerned about free software
> doesn't mean that security issues will be investigated and acted upon,
> neither does ownership by an entity pursuing different goals mean it
> will neglect security issues.

I am very practical, I am trusting OpenBSD various teams to make the
LibreSSL how it should be, I am trusting them with various security
issues like in OpenSSH, for reasons of their principles set long time
ago. So I am trusting various smaller communities such as those of
Guix, Trisquel, Hyperbola GNU/Linux-libre, for reason that they act
upon the notices related to freedom, security, you name it. For their
agility and responsiveness. I trust GNU packages and people behind it
for obviously good responsiveness in handling user issues or bugs.

I cannot trust Archlinux to handle issues of freedom because their
policy is such that they will include proprietary software if
proprietor is allowing them, and if they have such policy, then I
cannot trust them with security issues too.

You are right that all hostings or let us say digital backgrounds can
be compromised, but if proprietary background is compromised such as
Microsoft Github, for reasons of their business interests I cannot
trust them to be transparent, unless third party have found security
issue.

For free software communities, some of them I mentioned, I can trust
them based on my experience with them, to be very transparent with
security issues.

> > A package can be thus accepted in MELPA and become approved, then
> > later continuously updated, meaning without any control or
> > supervision, and then automatically offered to users. Backdoor can be
> > injected into users' Emacs and run on their computers. If I am wrong
> > on how MELPA works, you may tell me, that is what I understood from
> > their website.
> 
> That is indeed correct, there is no further quality assurance after the
> initial submission.  It takes a significant amount of time for that
> submission to be verified due to the few reviewers and given the sheer
> amount of packages (4750 packages at the time of writing), I do not see
> this working out with their current team.  For comparison, Arch has
> "Trusted Users" responsible for their official packages, with anywhere
> between 5 and 90 packages assigned to each of them.  Rest assured that
> they don't perform security audits, only basic quality assurance as
> packages are updated and bugs trickle in.  Having the kind of work force
> a GNU/Linux distribution has is a luxury, not something to take for
> granted.

What that setup of Archlinux does is that it decentralizes the package
delivery, so some of those maintainers will indeed look into code and
report issues when found, there is more care and automatically there
is more responsibility. I do not say how much, I just can think that
it is.

Security and freedom issues are mostly promptly handled in those FSF
endorsed distributions, those are significant differences originating
from various group attitudes or set of principles.

> That aside, how does this quality assurance look like for GNU ELPA?

I understand it fully and I wish more security in any kind of package
repository for GNU Emacs, as users are downloading blindly "packages"
many of them will find it funny, and most of time nothing bad will
happen until it happens.

> Is it users subscribing to a commit mailing list and looking out for
> anything suspect?  Fewer releases overall?  A basic level of vetting
> by asking for copyright assignment?  I doubt that code is audited
> for every release either and that security issues are instead dealt
> with in a reactive instead of proactive manner (meaning, as soon as
> one is known of, appropriate measures such as a security release are
> taken).  There is a lot more work to be done here, such as defining
> a comprehensive threat model to work with.  Another important
> aspect: Say that non-GNU ELPA manages to catch up with MELPA in
> terms of package count, does the existing review system still work?
> Will new measures be necessary to guarantee security (for example by
> designing packages in terms of a more restrictive approach with
> permissions, sandboxing and whatnot)?  I strongly suspect that they
> will be and merely mirroring MELPA packages won't suffice.

Thank you for thinkering about security. Resource scarcity is security
issue.

For GNU ELPA I know that principle is that packages are assigned with
copyright to FSF and by other principles which I am not authorized to
represent.

While it is great to have large choice, number of packages in an
archive is for sure not the principle for GNU ELPA.

For me personally, that count is not a feature, it is anti feature,
the more packages they are, the more I get concerned for reasons
explained by me, and reasons explained by you.

> > SO FAR I KNOW there is no system of signing packages and verification
> > of such. I have verified MELPA git, and I cannot find that they are
> > using GnuPG or gpg or other system that packages were really made by
> > the author they claim to be made. This may be true at GNU ELPA too, I
> > did not verify it, but I know at that GNU ELPA is not automatically
> > built and offered to users blindly without verification (I at least
> > believe they are verified).
> 
> As mentioned above, MELPA's principle of going with the most convenient
> option makes this a non-starter.  Vetting package authors for their
> identity, requiring a GPG key and signed releases is something I'd
> rather expect for GNU ELPA than them.

Turn on the option package-check-signature and start verifying.

> > 1. Github.com may not even know if somebody cracks some developer's
> >    account and changes few packages to misbehave, if they would be
> >    alerted, they would not maybe act in the best interest of the free
> >    software project, they would act in their own best interest.
> 
> Github offers 2FA to protect accounts and has a dedicated security team
> reacting to incidents.

See this example:
https://pberba.github.io/security/2020/05/28/lastpass-phishing/

You are right that neither free software repositories need be secure,
they have though different motivation and different level of
transparency as they do not work for their number of sales and good
image, they are more proud if they find their security incidents
themselves, speaking of OpenBSD principles of work.

> That aside, it's not even necessary to hack an account to distribute
> malicious software.  Popular package archives for other programming
> languages have seen attacks such as typo squatting [4]creating
> (uploading a package with a misleading name) and hostile maintainers [5]
> (where a package is transfered to a new maintainer who then abuses the
> trust created by the package by adding malicious code to it).  I have to
> confess I've pondered creating a malicious package for demonstrating how
> easy it would be to sneak something in without anyone noticing, but I'm
> afraid it will only lead to more emails sowing Fear, Uncertainty and
> Doubt.  Or people weaponizing the code, not sure is likelier.

You have got references, to me it is obvious. What I am surprised is
that I do not know of serious security incidents within Emacs
packages, that does not mean they are not about to come.

> Emacs does have basic verification already, but it is far from ideal.
> The failure case is not handled well at all, users are not told what to
> do when it fails verifying a package and are inclined to instead disable
> the mechanism once and for all.

Right.

> If an Emacs is too old to contain the appropriate GPG keys, the
> keyring will be needed to be updated outside of Emacs, a task that
> can fail for opaque reasons due to key servers being unreachable.
> There is initial work in form of the gnu-elpa-keyring-update package
> distributed via GNU ELPA, but it suffers from a bootstrapping
> problem and needs to be installed *before* the keys become outdated.
> Until this issue is solved, I cannot imagine making this opt-out or
> even mandatory.  For comparison, Arch ships its own package manager
> specific GPG keyring and updates it during each package update
> operation.

That is about that what I expect for Emacs as well!

Jean

P.S. Think about implementing OMEMO in Jabber.el, my offer stands

Re: Proposal to include obligatory PGP verification of packages from any repository

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I have not tried to study each point in your message -- there were so
many -- but I noticed criticism of Savannah for not offering
two-factor authentication.

When I was asked to do this, I couldn't do it, because it depended on
carrying a protable listening and surveillance device (aka cellular
phone).  MIT had to make a special exception for me, turning that
requirement off, when it demanded that I access its administrative
systems.

Savannah must not do this if it requires the user to use nonfree
programs.

If people are interested in discussing this point, let's use
gnu-prog-discuss@gnu.org as it is off-topic here.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal to include obligatory PGP verification of packages from any repository

[Stefan Monnier]

> I have not tried to study each point in your message -- there were so
> many -- but I noticed criticism of Savannah for not offering
> two-factor authentication.
>
> When I was asked to do this, I couldn't do it, because it depended on
> carrying a protable listening and surveillance device (aka cellular
> phone).

I don't see why: the way Gitlab does 2FA relies either on TOTP or
on a secure-key (such as the somu), neither of which requires
any kind of network correction (cellular or other).

See https://en.wikipedia.org/wiki/FIDO2_Project
and https://en.wikipedia.org/wiki/Time-based_One-time_Password_algorithm


        Stefan

Re: Proposal to include obligatory PGP verification of packages from any repository

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > When I was asked to do this, I couldn't do it, because it depended on
  > > carrying a protable listening and surveillance device (aka cellular
  > > phone).

  > I don't see why:

Why MIT chose to offer the options it chose and no others?
I don't know.

Why I couldn't use them?
I couldn't use them because each required me to run nonfree software.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal to include obligatory PGP verification of packages from any repository

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

> I have not tried to study each point in your message -- there were so
> many -- but I noticed criticism of Savannah for not offering
> two-factor authentication.
> 
> When I was asked to do this, I couldn't do it, because it depended on
> carrying a protable listening and surveillance device (aka cellular
> phone).  MIT had to make a special exception for me, turning that
> requirement off, when it demanded that I access its administrative
> systems.
> 
> Savannah must not do this if it requires the user to use nonfree
> programs.

There are many ways of doing 2FA and normally it's opt-in.

> If people are interested in discussing this point, let's use
> gnu-prog-discuss@gnu.org as it is off-topic here.

Sure, please include me in your mail there as I'm not subscribed to that
mailing list.  I could probably help out with the security side of
Savannah.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: Proposal for an Emacs User Survey

[Drew Adams]

> MELPA to a large part consists of a database of "recipes", one for each
> package that it builds and distributes. This tag can be put inside
> recipes, and thus be controlled by MELPA maintainers, and not by the
> packages authors themselves.
> 
> If we provide some well-defined criteria for such tags, and pick a
> neutral-enough name, I don't see why the MELPA maintainers (who are
> quite reasonable people IME) wouldn't go for it.

Just a minor comment/question about this, which
I think would be the first time such a thing
would be happening:

Do we really want to set a precedent that
someone other than the code author fiddles with
their code, adding comments or whatever?

Sure, the maintainers of a repo are, in a way,
administrators.  But should such administrators
be changing source code?  Adding other code or
whatever, to administer, label, treat, etc. the
code is, at least conceptually, different from
changing the source code itself.

No, adding a field/tag in a comment is not a big
deal.  And yes, GPL code is open to modification.

Still, is this a good door to open?

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-19 18:48]:
> > MELPA to a large part consists of a database of "recipes", one for each
> > package that it builds and distributes. This tag can be put inside
> > recipes, and thus be controlled by MELPA maintainers, and not by the
> > packages authors themselves.
> > 
> > If we provide some well-defined criteria for such tags, and pick a
> > neutral-enough name, I don't see why the MELPA maintainers (who are
> > quite reasonable people IME) wouldn't go for it.
> 
> Just a minor comment/question about this, which
> I think would be the first time such a thing
> would be happening:
> 
> Do we really want to set a precedent that
> someone other than the code author fiddles with
> their code, adding comments or whatever?
> 
> Sure, the maintainers of a repo are, in a way,
> administrators.  But should such administrators
> be changing source code?  Adding other code or
> whatever, to administer, label, treat, etc. the
> code is, at least conceptually, different from
> changing the source code itself.
> 
> No, adding a field/tag in a comment is not a big
> deal.  And yes, GPL code is open to modification.
> 
> Still, is this a good door to open?

That is similar to how many GNU/Linux software packages are
maintained, often they are modified before such enter distribution for
final users.

I do not care if package is original, not original, forked or not
forked, modified, what I care is which group of people is making it
trusted and by which principles.

If nobody is making package verifications by looking into it, then
such package is not trusted to me, with or without PGP signature, it
does not matter any more.

That is why some GNU/Linux distributions have so many maintainers,
each is responsible for some packages, there is no warranty, but there
is some implied moral obligation at least. Some OS like OpenBSD have
better security policies, they verify the code for security, not just
package and wrap it for delivery.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Sure, the maintainers of a repo are, in a way,
  > > administrators.  But should such administrators
  > > be changing source code?  Adding other code or
  > > whatever, to administer, label, treat, etc. the
  > > code is, at least conceptually, different from
  > > changing the source code itself.

  > That is similar to how many GNU/Linux software packages are
  > maintained, often they are modified before such enter distribution for
  > final users.

  > I do not care if package is original, not original, forked or not
  > forked, modified, what I care is which group of people is making it
  > trusted and by which principles.

That is my view also.  Free software means you're allowed to
distribute a modified version.  There is nothing wrong with making
changes.  The changes are good, or bad, based on what they say.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le lun. 19 oct. 2020 à 17:48, Drew Adams <drew.adams@oracle.com> a écrit :
>
> > MELPA to a large part consists of a database of "recipes", one for each
> > package that it builds and distributes. This tag can be put inside
> > recipes, and thus be controlled by MELPA maintainers, and not by the
> > packages authors themselves.
> >
> > If we provide some well-defined criteria for such tags, and pick a
> > neutral-enough name, I don't see why the MELPA maintainers (who are
> > quite reasonable people IME) wouldn't go for it.
>
> Just a minor comment/question about this, which
> I think would be the first time such a thing
> would be happening:
>
> Do we really want to set a precedent that
> someone other than the code author fiddles with
> their code, adding comments or whatever?
>
> Sure, the maintainers of a repo are, in a way,
> administrators.  But should such administrators
> be changing source code?  Adding other code or
> whatever, to administer, label, treat, etc. the
> code is, at least conceptually, different from
> changing the source code itself.
>
> No, adding a field/tag in a comment is not a big
> deal.  And yes, GPL code is open to modification.
>
> Still, is this a good door to open?

I'm not sure to understand the question. The recipes are stored in the
Melpa repository, so the maintainers of the repo are really the
maintainers of the source code (the recipes).

The recipes are written by the authors of the package, by forking and
filing a request to merge into the Melpa repository.

So if the administrators of Melpa agree to it, they are entirely in
their prerogative to alter the recipes provided by the users in their
code (list of recipes). It's not really different from adding a
comment to clarify a bug fix by an external contributor after the code
is merged.

They would not be modifying the code of the packages themselves, of course.

RE: Proposal for an Emacs User Survey

[Drew Adams]

> They would not be modifying the code of the packages themselves, of course.

I see.  I thought that was the case.
I didn't realize the "tag" would be in the recipe,
not in the submitted source code.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If we provide some well-defined criteria for such tags, and pick a 
  > neutral-enough name, I don't see why the MELPA maintainers (who are 
  > quite reasonable people IME) wouldn't go for it.

  > Question is, would you agree to add MELPA to the default list of package 
  > archives if they do?

I do not have confidence that this will be sufficient.
Melpa has caused other problems (discussed here some months ago).


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 20.10.2020 08:14, Richard Stallman wrote:
>    > If we provide some well-defined criteria for such tags, and pick a
>    > neutral-enough name, I don't see why the MELPA maintainers (who are
>    > quite reasonable people IME) wouldn't go for it.
> 
>    > Question is, would you agree to add MELPA to the default list of package
>    > archives if they do?
> 
> I do not have confidence that this will be sufficient.
> Melpa has caused other problems (discussed here some months ago).

"Caused" or "has"?

If the issue of it being hosted on Github is a show-stopper then ok, 
we'll probably not have much progress here.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le ven. 16 oct. 2020 à 05:59, Richard Stallman <rms@gnu.org> a écrit :
> I hope that only a minority of Emacs users know about MELPA, and I'd
> rather not inform the rest about it.  But if something is going to
> inform them anyway, it is better to do it with a denunciation.

I believe that it is too late for such hopes, and that a majority of
Emacs users know about Melpa and/or uses it. Some people know about it
without using it (most of them probably read this list) and some
people use it without knowing it (users of pre-configured emacs'es).

For instance, those three links were on the first page of a duckduckgo
search for "install emacs packages":

https://www.emacswiki.org/emacs/InstallingPackages
http://ergoemacs.org/emacs/emacs_package_system.html
http://pragmaticemacs.com/emacs/install-packages/

None of them denunciate anything, only one of them even mentions that
Melpa is not official.

I believe that if the Emacs webpage and manual were to mention Melpa,
it would inform a lot of users about the potential risks to freedom of
Melpa, and only inform a few users of the existence of Melpa.

Re: Proposal for an Emacs User Survey

[Philip K.]

Thibaut Verron <thibaut.verron@gmail.com> writes:

> Le ven. 16 oct. 2020 à 05:59, Richard Stallman <rms@gnu.org> a écrit :
>> I hope that only a minority of Emacs users know about MELPA, and I'd
>> rather not inform the rest about it.  But if something is going to
>> inform them anyway, it is better to do it with a denunciation.
>
> I believe that it is too late for such hopes, and that a majority of
> Emacs users know about Melpa and/or uses it. Some people know about it
> without using it (most of them probably read this list) and some
> people use it without knowing it (users of pre-configured emacs'es).
>
> For instance, those three links were on the first page of a duckduckgo
> search for "install emacs packages":
>
> https://www.emacswiki.org/emacs/InstallingPackages
> http://ergoemacs.org/emacs/emacs_package_system.html
> http://pragmaticemacs.com/emacs/install-packages/

And that's only if the user already knows that they want to install
packages (instead of "plug-ins" or "extensions", as other software would
refer to the concept). From my experience, almost every non-official
beginners tutorial will either mention or just give you the code to
configure MELPA. And with all the starter-packages that offer to
pre-configure Emacs, you don't even think about it.

But that's currently unavoidable, a lot of the advocacy for Emacs is
based on promoting certain packages (Magit, Evil, Org-mode extensions
such as Roam, etc.) that are for the most part only available on MELPA.
Non-GNU Elpa should help to mitigate this problem, as my impression is
that most people intrigued in Emacs, would quickly loose interest, if it
weren't for these specific packages.

(In general I think that that kind promotion isn't good, at least in the
long term. Emacs is seen as getting in the way of whatever package you
need, and they therefore follow whatever method they find to avoid
learning. While possible, it leaves many on a fragile tower of
abstractions and hacks, that could break at any moment.)

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-16 10:06]:
> I believe that if the Emacs webpage and manual were to mention Melpa,
> it would inform a lot of users about the potential risks to freedom of
> Melpa, and only inform a few users of the existence of Melpa.

It is alright to file issue at MELPA, to see if they will remedy any
non-free issue.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I believe that it is too late for such hopes, and that a majority of
  > Emacs users know about Melpa and/or uses it.

You may be right, but it is hard to tell.  We don't know what fraction
of Emacs users are even aware that there is such a thing as
"packages".

The survey that is now being done will not tell us, because it will be
publicized in certain forums so it will reflect the subset of users
that use those forums -- not a representative sample of all users.  I
woudl expoect that there is another population of users which has no
contact with those forums or startup kits.

So you might be right, but we don't know if that is actually so.

It makes a difference.  If we KNEW that almost all users of Emacs did
know about Melpa, that would change our policy towards it.  But it
would be an own-goal to assume, erroneously that they already know and
as a result end up being the ones to tell them.

Can you change emacswiki to state the two things that are bad
about Melpa?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Warning you mention is used on Emacs download page, see
  > > https://www.gnu.org/software/emacs/download.html

  > Great, so there is precedent ! Why is it acceptable and sufficient on
  > the main download page but not acceptable in a survey?

I am not sure what those words mean.  I looked at that page to see if
I could figure it out, but I was stuck.  Would people like to explain
what part or aspect of that page is at issue here?
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le jeu. 15 oct. 2020 à 05:52, Richard Stallman <rms@gnu.org> a écrit :
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Warning you mention is used on Emacs download page, see
>   > > https://www.gnu.org/software/emacs/download.html
>
>   > Great, so there is precedent ! Why is it acceptable and sufficient on
>   > the main download page but not acceptable in a survey?
>
> I am not sure what those words mean.  I looked at that page to see if
> I could figure it out, but I was stuck.  Would people like to explain
> what part or aspect of that page is at issue here?

As Jean-Louis said, the warning he was referring too is the preamble
of the section "Non-free systems".

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 06:33]:
> And I would be worried about selection bias in a survey which voluntarily
> omits the most popular options from the lists of choices: how many people
> would choose not to complete the survey of they feel that the questions are
> biased?

That is easy to solve:

- [ ] Check here if you do not wish to complete this survey

- [ ] Check here if you do not want to tell us anything about you

etc.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le lun. 12 oct. 2020 à 07:37, Jean Louis <bugs@gnu.support> a écrit :
>
> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-12 06:33]:
> > And I would be worried about selection bias in a survey which voluntarily
> > omits the most popular options from the lists of choices: how many people
> > would choose not to complete the survey of they feel that the questions are
> > biased?
>
> That is easy to solve:
>
> - [ ] Check here if you do not wish to complete this survey
>
> - [ ] Check here if you do not want to tell us anything about you
>
> etc.

Can't tell if you're serious. But just in case, no, they would just
close the page/buffer/whatever and not send the results.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Not mentioning MELPA as an option because you feel uncomfortable about
  > it will distort survey results, especially for the part where you ask
  > what package archives people use.

That is ok.  This is not a research project.  The reason we are
interested in a survey's results is to make practical decisions about
policies that we could possibly change.

We do not mention places to get software if they lead people to
nonfree software.  See the GNU Coding Standards, node References.  Our
stance about MELPA is determined directly by this principle.  Unless
MELPA's policies change, our stance about it cannot change.  We can
do no other.

  >   Remember, this is a
  > survey to find out how Emacs users use their editor, not about how much
  > they adhere to FSF ideology.

Nobody would disagree with that point, but that abstract question is
not where the disagreement is.

We need to inform people that there is a grave moral flaw in MELPA,
and we can't do that if we ask questions about it just the same as
we do about other repositories.

What we might do is ask

  Which of these exclusively libre Lisp package archives do you use

with a list of those, then

  Do you use any Lisp package archives not in that list?

and people could answer MELPA if they wish.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  ...
  > >   > What is free software (libre software)?
  > > 
  > >   > Which of these are free liceenses?
  > > 
  > >   > How does GNU relate to free software?

  > See my last sentence above.  What's the point of this quiz?  Finding out
  > what percentage of Emacs users fully believes in GNU's goals?  What will
  > that information be used for?

We can use it to try to improve our efforts to communicate those
crucial points about our overall goals and what we have achieved.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Adrien Brochard]

Thank you for your feedback!

I've read many emails on this thread now and I think we've reached a
point where some action needs to be taken. I have purchased
emacssurvey.org and I will proceed to:
- digest all the feedback I got into a final list of questions
- announce the survey opening date on as many channels as I can
- build emacssurvey.org as a no-JS site such that everyone can visit it
- on the survey start date, emacssurvey.org will have two options for
submitting responses
   - visit the questionnaire as plain text and email it to a dedicated
address hosted on emacssurvey.org (which will be possible from Emacs)
   - use an online form which will potentially require non-free JS
- the survey will remain open for 6 weeks, plus some extra time for late
responders
- I will post on as many channels every week to remind people who
haven't responded yet (but obviously not share any partial results)
- after the survey is over and all data is compiled, all results will be
released on emacssurvey.org (and again it will be viewable from Emacs)

By hosting the survey on a different website, I am hoping to accomplish
something similar to the "Devil as Install Fest"
(https://www.gnu.org/philosophy/install-fest-devil.en.html).

What we could still discuss is a disclaimer on emacssurvey.org to be
explicit on how gnu.org is not sponsoring the survey questions. It would
be really helpful if gnu.org could link to emacssurvey.org, but if not,
that's ok too.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Adrien Brochard <abrochard@gmx.com> [2020-10-12 18:17]:
> Thank you for your feedback!
> 
> I've read many emails on this thread now and I think we've reached a
> point where some action needs to be taken. I have purchased
> emacssurvey.org and I will proceed to:

Well that is what I said, you can do it independent of GNU project.

> - digest all the feedback I got into a final list of questions

I hope you will use "check all that apply" method, and not have "one
option" among the choice of lists.

> - announce the survey opening date on as many channels as I can
> - build emacssurvey.org as a no-JS site such that everyone can visit
> - it

That is right, not hard to do, I have done very complex forms, there
are many ways how to do complex forms, I was using CGI and Perl, but
simple HTML could be just enough.

Always add: "Other" field, which is enough wide and welcoming, to help
people write what they wish else.

When doing professional survey you could not change the parameters
during the survey period, so changing questions would not be good. I
suggest keeping versions, you may do more versions in future.

I hope you will align the purpose of the survey with Emacs
development, so that development may find out what is number one
feature needed and wanted in Emacs, to improve Emacs that way.

> - on the survey start date, emacssurvey.org will have two options for
> submitting responses
>   - visit the questionnaire as plain text and email it to a dedicated
> address hosted on emacssurvey.org (which will be possible from
> Emacs)

That is great.

>   - use an online form which will potentially require non-free JS

It does not require non-free JS or any Javascript. It requires plain
HTML, easy to do, it is even possible to make the Org file, and
process by Emacs Lisp that in turn makes the full survey online.

If your survey page is more complex, and the more complex you make it,
the less results you will get, but if it is multi-page form, then
there are nice Perl modules handling the job. Just see
https://cpan.org and search for forms.

But using any other programming language to generate the forms and
capture its information in the database other than Emacs Lisp, would
be disgrace for me, so just do it in Emacs Lisp, it will perform well.

> - the survey will remain open for 6 weeks, plus some extra time for late
> responders

For 6 weeks survey, you need to put up some money to capture the
users, and if you advertise, you should disclose on which
communication channels you advertised, and evaluation should involve
that advertising.

Your advertising would need to run for exact time of the survey, like
6 weeks, and then the survey results are specific only for that
specific communication channels you used.

Let us say you advertise on Reddit, you would get quite different
results then if you advertise on other network, attracting for example
Github users, or if you advertise so that you attract Debian users, or
scientists, students in universities and similar.

> - I will post on as many channels every week to remind people who
> haven't responded yet (but obviously not share any partial results)

If you do a survey for which you still do various advertising during
the survey time, that cannot be just survey, you would need to reach
users quickly to answer quickly their questions. You could advertise
on some networks using keywords like "emacs" and reach 1000 users
within 24 hours, rather then posting links online for 6 weeks, as in
that way you are changing the dynamic of the survey and also
influencing it consciously or not consciously.

How you have advertised it, and in what time, you would also need to
take in evaluation, along with your personal experience or skills.


Jean

RE: Proposal for an Emacs User Survey

[Drew Adams]

> I've read many emails on this thread now and I think we've reached a
> point where some action needs to be taken.

Why do you think so?

> I have purchased emacssurvey.org and I will proceed to:...

FWIW, I'm not in favor of proceeding this way.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > - digest all the feedback I got into a final list of questions

I am surprised and disappointed that you want to do this on your own,
without the participation of the GNU Project.

Nonetheless, would you please let me review and comment on your text
before you publish it?

  > - after the survey is over and all data is compiled, all results will be
  > released on emacssurvey.org (and again it will be viewable from Emacs)

Please make the results available in a simple textual format as a
single file, as well as any other formats you like.

     > - use an online form which will potentially require non-free JS

Please do not have any nonfree JS code for this site.
That would work directly against the free software movement.
Asking people to run nonfree software is an injustice.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Bonface M. K.]

[-- Attachment #1: Type: text/plain, Size: 6164 bytes --]


Hi all! Just some minor feedback.

Adrien Brochard <abrochard@gmx.com> writes:

[...]
>
> I also agree with this. I think no matter what we are going to get
> selection bias and we should not rush any decision based on the survey
> results.
>
>
> * Survey Questions
> (Please note that any "other" should allow for free text entry for the
> respondent to elaborate)
>
> ** How would you characterize your use of Emacs?
>    - Use it for work
>    - I use it for serious "hobby" projects
>    - I'm just tinkering
>    - I use it for my studies
>    - Other
>

You could use Emacs for "everything". I'd propose
you make this question open-ended?

> ** What do you use Emacs for?
>     - Software development
>     - Research writing
>     - Data science
>     - Writing
>     - Other
>

Same as above. You could use Emacs as a
combination of the above.

> ** How long have you been using Emacs?
>
> ** Which version of Emacs do you use?
>
> ** What OS do you primary use?
>     - Linux
>     - Windows
>     - MacOS
>     - BSD
>     - Other
>
> ** How do you use Emacs?
>     - GUI
>     - Terminal (TUI)
>     - Both
>
> ** How do you run Emacs?
>     - Client/Server mode only
>     - Standalone application only
>     - Both
>
> ** Describe your configuration
>    - I am using vanilla emacs (little to no configuration).
>    - Custom configuration
>    - Spacemacs - https://github.com/syl20bnr/spacemacs
>    - Doom Emacs - https://github.com/hlissner/doom-emacs
>    - Prelude - https://github.com/bbatsov/prelude
>    - purcell emacs.d - https://github.com/purcell/emacs.d
>    - magnars emacs.d - https://github.com/magnars/.emacs.d
>    - Emacs Starter Kit - https://github.com/eschulte/emacs-starter-kit
>    - oh-my-emacs - https://github.com/xiaohanyu/oh-my-emacs
>    - Better Defaults - https://github.com/technomancy/better-defaults
>    - Graphene - https://github.com/rdallasgray/graphene
>    - ohai-emacs - https://github.com/bodil/ohai-emacs
>    - ergoemacs-mode - https://github.com/ergoemacs/ergoemacs-mode
>    - Rho Emacs - https://github.com/GChristensen/rho-emacs
>    - Radian - https://github.com/raxod502/radian
>    - Centaur Emacs - https://github.com/seagle0128/.emacs.d
>    - Other
>
> ** What keybindings do you use?
>    -  Emacs defaults
>    -  Evil/Spacemacs/Doom-Emacs (all the vim-likes)
>    -  CUA-mode
>    -  God-mode
>    -  Boon
>    -  Xah-Fly-Keys
>    -  Custom modal (ryo-modal, etc)
>    -  Custom modifiers (Emacs from scratch)
>    -  Other
>
> ** Prior to using Emacs what was your primary editor?
>    - VIM
>    - VScode
>    - Eclipse
>    - Notepad++
>    - Sublime
>    - Intelij
>    - Other
>
> ** org-mode usage
>     - I use emacs only for org-mode
>     - dayly
>     - from time to time
>     - Not a org-mode user
>
> ** Which completion/selection framework do you use?
>    - helm
>    - ivy
>    - ido
>    - icomplete
>    - other
>    - i don't use a completion framework
>
> ** How do you manage third-party elisp?
>     - built-in package.el
>     - Spacemacs does it for me
>     - straight.el
>     - borg
>     - leaf
>     - el-get
>     - Nix
>     - git submodules without Borg
>     - git subtrees
>     - git clones
>     - guix
>     - quelpa
>     - cask
>     - No third-party deps
>     - other
>
> ** How do you get emacs packages(if applicable)?
>     - No repos
>     - Gnu Elpa
>     - Melpa/Melpa Stable
>     - Directly from the source (e. g. using straight).
>
> ** Can you list some of your favorite package?
>
> ** What package do you use for error checking?
>    - Flymake
>    - Flycheck
>    - None
>
> ** Do you use TRAMP?
>
> ** DO you use Magit?
>
> ** What package do you use for project management?
>     - project.el
>     - projectile
>     - Other(mention)
>     - None
>
> ** Do you use shell/terminal emulator in Emacs?
>     - eshell
>     - shell
>     - term
>     - ansi-term
>     - do not use.
>
> ** Do you use mail client in Emacs?
>     - Mu4e
>     - Gnus
>     - Mut
>     - notmuch
>     - do not use
>
> ** What is your Elisp proficiency?
>     - Beginner/No knowledge
>     - Basic elisp understanding
>     - Intermediate
>     - Advanced
>     - Expert
>
> ** If you use Emacs for programming, which languages do you program in?
>
> ** do you use a language server with lsp-mode? With what languages?
>
> ** Do you use Emacs for debugging? What do you use? (Gdb, dap-mode etc)
>
> ** Which theme do you use?
>
> ** Have you ever contributed to GNU Emacs core/Elpa?
>     - No
>     - No, but I would do that if the process is changed(e. g. using
> github pull requests instead of the mailing list, no papers, etc**.
>     - I do PRs from time to time
>     - I provide PRs regularly
>     - I am active contributor/maintainer
>
> ** Have you ever contributed to Melpa package?
>     - No
>     - I do PRs from time to time
>     - I provide PRs regularly
>     - I am a package maintainer
>
> ** What Emacs community forums have you visited in the past year?
> Answers would be things like
>     - r/emacs
>     - Emacs mailing list
>     - irc
>     - Emacs meetups
>     - I follow twitter Emacs related accounts
>     - Other
>
> ** What are some of the Emacs improvements you are the most interested in?
> ** what do you think are Emacs' greatest strengths?
> ** can you recall any difficulties you faced initially learning Emacs?
> ** What is the one thing you would like emacs to do differently?
> ** How did you hear about this survey?
> ** If there is another survey in 2021, would you be opposed to it
> containing optional & general demographics questions?
>    It could include age backets, gender, country or language
> ** Do you have a preferred platform for filling out the survey in the
> future?
> ** General feedback about the survey process?
>
>
>

-- 
Bonface M. K. (https://www.bonfacemunyoki.com)
Chief Emacs Mchochezi / Twitter: @BonfaceKilz
GPG key = D4F09EB110177E03C28E2FE1F5BBAE1E0392253F

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 869 bytes --]

Re: Proposal for an Emacs User Survey

[Philip K.]

Adrien Brochard <abrochard@gmx.com> writes:

> ** How do you use Emacs?
>     - GUI
>     - Terminal (TUI)
>     - Both

I'd suggest also asking how they use the GUI. Do they keep everything as
it is, or hide the scroll-, tool- and menu bar (or just parts).

Perhaps, if it turns out that most people disable some feature, it could
be turned off by default in the next release? If you ask me, just
removing the tool-bar would improve the default look more than anything
else.

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Philip K.]

I can't seem to find anyone here mentioning it, but it seems the survey
has been opened today:

    https://emacssurvey.org/

Sadly, it seems like they decided to use non-free JS for the web-form
variant :(

Adrien Brochard <abrochard@gmx.com> writes:

> Hi everyone,
>
> I posted on reddit.com/r/emacs yesterday a proposal to craft and
> distribute a survey for Emacs users to better grasp the diversity and
> various usages out there. I got some very good feedback but there could
> always be more.
>
> The main points that need to be worked out are:
> 1. the actual questions to be asked
> 2. the platform to collect responses
>
> Regarding the questions, I have a decent skeleton and a lot of suggestions.
>
> But the for the platform to collect responses, this is where it gets
> difficult. Something like a Google survey might the easiest/cheapest to
> make and distribute and will work on most devices, but it obviously has
> privacy concerns. On the other hand, doing something like self-hosting a
> no-JS form is very time consuming. I was also thinking of allowing
> responses via email for people who do not want to bother with a survey
> link and then manually reconcile responses.
>
> Links to the posts with the tentative questions:
> -
> https://www.reddit.com/r/emacs/comments/j6x7ad/proposal_for_an_emacs_user_survey/
> -
> https://www.reddit.com/r/emacs/comments/b2fh4y/help_reviewprovide_feedback_on_state_of_emacs/
>
> Thank you for reading!
> Best,
> Adrien
>
>

-- 
	Philip K.

Re: Proposal for an Emacs User Survey

[Dmitry Gutov]

On 19.10.2020 19:20, Philip K. wrote:
> I can't seem to find anyone here mentioning it, but it seems the survey
> has been opened today:
> 
>      https://emacssurvey.org/
> 
> Sadly, it seems like they decided to use non-free JS for the web-form
> variant:(

That's unfortunate.

Still, they included added the free-form option.

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >     https://emacssurvey.org/

  > Sadly, it seems like they decided to use non-free JS for the web-form
  > variant :(

This means that the results will be biased in two ways:

* In favor of people who use the forums he posted it in.

* Against people who won't run nonfree software.

The results may be of use anyway, but they will not describe a
represent sample of all Emacs users.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

> This means that the results will be biased in two ways:
>
> * In favor of people who use the forums he posted it in.

It has been (and can be) posted in many places, including presently emacs-devel.

> * Against people who won't run nonfree software.

The survey can be completed without running nonfree software, by
sending an e-mail.

> The results may be of use anyway, but they will not describe a
> represent sample of all Emacs users.

I agree, but that's because of difficulties inherent to polling, not
because of a bias that could have been easily avoided.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-20 11:23]:
> I agree, but that's because of difficulties inherent to polling, not
> because of a bias that could have been easily avoided.

It is not difficult, it is one simple stupid HTML, make it nice with
CSS, form can be accepted and saved in file for later processing.

Further there is free software for that, there are systems like
Wordpres, Drupal and other website revision software.

There is something like LimeSurvey for complexities:
https://community.limesurvey.org/downloads/

You can survey in pure bash shell:
https://blog.eduonix.com/shell-scripting/learn-cgi-scripting-using-bash-in-linux-shell-scripting/

I would do it in nothing else but Emacs to write the form, and to
accept the form over CGI. It would then be stored as encrypted Emacs
Lisp data in file, and later could be evaluated.

Re: Proposal for an Emacs User Survey

[Thibaut Verron]

Le mar. 20 oct. 2020 à 11:48, Jean Louis <bugs@gnu.support> a écrit :
>
> * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-20 11:23]:
> > I agree, but that's because of difficulties inherent to polling, not
> > because of a bias that could have been easily avoided.
>
> It is not difficult, it is one simple stupid HTML, make it nice with
> CSS, form can be accepted and saved in file for later processing.

Yes, I thought I was clearly stating that the difficulties I was
referring to were not related to the form of the survey, where in my
opinion they did choose a reasonable path.

The difficulties are inherent to polling and concern reaching a
sufficiently large sample, weighing it for representativity, etc.

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Thibaut Verron <thibaut.verron@gmail.com> [2020-10-20 14:04]:
> Le mar. 20 oct. 2020 ?? 11:48, Jean Louis <bugs@gnu.support> a ??crit :
> >
> > * Thibaut Verron <thibaut.verron@gmail.com> [2020-10-20 11:23]:
> > > I agree, but that's because of difficulties inherent to polling, not
> > > because of a bias that could have been easily avoided.
> >
> > It is not difficult, it is one simple stupid HTML, make it nice with
> > CSS, form can be accepted and saved in file for later processing.
> 
> Yes, I thought I was clearly stating that the difficulties I was
> referring to were not related to the form of the survey, where in my
> opinion they did choose a reasonable path.

20 years ago we were handling forms with Perl's form.cgi

For me is surprising that Emacs survey has to be done with third party
providers. If anybody would ask me, I would make it in simple HTML,
what I already proposed, and I could install the background CGI to
accept the form.

> The difficulties are inherent to polling and concern reaching a
> sufficiently large sample, weighing it for representativity, etc.

Both ways how the surve have been presented with the proprietary
Javascript and with the download of the Org file, are not user
friendly and cannot reach large sample. One also needs marketing
skills.

Re: Proposal for an Emacs User Survey

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

* 2020-10-20 14:38:07+03, Jean Louis wrote:

> For me is surprising that Emacs survey has to be done with third party
> providers. If anybody would ask me, I would make it in simple HTML,
> what I already proposed, and I could install the background CGI to
> accept the form.

> Both ways how the surve have been presented with the proprietary
> Javascript and with the download of the Org file, are not user
> friendly and cannot reach large sample. One also needs marketing
> skills.

I think we should appreciate the fact that the original poster (Adrien
Brochard) actually did something. He is really conducting a survey. He
is not just talking and designing a theoretically and ethically perfect
survey, as the rest of us.

https://emacssurvey.org/

-- 
/// Teemu Likonen - .-.. https://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Teemu Likonen <tlikonen@iki.fi> [2020-10-26 20:44]:
> * 2020-10-20 14:38:07+03, Jean Louis wrote:
> 
> > For me is surprising that Emacs survey has to be done with third party
> > providers. If anybody would ask me, I would make it in simple HTML,
> > what I already proposed, and I could install the background CGI to
> > accept the form.
> 
> > Both ways how the surve have been presented with the proprietary
> > Javascript and with the download of the Org file, are not user
> > friendly and cannot reach large sample. One also needs marketing
> > skills.
> 
> I think we should appreciate the fact that the original poster (Adrien
> Brochard) actually did something. He is really conducting a survey. He
> is not just talking and designing a theoretically and ethically perfect
> survey, as the rest of us.
> 
> https://emacssurvey.org/

http://www.gnu.org/fun/jokes/users-lightbulb.html

-- 
Jean Louis

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Teemu Likonen <tlikonen@iki.fi> [2020-10-26 20:44]:
> I think we should appreciate the fact that the original poster (Adrien
> Brochard) actually did something. He is really conducting a survey. He
> is not just talking and designing a theoretically and ethically perfect
> survey, as the rest of us.
> 
> https://emacssurvey.org/

The bug database
https://debbugs.gnu.org/cgi/pkgreport.cgi?which=pkg&data=emacs is like
a pinpointed poll on what people would like to have improved.

While the survey is interesting, it went into public with little good
preparation now driving users to . Impression is that there was impatience and rush to make
the survey or poll.

Comparing the Emacs bug system to the survey I see that bug system
actually works very well including this mailing list. The type of
discussions may seem talky though in background it is very productive
and Emacs gets speedily improved.

When doing such outreach it would be useful to reference to the bug
system and help-gnu-emacs@gnu.org mailing list where every user may
ask for general help on Emacs.

> He is really conducting a survey. He is not just talking and
> designing a theoretically and ethically perfect survey, as the rest
> of us.

The survey is not well planned and implemented technically. That is
beyond theoretical or ethical subjects. Survey gives two options for
user to fill it online and to fill the Org file, I did the last and
signed it with my GPG key.

Example is here:

- HOW WOULD YOU CHARACTERIZE YOUR USE OF EMACS?

- [X] Use it for work
- [X] I use it for serious "hobby" projects
- [X] I'm just tinkering
- [X] I use it for my studies
- [X] Other (please specify)
  - for browsing Internet
  - for database management
  - CRM or Customer Relationship Management
  - WRS or Website Revision System
  - for management of mailing lists and marketing

The online form offers only one option among those above options. And
so on for each section of the survey, there is option list instead of
a checklist.

To make a survey form like that is maybe 20 minutes to 60 minutes of
work and it would cost maybe $5 to $20 to hire somebody to do it, many
would do it for free and it would be technically readable and
accessible page without any non-free Javascript or third parties being
involved.

-- 
Jean Louis

Re: Proposal for an Emacs User Survey

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think we should appreciate the fact that the original poster (Adrien
  > Brochard) actually did something.

He did something, in a way that has good effects and bad ones.
He could have cooperated and done it in a way that
would have had the same good effects and not the bad effects,
but that's not what he chose.

My appreciation will therefore be limited.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Proposal for an Emacs User Survey

[Ivan Yonchovski]

Jean Louis writes:

> Both ways how the surve have been presented with the proprietary
> Javascript and with the download of the Org file, are not user
> friendly and cannot reach large sample. One also needs marketing
> skills.

What number of responders will be considered large enough sample?

Thanks,
Ivan

Re: Proposal for an Emacs User Survey

[Jean Louis]

* Ivan Yonchovski <yyoncho@gmail.com> [2020-10-26 21:14]:
> 
> Jean Louis writes:
> 
> > Both ways how the surve have been presented with the proprietary
> > Javascript and with the download of the Org file, are not user
> > friendly and cannot reach large sample. One also needs marketing
> > skills.
> 
> What number of responders will be considered large enough sample?

That is not easy to determine. I would say that 1000 people would be
minimum in each case and it would never be enough, as that comes from
personal experience from work in two organizations, one was
specifically a survey company and other was conducting always surveys
for its own humanitarian purposes.

From:
https://www.cloudresearch.com/resources/guides/statistical-significance/determine-sample-size/

As you can see, even when a population is large, researchers can often
understand the entire group with about 1,000 respondents.

  Population Size  Sample Size Based on ±3%  Sample Size Based on ±5%  Sample Size Based on ±10%    
    Margin of Error  Margin of Error  Margin of Error    
  500  345  220  80    
  1,000  525  285  90    
  3,000  810  350  100    
  5,000  910  370  100    
  10,000  1,000  385  100    
  100,00+  1,100  400  100    
----- end of quote --------

Every survey should have its purpose, usually to improve the product
or service, in this case Emacs. Improvements are constantly made and
public opinion is looked upon.

To discuss issues with people is human, friendly and vibrant activity
and is way of taking opinions (bug reports, emails, opinions, package
contributions) and putting them in reality (patches, improvements,
ELPA...).

To ask people questions online without opportunity for feedback is
one-way, dead end type of a communication.

There are few questions that look like there is question about
accessibility yet we will see later what comes out of it. If website
is not made well accessible those questions may not be productive.

For solving usability issues, instead of 1000 people one could as well
follow feedback from 5 users, reference:
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

Article refers to web, it may be applied on Emacs user interface.

Quote:

Usability is a quality attribute that assesses how easy user
interfaces are to use. The word "usability" also refers to methods for
improving ease-of-use during the design process.

Usability is defined by 5 quality components:

- Learnability: How easy is it for users to accomplish basic tasks the
  first time they encounter the design?

- Efficiency: Once users have learned the design, how quickly can they
  perform tasks?
  
- Memorability: When users return to the design after a period of not
  using it, how easily can they reestablish proficiency?
  
- Errors: How many errors do users make, how severe are these errors,
  and how easily can they recover from the errors?
  
- Satisfaction: How pleasant is it to use the design?


-- 
Jean Louis