bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>
To: bug-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
Subject: 
--text follows this line--

In 9973019764250ac1f4d77a6b426cdd9c241151c5

* lisp/emacs-lisp/package.el: Load package-quickstart without package.el
    
the package-user-dir defcustom gets the custom-initialize-delay
intializer.  At least for me, this overrides any setting of
package-user-dir that is done in early-init.el.  As far as I can tell,
the documentation still says it should be possible and even recommended
to set package-user-dir in early-init.el.

/Ola


In GNU Emacs 28.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.5, cairo version 1.16.0)
 of 2021-01-14 built on lap5cg80647hz
Repository revision: ebab8898cad35b07c703c62d62dcd2aebd51d637
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SOUND
THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: sv_SE.UTF-8
  value of $LC_NUMERIC: sv_SE.UTF-8
  value of $LC_TIME: sv_SE.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Debugger

Minor modes in effect:
  hes-mode: t
  desktop-save-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  which-function-mode: t
  show-paren-mode: t
  display-time-mode: t
  global-auto-revert-mode: t
  override-global-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
/home/olani/emacs-packages/cov/cov hides /home/olani/.emacs.d/elpa-lap5cg80647hz-28.0.50/cov-20201214.550/cov
/home/olani/.emacs.d/elpa-lap5cg80647hz-28.0.50/dpkg-dev-el-20190824.2314/debian-autoloads hides /home/olani/.emacs.d/elpa-lap5cg80647hz-28.0.50/debian-el-20201011.1543/debian-autoloads
/home/olani/.emacs.d/sed-mode hides /home/olani/.emacs.d/elpa-lap5cg80647hz-28.0.50/sed-mode-1.0/sed-mode

Features:
(shadow sort mail-extr emacsbug sendmail coan conf-bitbake-mode checkdoc
coan-elisp skeleton delete-trailing-whitespace-mode init
highlight-escape-sequences org-jira org-jira-sdk jiralib soap-client
rng-xsd xsd-regexp org-clock woman man dired-rainbow dired-hacks-utils
autoinsert mmm-sample mmm-auto mmm-vars mmm-utils mmm-compat t32-auto
coan-list midnight desktop frameset python tramp-sh flycheck-flawfinder
flycheck-cstyle flycheck git-commit with-editor transient async-bytecomp
async server magit-git magit-section magit-utils crm log-edit message
dired dired-loaddefs rfc822 mml mml-sec epa epg epg-config gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev gmm-utils
mailheader pcvs-util add-log matrix-client matrix-client-room-list
derived matrix-client-frame matrix-client-images matrix-notifications
notifications dbus matrix-client-room esxml-query ordered-buffer
matrix-client-rainbow ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar ox-html table ox-ascii ox-publish ox org-element org ob
ob-tangle ob-ref ob-lob ob-table org-macro org-footnote org-src
ob-comint org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp org-table org-keys org-loaddefs avl-tree
generator ol ob-exp ob-core org-compat ob-eval org-macs shr kinsoku svg
dom matrix-client-faces matrix-api-r0.3.0 let-alist matrix-helpers
matrix-macros url-http mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm rmc puny dns cal-menu calendar
cal-loaddefs frame-purpose rainbow-identifiers color ht esxml pcase xml
cl tracking shorten ov a request mail-utils f s dash-functional dash
anaphora gitignore-mode conf-mode flymake-git-whitespace flymake-proc
flymake compile text-property-search thingatpt vc-git diff-mode advice
coan-28.0.50-autoloads autoload lisp-mnt whitespace which-func imenu
cl-extra warnings paren time avoid autorevert filenotify jka-compr
use-package use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
coan-system docker-tramp tramp-cache tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat shell pcomplete comint ansi-color
ring parse-time iso8601 time-date ls-lisp format-spec help-fns
radix-tree cl-print dpkg-dev-el debian-el rx url-auth edmacro kmacro
info package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv early-init debug backtrace help-mode easymenu
find-func cl-loaddefs cl-lib iso-transl tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 512895 22186)
 (symbols 48 36265 1)
 (strings 32 168891 4833)
 (string-bytes 1 5056307)
 (vectors 16 53977)
 (vector-slots 8 638348 28943)
 (floats 8 272 120)
 (intervals 56 346 0)
 (buffers 984 17))

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> the package-user-dir defcustom gets the custom-initialize-delay
> intializer.  At least for me, this overrides any setting of
> package-user-dir that is done in early-init.el.

Oh, I see you're right, and I think this issue is wider than
`package-user-dir` since it affects all the other vars that use
`custom-initialize-delay`.


        Stefan

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

forcemerge 45857 30994
thanks

>> the package-user-dir defcustom gets the custom-initialize-delay
>> intializer.  At least for me, this overrides any setting of
>> package-user-dir that is done in early-init.el.
> Oh, I see you're right, and I think this issue is wider than
> `package-user-dir` since it affects all the other vars that use
> `custom-initialize-delay`.

And I see that this is actually a known problem: bug#30994


        Stefan

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

>> the package-user-dir defcustom gets the custom-initialize-delay
>> intializer.  At least for me, this overrides any setting of
>> package-user-dir that is done in early-init.el.
> Oh, I see you're right, and I think this issue is wider than
> `package-user-dir` since it affects all the other vars that use
> `custom-initialize-delay`.

Back in June 2018, Eli wrote:
> I can see several ways out of this:
> 
>   . Move the call to custom-reevaluate-setting back where it was, and
>     instead call custom-reevaluate-setting in that earlier place only
>     for Info-default-directory-list, which was the original reason why
>     the call was moved;
>   . Add a call to custom-reevaluate-setting for blink-cursor-mode
>     where the window-system is already set up;
>   . Call custom-reevaluate-setting for all the delayed-init variables
>     twice: once where we do that now, and again where we did that
>     before the above commit;
>   . Introduce yet another variable similar to
>     custom-delayed-init-variables, but separate from it, and put on
>     it all the delayed-init variables that need the window-system to
>     be set up, leaving all the rest in custom-delayed-init-variables.

We ended up going with the 3rd option, but this is the source of the
problem here.  I think we should change to option 2 for now and plan on
fixing it for good by removing this exception, i.e. by making
the default value of `blink-cursor-mode` be t.

More concretely, I suggest the patch below,


        Stefan


diff --git a/lisp/startup.el b/lisp/startup.el
index 57fd87f20f..8c9ac4e30f 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1311,12 +1312,13 @@ command-line
     (startup--setup-quote-display)
     (setq internal--text-quoting-flag t))
 
-  ;; Re-evaluate again the predefined variables whose initial value
-  ;; depends on the runtime context, in case some of them depend on
-  ;; the window-system features.  Example: blink-cursor-mode.
-  (let (current-load-list) ; c-r-s may call defvar, and hence LOADHIST_ATTACH
-    (mapc 'custom-reevaluate-setting custom-delayed-init-variables)
-    (setq custom-delayed-init-variables nil))
+  ;; Some vars's initialization need to be delayed yet further, until
+  ;; after we setup the real initial frame, because their initial
+  ;; value depends on the window-system features.
+  ;; Currently, the only such var is `blink-cursor-mode'.  See bug#30994.
+  ;; FIXME: We should arrange for `blink-cursor-mode's default to be
+  ;; unconditionally `t' so as to finally eliminate this problem.
+  (custom-reevaluate-setting 'blink-cursor-mode)
 
   (normal-erase-is-backspace-setup-frame)
 

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 14 Jan 2021 13:20:53 -0500
> Cc: 45857@debbugs.gnu.org, 30994@debbugs.gnu.org
> 
> >> the package-user-dir defcustom gets the custom-initialize-delay
> >> intializer.  At least for me, this overrides any setting of
> >> package-user-dir that is done in early-init.el.
> > Oh, I see you're right, and I think this issue is wider than
> > `package-user-dir` since it affects all the other vars that use
> > `custom-initialize-delay`.
> 
> Back in June 2018, Eli wrote:
> > I can see several ways out of this:
> > 
> >   . Move the call to custom-reevaluate-setting back where it was, and
> >     instead call custom-reevaluate-setting in that earlier place only
> >     for Info-default-directory-list, which was the original reason why
> >     the call was moved;
> >   . Add a call to custom-reevaluate-setting for blink-cursor-mode
> >     where the window-system is already set up;
> >   . Call custom-reevaluate-setting for all the delayed-init variables
> >     twice: once where we do that now, and again where we did that
> >     before the above commit;
> >   . Introduce yet another variable similar to
> >     custom-delayed-init-variables, but separate from it, and put on
> >     it all the delayed-init variables that need the window-system to
> >     be set up, leaving all the rest in custom-delayed-init-variables.
> 
> We ended up going with the 3rd option, but this is the source of the
> problem here.  I think we should change to option 2 for now and plan on
> fixing it for good by removing this exception, i.e. by making
> the default value of `blink-cursor-mode` be t.
> 
> More concretely, I suggest the patch below,

Is there any place where all these complications are described, so
that people (and we ourselves) could have a complete, comprehensive
overview of this stuff and the various problems it needs to solve?  If
not, how about taking this opportunity to describe that?  Because
otherwise these incidents will keep happening.

FWIW, I'm not sure I have a good understand how is this old issue
related to the new one.

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> Is there any place where all these complications are described, so
> that people (and we ourselves) could have a complete, comprehensive
> overview of this stuff and the various problems it needs to solve?  If
> not, how about taking this opportunity to describe that?  Because
> otherwise these incidents will keep happening.

Here's the story:

We have a few variables which are predefined in the dumped Emacs but
whose default value depends on information only available when Emacs is
executed, such as the user's $HOME.

We handle those using `custom-initialize-delayed`, so when the
variable's `defcustom` is procesed the variable isn't initialized yet,
and instead initialization happens in `startup.el` by calling
`custom-reevaluate-setting` on each one of those delayed variables.

In Emacs-26, we did that just before loading the `.emacs` file and
everybody was happy.  But the `early-init.el` file added to Emacs-27
introduced a difficulty there:

- Those vars's values might be needed in `early-init.el` so we should
  call `custom-reevaluate-setting` before we load `early-init.el`.

- The value of `blink-cursor-mode` depends on whether window-system is
  nil or not, which is only known after loading `early-init.el`, so we
  should call `custom-reevaluate-setting` after we load `early-init.el`.

Currently we solve this by calling `custom-reevaluate-setting` twice for
every one of those variables.  But it means that we silently override
any change made to those variables from `early-init.el`.

As noted, the only var whose default value needs to be (re)computed
after loading `early-init.el` is `blink-cursor-mode`.  In theory there
could be more, but I think we should aim to reduce rather than increase
those cases.  E.g. the current way `blink-cursor-mode` is defined means
that if I start `emacs -Q -nw` and later create a GUI frame, that frame
won't have a blinking cursor.  I think this is a bug (tho I can see
that some people may consider it a feature if they (like me) dislike
blinking cursors ;-0)


        Stefan

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ola.x.nilsson@axis.com,  45857@debbugs.gnu.org,  30994@debbugs.gnu.org
> Date: Thu, 14 Jan 2021 14:32:13 -0500
> 
> > Is there any place where all these complications are described, so
> > that people (and we ourselves) could have a complete, comprehensive
> > overview of this stuff and the various problems it needs to solve?  If
> > not, how about taking this opportunity to describe that?  Because
> > otherwise these incidents will keep happening.
> 
> Here's the story:

Thanks, I actually meant to have that in the sources in some prominent
place.

> Currently we solve this by calling `custom-reevaluate-setting` twice for
> every one of those variables.  But it means that we silently override
> any change made to those variables from `early-init.el`.
> 
> As noted, the only var whose default value needs to be (re)computed
> after loading `early-init.el` is `blink-cursor-mode`.  In theory there
> could be more, but I think we should aim to reduce rather than increase
> those cases.

I think relying on a small number of such variables is not
future-proof enough.  This case is a living proof: we decided
something 2 years ago, but changes we did since then require us now to
change that decision, which means we risk bumping into issues which we
wanted to avoid back then.  That's a general problem with kludgey
solutions.

I think we need some new infrastructure that would support the two
init files that are loaded in different stages of startup.

Basically, some variables can only be usefully initialized after some
part(s) of startup have happened already.  One way of dealing with
this is to have the variables record this information (e.g., in a
plist of their symbol) that would allow us evaluate each variable only
once, at the earliest opportunity where the prerequisites are
fulfilled.

> E.g. the current way `blink-cursor-mode` is defined means that if I
> start `emacs -Q -nw` and later create a GUI frame, that frame won't
> have a blinking cursor.  I think this is a bug (tho I can see that
> some people may consider it a feature if they (like me) dislike
> blinking cursors ;-0)

I think that's a separate issue.

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> Thanks, I actually meant to have that in the sources in some prominent
> place.

Ah, OK, I'll do that.

>> Currently we solve this by calling `custom-reevaluate-setting` twice for
>> every one of those variables.  But it means that we silently override
>> any change made to those variables from `early-init.el`.
>> 
>> As noted, the only var whose default value needs to be (re)computed
>> after loading `early-init.el` is `blink-cursor-mode`.  In theory there
>> could be more, but I think we should aim to reduce rather than increase
>> those cases.
>
> I think relying on a small number of such variables is not
> future-proof enough.  This case is a living proof: we decided
> something 2 years ago, but changes we did since then require us now to
> change that decision, which means we risk bumping into issues which we
> wanted to avoid back then.  That's a general problem with kludgey
> solutions.

Indeed.  Other than eliminate the `blink-cursor-mode` special case,
I can't see how to make it less kludgey.

> I think we need some new infrastructure that would support the two
> init files that are loaded in different stages of startup.
>
> Basically, some variables can only be usefully initialized after some
> part(s) of startup have happened already.  One way of dealing with
> this is to have the variables record this information (e.g., in a
> plist of their symbol) that would allow us evaluate each variable only
> once, at the earliest opportunity where the prerequisites are
> fulfilled.

In theory I would agree, but:
- We don't have any such system to record dependencies, so we'd have to
  design and implement it.  A minimal version would simply duplicate
  `customize-initialize-delayed` into two different options depending on
  the stage at which we should initialize it, but that'd still be pretty
  ad-hoc.
- The only need for this complexity is `blink-cursor-mode` and it's only
  needed because we currently handle `blink-cursor-mode` incorrectly.
  So, I'd rather fix the bug and avoid the complexity.

>> E.g. the current way `blink-cursor-mode` is defined means that if I
>> start `emacs -Q -nw` and later create a GUI frame, that frame won't
>> have a blinking cursor.  I think this is a bug (tho I can see that
>> some people may consider it a feature if they (like me) dislike
>> blinking cursors ;-0)
> I think that's a separate issue.

Agreed, which is why the patch I propose keeps this for later.


        Stefan

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ola.x.nilsson@axis.com,  45857@debbugs.gnu.org,  30994@debbugs.gnu.org
> Date: Thu, 14 Jan 2021 16:02:30 -0500
> 
> > I think relying on a small number of such variables is not
> > future-proof enough.  This case is a living proof: we decided
> > something 2 years ago, but changes we did since then require us now to
> > change that decision, which means we risk bumping into issues which we
> > wanted to avoid back then.  That's a general problem with kludgey
> > solutions.
> 
> Indeed.  Other than eliminate the `blink-cursor-mode` special case,
> I can't see how to make it less kludgey.

But that's still the same kludge: we will rely on the fact that there
are currently no (i.e. zero, a.k.a. "a small number") of such
variables.

> > Basically, some variables can only be usefully initialized after some
> > part(s) of startup have happened already.  One way of dealing with
> > this is to have the variables record this information (e.g., in a
> > plist of their symbol) that would allow us evaluate each variable only
> > once, at the earliest opportunity where the prerequisites are
> > fulfilled.
> 
> In theory I would agree, but:
> - We don't have any such system to record dependencies, so we'd have to
>   design and implement it.  A minimal version would simply duplicate
>   `customize-initialize-delayed` into two different options depending on
>   the stage at which we should initialize it, but that'd still be pretty
>   ad-hoc.

It isn't ad-hoc, because the stages in the startup process and their
effects are clearly defined and didn't change much for a long time.

> - The only need for this complexity is `blink-cursor-mode` and it's only
>   needed because we currently handle `blink-cursor-mode` incorrectly.
>   So, I'd rather fix the bug and avoid the complexity.

That'd probably work for another couple of years, and then break
again.  The early-init file introduction is letting a genie out of the
bottle: we don't yet know what it will eventually require, but we
already see some serious problems it causes that we need to adapt to.
I say we should get ready for the future now.  Introducing the
infrastructure I mentioned is not a big deal.

I don't want to argue further about this, so if you are still
unconvinced, so be it.

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> But that's still the same kludge: we will rely on the fact that there
> are currently no (i.e. zero, a.k.a. "a small number") of such
> variables.

I'm hoping that there's a good reason why the one exception we have
suffers from a bug, and for that reason we should resist the temptation
to add more such cases.

>> > Basically, some variables can only be usefully initialized after some
>> > part(s) of startup have happened already.  One way of dealing with
>> > this is to have the variables record this information (e.g., in a
>> > plist of their symbol) that would allow us evaluate each variable only
>> > once, at the earliest opportunity where the prerequisites are
>> > fulfilled.
[...]
> I say we should get ready for the future now.  Introducing the
> infrastructure I mentioned is not a big deal.

Hmm... maybe I misunderstood what you meant by that infrastructure, then.
Could you outline more concretely what it would look like?

Also, would it currently result in a behavior that's different from:

- initialize all delayed vars before early-init
- re-initialize blink-cursor-mode after early-init

?


        Stefan

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ola.x.nilsson@axis.com,  45857@debbugs.gnu.org,  30994@debbugs.gnu.org
> Date: Fri, 15 Jan 2021 11:08:52 -0500
> 
> >> > Basically, some variables can only be usefully initialized after some
> >> > part(s) of startup have happened already.  One way of dealing with
> >> > this is to have the variables record this information (e.g., in a
> >> > plist of their symbol) that would allow us evaluate each variable only
> >> > once, at the earliest opportunity where the prerequisites are
> >> > fulfilled.
> [...]
> > I say we should get ready for the future now.  Introducing the
> > infrastructure I mentioned is not a big deal.
> 
> Hmm... maybe I misunderstood what you meant by that infrastructure, then.
> Could you outline more concretely what it would look like?

In a nutshell: record the startup phase for each variable in a plits
of its symbol.

> Also, would it currently result in a behavior that's different from:
> 
> - initialize all delayed vars before early-init
> - re-initialize blink-cursor-mode after early-init
> 
> ?

With my proposal, blink-cursor-mode will not be mentioned by name in
any of these places.

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

>> >> > Basically, some variables can only be usefully initialized after some
>> >> > part(s) of startup have happened already.  One way of dealing with
>> >> > this is to have the variables record this information (e.g., in a
>> >> > plist of their symbol) that would allow us evaluate each variable only
>> >> > once, at the earliest opportunity where the prerequisites are
>> >> > fulfilled.
>> [...]
>> > I say we should get ready for the future now.  Introducing the
>> > infrastructure I mentioned is not a big deal.
>> Hmm... maybe I misunderstood what you meant by that infrastructure, then.
>> Could you outline more concretely what it would look like?
> In a nutshell: record the startup phase for each variable in a plits
> of its symbol.

Ah, so instead of just

    :initialize #'custom-initialize-delay

we'd have something like

    :initialize (custom-initialize-delay 'before-early-init)
or
    :initialize (custom-initialize-delay 'after-early-init)

Where `custom-initialize-delay` becomes something like

    (defun custom-initialize-delay (phase)
      (lambda (symbol)
        (push (cons symbol phase) custom-delayed-init-variables)))

depending on when we want the initialization to take place?

>> Also, would it currently result in a behavior that's different from:
>>
>> - initialize all delayed vars before early-init
>> - re-initialize blink-cursor-mode after early-init
>>
>> ?
>
> With my proposal, blink-cursor-mode will not be mentioned by name in
> any of these places.

Yes, of course, but I'm asking whether the resulting *behavior* you're
after is this one.


        Stefan

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

I looked at the issue of the default value of `blink-cursor-mode` and
it's actually easy to make it be t unconditionally, so as to eliminate
the need to re-evaluate variables after loading `early-init.el`.

Maybe some time in the future this need will re-appear at which point we
may want to introduce some fancier system to control which var is
initialized when, but I think for now the patch below is
a better solution.


        Stefan


2021-01-15  Stefan Monnier  <monnier@iro.umontreal.ca>

    * lisp/startup.el: Fix bug#45857, bug#30994, and bug#45913.

    (command-line): Don't re-evaluate the `custom-delayed-init-variables`
    a second time after reading the `early-init.el` file.
    (x-apply-session-resources): Set `blink-cursor-mode` rather than
    `no-blinking-cursor`.

    * lisp/frame.el (blink-cursor-start): Turn `blink-cursor-mode` off
    if `blink-cursor-mode` was set to nil.
    (blink-cursor-mode): Default to it being enabled regardless of
    `window-system`.


diff --git a/lisp/frame.el b/lisp/frame.el
index e2d7f21a49..06aab269dd 100644
--- a/lisp/frame.el
+++ b/lisp/frame.el
@@ -2552,13 +2552,15 @@ blink-cursor-start
 This starts the timer `blink-cursor-timer', which makes the cursor blink
 if appropriate.  It also arranges to cancel that timer when the next
 command starts, by installing a pre-command hook."
-  (when (null blink-cursor-timer)
+  (cond
+   ((null blink-cursor-mode) (blink-cursor-mode -1))
+   ((null blink-cursor-timer)
     ;; Set up the timer first, so that if this signals an error,
     ;; blink-cursor-end is not added to pre-command-hook.
     (setq blink-cursor-blinks-done 1)
     (blink-cursor--start-timer)
     (add-hook 'pre-command-hook #'blink-cursor-end)
-    (internal-show-cursor nil nil)))
+    (internal-show-cursor nil nil))))
 
 (defun blink-cursor-timer-function ()
   "Timer function of timer `blink-cursor-timer'."
@@ -2637,9 +2639,8 @@ blink-cursor-mode
 terminals, cursor blinking is controlled by the terminal."
   :init-value (not (or noninteractive
 		       no-blinking-cursor
-		       (eq system-type 'ms-dos)
-		       (not (display-blink-cursor-p))))
-  :initialize 'custom-initialize-delay
+		       (eq system-type 'ms-dos)))
+  :initialize #'custom-initialize-delay
   :group 'cursor
   :global t
   (blink-cursor-suspend)
diff --git a/lisp/startup.el b/lisp/startup.el
index 552802a38d..7011fbf458 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1172,6 +1172,7 @@ command-line
         ;; are dependencies between them.
         (nreverse custom-delayed-init-variables))
   (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
+  (setq custom-delayed-init-variables nil)
 
   ;; Warn for invalid user name.
   (when init-file-user
@@ -1301,12 +1302,6 @@ command-line
     (startup--setup-quote-display)
     (setq internal--text-quoting-flag t))
 
-  ;; Re-evaluate again the predefined variables whose initial value
-  ;; depends on the runtime context, in case some of them depend on
-  ;; the window-system features.  Example: blink-cursor-mode.
-  (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
-  (setq custom-delayed-init-variables nil)
-
   (normal-erase-is-backspace-setup-frame)
 
   ;; Register default TTY colors for the case the terminal hasn't a
@@ -1487,13 +1482,13 @@ x-apply-session-resources
 opens a graphical frame.
 
 This can set the values of `menu-bar-mode', `tool-bar-mode',
-`tab-bar-mode', and `no-blinking-cursor', as well as the `cursor' face.
+`tab-bar-mode', and `blink-cursor-mode', as well as the `cursor' face.
 Changed settings will be marked as \"CHANGED outside of Customize\"."
   (let ((no-vals  '("no" "off" "false" "0"))
 	(settings '(("menuBar" "MenuBar" menu-bar-mode nil)
 		    ("toolBar" "ToolBar" tool-bar-mode nil)
 		    ("scrollBar" "ScrollBar" scroll-bar-mode nil)
-		    ("cursorBlink" "CursorBlink" no-blinking-cursor t))))
+		    ("cursorBlink" "CursorBlink" blink-cursor-mode nil))))
     (dolist (x settings)
       (if (member (x-get-resource (nth 0 x) (nth 1 x)) no-vals)
 	  (set (nth 2 x) (nth 3 x)))))

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ola.x.nilsson@axis.com,  45857@debbugs.gnu.org,  30994@debbugs.gnu.org
> Date: Fri, 15 Jan 2021 17:19:33 -0500
> 
> > In a nutshell: record the startup phase for each variable in a plits
> > of its symbol.
> 
> Ah, so instead of just
> 
>     :initialize #'custom-initialize-delay
> 
> we'd have something like
> 
>     :initialize (custom-initialize-delay 'before-early-init)
> or
>     :initialize (custom-initialize-delay 'after-early-init)
> 
> Where `custom-initialize-delay` becomes something like
> 
>     (defun custom-initialize-delay (phase)
>       (lambda (symbol)
>         (push (cons symbol phase) custom-delayed-init-variables)))
> 
> depending on when we want the initialization to take place?

Yes, something like that.  Also 'after-init-file, 'after-term-setup,
'after-window-setup -- per the startup phases we already have.

> >> Also, would it currently result in a behavior that's different from:
> >>
> >> - initialize all delayed vars before early-init
> >> - re-initialize blink-cursor-mode after early-init
> >>
> >> ?
> >
> > With my proposal, blink-cursor-mode will not be mentioned by name in
> > any of these places.
> 
> Yes, of course, but I'm asking whether the resulting *behavior* you're
> after is this one.

The current result will be the same, yes.  But I think it's more
future-proof, the only thing that needs to be figured out is the
earliest startup phase after which a given variable needs to be
initialized.

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> Maybe some time in the future this need will re-appear at which point we
> may want to introduce some fancier system to control which var is
> initialized when, but I think for now the patch below is
> a better solution.

Pushed,


        Stefan

>
>         Stefan
>
>
> 2021-01-15  Stefan Monnier  <monnier@iro.umontreal.ca>
>
>     * lisp/startup.el: Fix bug#45857, bug#30994, and bug#45913.
>
>     (command-line): Don't re-evaluate the `custom-delayed-init-variables`
>     a second time after reading the `early-init.el` file.
>     (x-apply-session-resources): Set `blink-cursor-mode` rather than
>     `no-blinking-cursor`.
>
>     * lisp/frame.el (blink-cursor-start): Turn `blink-cursor-mode` off
>     if `blink-cursor-mode` was set to nil.
>     (blink-cursor-mode): Default to it being enabled regardless of
>     `window-system`.
>
>
> diff --git a/lisp/frame.el b/lisp/frame.el
> index e2d7f21a49..06aab269dd 100644
> --- a/lisp/frame.el
> +++ b/lisp/frame.el
> @@ -2552,13 +2552,15 @@ blink-cursor-start
>  This starts the timer `blink-cursor-timer', which makes the cursor blink
>  if appropriate.  It also arranges to cancel that timer when the next
>  command starts, by installing a pre-command hook."
> -  (when (null blink-cursor-timer)
> +  (cond
> +   ((null blink-cursor-mode) (blink-cursor-mode -1))
> +   ((null blink-cursor-timer)
>      ;; Set up the timer first, so that if this signals an error,
>      ;; blink-cursor-end is not added to pre-command-hook.
>      (setq blink-cursor-blinks-done 1)
>      (blink-cursor--start-timer)
>      (add-hook 'pre-command-hook #'blink-cursor-end)
> -    (internal-show-cursor nil nil)))
> +    (internal-show-cursor nil nil))))
>  
>  (defun blink-cursor-timer-function ()
>    "Timer function of timer `blink-cursor-timer'."
> @@ -2637,9 +2639,8 @@ blink-cursor-mode
>  terminals, cursor blinking is controlled by the terminal."
>    :init-value (not (or noninteractive
>  		       no-blinking-cursor
> -		       (eq system-type 'ms-dos)
> -		       (not (display-blink-cursor-p))))
> -  :initialize 'custom-initialize-delay
> +		       (eq system-type 'ms-dos)))
> +  :initialize #'custom-initialize-delay
>    :group 'cursor
>    :global t
>    (blink-cursor-suspend)
> diff --git a/lisp/startup.el b/lisp/startup.el
> index 552802a38d..7011fbf458 100644
> --- a/lisp/startup.el
> +++ b/lisp/startup.el
> @@ -1172,6 +1172,7 @@ command-line
>          ;; are dependencies between them.
>          (nreverse custom-delayed-init-variables))
>    (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
> +  (setq custom-delayed-init-variables nil)
>  
>    ;; Warn for invalid user name.
>    (when init-file-user
> @@ -1301,12 +1302,6 @@ command-line
>      (startup--setup-quote-display)
>      (setq internal--text-quoting-flag t))
>  
> -  ;; Re-evaluate again the predefined variables whose initial value
> -  ;; depends on the runtime context, in case some of them depend on
> -  ;; the window-system features.  Example: blink-cursor-mode.
> -  (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
> -  (setq custom-delayed-init-variables nil)
> -
>    (normal-erase-is-backspace-setup-frame)
>  
>    ;; Register default TTY colors for the case the terminal hasn't a
> @@ -1487,13 +1482,13 @@ x-apply-session-resources
>  opens a graphical frame.
>  
>  This can set the values of `menu-bar-mode', `tool-bar-mode',
> -`tab-bar-mode', and `no-blinking-cursor', as well as the `cursor' face.
> +`tab-bar-mode', and `blink-cursor-mode', as well as the `cursor' face.
>  Changed settings will be marked as \"CHANGED outside of Customize\"."
>    (let ((no-vals  '("no" "off" "false" "0"))
>  	(settings '(("menuBar" "MenuBar" menu-bar-mode nil)
>  		    ("toolBar" "ToolBar" tool-bar-mode nil)
>  		    ("scrollBar" "ScrollBar" scroll-bar-mode nil)
> -		    ("cursorBlink" "CursorBlink" no-blinking-cursor t))))
> +		    ("cursorBlink" "CursorBlink" blink-cursor-mode nil))))
>      (dolist (x settings)
>        (if (member (x-get-resource (nth 0 x) (nth 1 x)) no-vals)
>  	  (set (nth 2 x) (nth 3 x)))))

bug#30994: bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Phillip Lord]

I am still getting start up package.el oddities. While I can reset
package-user-dir in early-init.el now and it is correct when init.el
starts up, my package-initialize is raising an error from auctex in my
case.

    Warning (initialization): An error occurred while loading ‘/home/phillord/.emacs.d/init.el’:

    File is missing: Cannot open load file, No such file or directory, tex-site


package-user-dir looks right

    package-user-dir is a variable defined in ‘package.el’.
    Its value is "~/.emacs.d/elpa/28.0"
    Original value was "~/.emacs.d/elpa"

but load-path is not correct, having been set up for the default
location of package-user-dir


load-path is a variable defined in ‘C source code’.
Its value is
("/home/phillord/.emacs.d/elpa/auctex-13.0.3" "/home/phillord/.emacs.d/elpa/cider-1.0.0" "/home/phillord/.emacs.d/elpa/clojure-mode-5.12.0"

.....


Phil



Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Maybe some time in the future this need will re-appear at which point we
>> may want to introduce some fancier system to control which var is
>> initialized when, but I think for now the patch below is
>> a better solution.
>
> Pushed,
>
>
>         Stefan
>
>>
>>         Stefan
>>
>>
>> 2021-01-15  Stefan Monnier  <monnier@iro.umontreal.ca>
>>
>>     * lisp/startup.el: Fix bug#45857, bug#30994, and bug#45913.
>>
>>     (command-line): Don't re-evaluate the `custom-delayed-init-variables`
>>     a second time after reading the `early-init.el` file.
>>     (x-apply-session-resources): Set `blink-cursor-mode` rather than
>>     `no-blinking-cursor`.
>>
>>     * lisp/frame.el (blink-cursor-start): Turn `blink-cursor-mode` off
>>     if `blink-cursor-mode` was set to nil.
>>     (blink-cursor-mode): Default to it being enabled regardless of
>>     `window-system`.
>>
>>
>> diff --git a/lisp/frame.el b/lisp/frame.el
>> index e2d7f21a49..06aab269dd 100644
>> --- a/lisp/frame.el
>> +++ b/lisp/frame.el
>> @@ -2552,13 +2552,15 @@ blink-cursor-start
>>  This starts the timer `blink-cursor-timer', which makes the cursor blink
>>  if appropriate.  It also arranges to cancel that timer when the next
>>  command starts, by installing a pre-command hook."
>> -  (when (null blink-cursor-timer)
>> +  (cond
>> +   ((null blink-cursor-mode) (blink-cursor-mode -1))
>> +   ((null blink-cursor-timer)
>>      ;; Set up the timer first, so that if this signals an error,
>>      ;; blink-cursor-end is not added to pre-command-hook.
>>      (setq blink-cursor-blinks-done 1)
>>      (blink-cursor--start-timer)
>>      (add-hook 'pre-command-hook #'blink-cursor-end)
>> -    (internal-show-cursor nil nil)))
>> +    (internal-show-cursor nil nil))))
>>  
>>  (defun blink-cursor-timer-function ()
>>    "Timer function of timer `blink-cursor-timer'."
>> @@ -2637,9 +2639,8 @@ blink-cursor-mode
>>  terminals, cursor blinking is controlled by the terminal."
>>    :init-value (not (or noninteractive
>>  		       no-blinking-cursor
>> -		       (eq system-type 'ms-dos)
>> -		       (not (display-blink-cursor-p))))
>> -  :initialize 'custom-initialize-delay
>> +		       (eq system-type 'ms-dos)))
>> +  :initialize #'custom-initialize-delay
>>    :group 'cursor
>>    :global t
>>    (blink-cursor-suspend)
>> diff --git a/lisp/startup.el b/lisp/startup.el
>> index 552802a38d..7011fbf458 100644
>> --- a/lisp/startup.el
>> +++ b/lisp/startup.el
>> @@ -1172,6 +1172,7 @@ command-line
>>          ;; are dependencies between them.
>>          (nreverse custom-delayed-init-variables))
>>    (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
>> +  (setq custom-delayed-init-variables nil)
>>  
>>    ;; Warn for invalid user name.
>>    (when init-file-user
>> @@ -1301,12 +1302,6 @@ command-line
>>      (startup--setup-quote-display)
>>      (setq internal--text-quoting-flag t))
>>  
>> -  ;; Re-evaluate again the predefined variables whose initial value
>> -  ;; depends on the runtime context, in case some of them depend on
>> -  ;; the window-system features.  Example: blink-cursor-mode.
>> -  (mapc #'custom-reevaluate-setting custom-delayed-init-variables)
>> -  (setq custom-delayed-init-variables nil)
>> -
>>    (normal-erase-is-backspace-setup-frame)
>>  
>>    ;; Register default TTY colors for the case the terminal hasn't a
>> @@ -1487,13 +1482,13 @@ x-apply-session-resources
>>  opens a graphical frame.
>>  
>>  This can set the values of `menu-bar-mode', `tool-bar-mode',
>> -`tab-bar-mode', and `no-blinking-cursor', as well as the `cursor' face.
>> +`tab-bar-mode', and `blink-cursor-mode', as well as the `cursor' face.
>>  Changed settings will be marked as \"CHANGED outside of Customize\"."
>>    (let ((no-vals  '("no" "off" "false" "0"))
>>  	(settings '(("menuBar" "MenuBar" menu-bar-mode nil)
>>  		    ("toolBar" "ToolBar" tool-bar-mode nil)
>>  		    ("scrollBar" "ScrollBar" scroll-bar-mode nil)
>> -		    ("cursorBlink" "CursorBlink" no-blinking-cursor t))))
>> +		    ("cursorBlink" "CursorBlink" blink-cursor-mode nil))))
>>      (dolist (x settings)
>>        (if (member (x-get-resource (nth 0 x) (nth 1 x)) no-vals)
>>  	  (set (nth 2 x) (nth 3 x)))))

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Stefan Monnier]

> I am still getting start up package.el oddities. While I can reset
> package-user-dir in early-init.el now and it is correct when init.el
> starts up, my package-initialize is raising an error from auctex in my
> case.
>
>     Warning (initialization): An error occurred while loading ‘/home/phillord/.emacs.d/init.el’:
>
>     File is missing: Cannot open load file, No such file or directory, tex-site
>
>
> package-user-dir looks right
>
>     package-user-dir is a variable defined in ‘package.el’.
>     Its value is "~/.emacs.d/elpa/28.0"
>     Original value was "~/.emacs.d/elpa"
>
> but load-path is not correct, having been set up for the default
> location of package-user-dir
>
>
> load-path is a variable defined in ‘C source code’.
> Its value is
> ("/home/phillord/.emacs.d/elpa/auctex-13.0.3"
> "/home/phillord/.emacs.d/elpa/cider-1.0.0"
> "/home/phillord/.emacs.d/elpa/clojure-mode-5.12.0"

Does /home/phillord/.emacs.d/elpa/auctex-13.0.3/ exist?
How 'bout /home/phillord/.emacs.d/elpa/auctex-13.0.3/tex-site.el ?
How 'bout /home/phillord/.emacs.d/elpa/28.0/auctex-13.0.3/ ?


        Stefan

bug#45857: 28.0.50; Not possible to set package-user-dir in early-init.el

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> load-path is a variable defined in ‘C source code’.
>> Its value is
>> ("/home/phillord/.emacs.d/elpa/auctex-13.0.3"
>> "/home/phillord/.emacs.d/elpa/cider-1.0.0"
>> "/home/phillord/.emacs.d/elpa/clojure-mode-5.12.0"
>
> Does /home/phillord/.emacs.d/elpa/auctex-13.0.3/ exist?
> How 'bout /home/phillord/.emacs.d/elpa/auctex-13.0.3/tex-site.el ?
> How 'bout /home/phillord/.emacs.d/elpa/28.0/auctex-13.0.3/ ?


Ah, okay, got it. It was package-quickstart.el by the look of things.

Apologies for the noise.

Phil