bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Hi,

Some user settings in Org mode need to be set before loading the major mode.
Further, some people want to set such settings in individual files using file-local variables.

However, Emacs loads file-local variables only _after_ major mode is being loaded.
As a result, we are getting user requests like https://list.orgmode.org/587be554-906c-5370-2cf2-f08b14fa58ff@gmail.com/T/#u

Currently, Elisp allows manual loading of file-local variables via `hack-local-variables'. However, this function itself is too forceful, and it is running, e.g. in non-file buffers. Moreover, calling `hack-local-variables' does not prevent its subsequent call after Org mode is loaded, which sometimes causes double unsafe variable prompts.
See https://list.orgmode.org/87r11wkmew.fsf@ucl.ac.uk/T/#mab6359ed2107d5515c6bb6b266551f0c5049ceca

Would it be possible in Emacs to implement a function or possibly major-mode option that will make Emacs load local variables before loading major mode, not after?

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> Some user settings in Org mode need to be set before loading the major mode.
> Further, some people want to set such settings in individual files
> using file-local variables.
>
> However, Emacs loads file-local variables only _after_ major mode is
> being loaded.
> As a result, we are getting user requests like
> https://list.orgmode.org/587be554-906c-5370-2cf2-f08b14fa58ff@gmail.com/T/#u

Is the problem here that there are defvars in the major-mode file being
loaded that depend on one another?  (I'm not sure I understand the
explanation in that bug report.)

If that's the case, then this is inherently fragile, since the user may
have already loaded the .el file otherwise.  So visiting files in a
different order will lead to other results, and eval-ing local variables
before loading the files won't really help, in general.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Is the problem here that there are defvars in the major-mode file being
> loaded that depend on one another?  (I'm not sure I understand the
> explanation in that bug report.)

No.

Let me try to explain a simple case when setting variables before/after
loading Org mode matters.

We have a variable org-startup-folded:

   Non-nil means entering Org mode will switch to OVERVIEW.

This variable controls the initial state of the Org buffer.
When, non-nil, headings will be folded at the end of Org initialization.
This folding happens inside org-mode specification and hence setting the
variable after Org mode is loaded makes no difference.

There are also less trivial cases like some variables controlling Org
parser. For example, we have org-todo-keywords. Its value controls how
the headlines are parsed. Because Org parser is initialized during Org
loading, setting this variable after loading Org mode can be simply
disastrous.

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Visuwesh]

[சனி ஆகஸ்ட் 06, 2022] Ihor Radchenko wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Is the problem here that there are defvars in the major-mode file being
>> loaded that depend on one another?  (I'm not sure I understand the
>> explanation in that bug report.)
>
> No.
>
> Let me try to explain a simple case when setting variables before/after
> loading Org mode matters.
>
> We have a variable org-startup-folded:
>
>    Non-nil means entering Org mode will switch to OVERVIEW.
>
> This variable controls the initial state of the Org buffer.
> When, non-nil, headings will be folded at the end of Org initialization.
> This folding happens inside org-mode specification and hence setting the
> variable after Org mode is loaded makes no difference.
>
> There are also less trivial cases like some variables controlling Org
> parser. For example, we have org-todo-keywords. Its value controls how
> the headlines are parsed. Because Org parser is initialized during Org
> loading, setting this variable after loading Org mode can be simply
> disastrous.

Correct me if I'm wrong, this is about the major-mode definition
clearing the buffer's local variables before turning on the major-mode,
right?  If so, I was hit by this for a personal package too where I
needed to expose a local variable for the major-mode-hook but failed
because the function created `define-derived-mode' clears the buffer
local variables before running the major-mode-hook and the major-mode's
BODY.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Phil Sainty]

On 2022-08-07 02:00, Visuwesh wrote:
> Correct me if I'm wrong, this is about the major-mode definition
> clearing the buffer's local variables before turning on the major-mode,
> right?

To elaborate, every major mode calls `kill-all-local-variables',
and this is pretty much the first thing that happens (although
`change-major-mode-hook' runs before the local variables are
actually clobbered).

https://stackoverflow.com/a/19295380/324105 (specifically the
"Derived modes, and mode hooks" section) might be a useful write-up
if you want to get your head around the sequence of events.

Note that variables which are marked as permanent-local will
retain their values.  Refer to:

M-x elisp-index-search RET kill-all-local-variables


-Phil

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> We have a variable org-startup-folded:
>
>    Non-nil means entering Org mode will switch to OVERVIEW.
>
> This variable controls the initial state of the Org buffer.
> When, non-nil, headings will be folded at the end of Org initialization.
> This folding happens inside org-mode specification and hence setting the
> variable after Org mode is loaded makes no difference.

I see.  In that case, it does seem attractive to allow file-local
variables to set the variables before running the mode.  However, as
Visuwesh points out, a major mode clears buffer-local variables, so just
setting them before running the mode won't do anything.

So I think the interface here is just a bit confused and confusing.  It
would make more sense to allow users to do these things via mode
hooks/interface functions instead of variables.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I see.  In that case, it does seem attractive to allow file-local
> variables to set the variables before running the mode.  However, as
> Visuwesh points out, a major mode clears buffer-local variables, so just
> setting them before running the mode won't do anything.

Indeed. I imagine something like :before-hook in `define-derived-mode'
that will run just before the mode body.

Or maybe more direct :hack-local-variables-when defining when to run
`hack-local-variables' in contrast to currently hard-coded call in
`run-mode-hooks'. This is not equivalent to calling
`hack-local-variables' in :before-hook because `run-mode-hooks' would
call `hack-local-variables' second time in such a case.

Or maybe something more user-facing like major-mode-before-init-hook
that will be executed before the major mode body form in
`define-derived-mode'.

> So I think the interface here is just a bit confused and confusing.  It
> would make more sense to allow users to do these things via mode
> hooks/interface functions instead of variables.

Agree. However, the file-local variables specifically are not equivalent
to the mode hooks. File/directory-local variables can be set on
per-file/per-project basis, which is much more convenient compared to
explicitly defining a global hook.

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> Agree. However, the file-local variables specifically are not equivalent
> to the mode hooks. File/directory-local variables can be set on
> per-file/per-project basis, which is much more convenient compared to
> explicitly defining a global hook.

I think you can say something like

(eval . (org-startup-folded-mode))

to your .dir-locals (etc) to achieve this.  (And then you have to add
that to `safe-local-eval-forms' to avoid getting prompted, but Org could
do that in general.  And, of course, add this functionality as a minor
mode instead of just a variable.)

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Agree. However, the file-local variables specifically are not equivalent
>> to the mode hooks. File/directory-local variables can be set on
>> per-file/per-project basis, which is much more convenient compared to
>> explicitly defining a global hook.
>
> I think you can say something like
>
> (eval . (org-startup-folded-mode))
>
> to your .dir-locals (etc) to achieve this.  (And then you have to add
> that to `safe-local-eval-forms' to avoid getting prompted, but Org could
> do that in general.  And, of course, add this functionality as a minor
> mode instead of just a variable.)

I do not think that it is a good idea.

1. Having a need to do the eval is not expected by users.
2. Allowing eval is not safe compared to setting variables. Your
   suggestion about `safe-local-eval-forms' will not work well because
   even in your example it is not just about running
   (org-startup-folded-mode); org-startup-folded has 7 allowed values -
   marking each (org-startup-folded-mode value) form as safe sounds
   awkward.
3. org-startup-folded is just an example. We have numerous variables
   like this. I can recall at least several dozens without looking into
   source code. Creating a minor mode for each and every single possible
   variable like this is not maintainable. We will certainly forget
   creating dedicated modes when adding similar variables in future.
4. We have a somewhat similar approach in org-link settings - each link
   type is activated by evaluating a sexp. The result is slow Org
   startup time.

   If we implement per-file settings like you suggested, each setting
   will require partial re-evaluation of (org-mode). 
   A handful of settings like you suggested will easily degrade Org
   startup time by an order of magnitude.

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> 1. Having a need to do the eval is not expected by users.
> 2. Allowing eval is not safe compared to setting variables. Your
>    suggestion about `safe-local-eval-forms' will not work well because
>    even in your example it is not just about running
>    (org-startup-folded-mode); org-startup-folded has 7 allowed values -
>    marking each (org-startup-folded-mode value) form as safe sounds
>    awkward.

I'd forgotten that you can just enable minor modes without any of that.
You can just say

((org-mode . ((mode . org-startup-folded-mode))))

Or of there are different types:

((org-mode . ((org-startup-type . foo)
              (mode . org-startup-folded-mode))))

> 3. org-startup-folded is just an example. We have numerous variables
>    like this. I can recall at least several dozens without looking into
>    source code. Creating a minor mode for each and every single possible
>    variable like this is not maintainable. We will certainly forget
>    creating dedicated modes when adding similar variables in future.

There's dozens of variables that have to be set before `org-mode' is
called?  That sounds like an fragile design.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> 3. org-startup-folded is just an example. We have numerous variables
>>    like this. I can recall at least several dozens without looking into
>>    source code. Creating a minor mode for each and every single possible
>>    variable like this is not maintainable. We will certainly forget
>>    creating dedicated modes when adding similar variables in future.
>
> There's dozens of variables that have to be set before `org-mode' is
> called?  That sounds like an fragile design.

This is not strictly necessary to set them before the whole Org is
loaded. In some cases, only part of Org has to be reloaded: font-locking
setup; parser; folding state; etc (which is a normal scenario when done
interactively). However, such reloading adds up to file opening time. It
is much faster to set everything once and load Org compared to (1) load
org; (2) load variable and reload org; (3) load another variable and
reload org; and so on.

+ creating a dedicated minor mode for every variable is not maintainable.

Having Emacs support early setting would be much easier.

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> Having Emacs support early setting would be much easier.

You could make the relevant Org variables `permanent-local' -- then
calling `org-mode' won't clear them out.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Having Emacs support early setting would be much easier.
>
> You could make the relevant Org variables `permanent-local' -- then
> calling `org-mode' won't clear them out.

Could you please elaborate on how it is going to help with loading
file-local variables before loading Org mode?

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

>>> Having Emacs support early setting would be much easier.
>>
>> You could make the relevant Org variables `permanent-local' -- then
>> calling `org-mode' won't clear them out.
>
> Could you please elaborate on how it is going to help with loading
> file-local variables before loading Org mode?

Sorry, phrased that all wrong -- I meant that if we had a mechanism to
read the dir-locals earlier, then if you make the variables
permanently-local, then things would indeed work fine without any
further scaffolding here.

So perhaps we should just allow that, but I'm not sure how we'd specify
this.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Sorry, phrased that all wrong -- I meant that if we had a mechanism to
> read the dir-locals earlier, then if you make the variables
> permanently-local, then things would indeed work fine without any
> further scaffolding here.
>
> So perhaps we should just allow that, but I'm not sure how we'd specify
> this.

I am not sure what you mean by earlier.

The first thing executed when activating a major-mode is (according to
define-derived-mode) kill-all-local-variables which consist of

  run_hook (Qchange_major_mode_hook);
  reset_buffer_local_variables (current_buffer, 0);
  bset_update_mode_line (current_buffer);

Having 'permanent-local variables will only affect
change-major-mode-hook. I know no other hook ran earlier.

Why not instead add something that is run _after_
kill-all-local-variables? Then, there will be no need to fiddle with
'permanently-local (which might have undesired side effects).

Best,
Ihor

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> The first thing executed when activating a major-mode is (according to
> define-derived-mode) kill-all-local-variables which consist of
>
>   run_hook (Qchange_major_mode_hook);
>   reset_buffer_local_variables (current_buffer, 0);
>   bset_update_mode_line (current_buffer);
>
> Having 'permanent-local variables will only affect
> change-major-mode-hook. I know no other hook ran earlier.
>
> Why not instead add something that is run _after_
> kill-all-local-variables? Then, there will be no need to fiddle with
> 'permanently-local (which might have undesired side effects).

That'd work for things that use `define-derived-mode', but not all modes
are defined that way and call `kill-all-local-variables' "manually".

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Phil Sainty]

On 2022-08-13 00:38, Lars Ingebrigtsen wrote:
> Ihor Radchenko <yantar92@gmail.com> writes:
>> Why not instead add something that is run _after_
>> kill-all-local-variables?
> 
> That'd work for things that use `define-derived-mode', but not all 
> modes
> are defined that way and call `kill-all-local-variables' "manually".

Just as `kill-all-local-variables' runs `change-major-mode-hook'
before doing anything else, it could run another hook at the end
before returning.

(Caveat: I've not been following this discussion recently, so I'm
not actually sure if that covers the requirement or not.)

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Why not instead add something that is run _after_
>> kill-all-local-variables? Then, there will be no need to fiddle with
>> 'permanently-local (which might have undesired side effects).
>
> That'd work for things that use `define-derived-mode', but not all modes
> are defined that way and call `kill-all-local-variables' "manually".

I found such a case in ses.el. However, I am not sure if much can be
done mode-specifically in such scenario. Major modes using a custom
function may set the value of `major-mode' anywhere down to right before
`run-mode-hooks' call.

The only universal option then is indeed your suggestion about permanent
local variables. They were overridden by hack-local-variables in the
past anyway.

What can be probably done is a call to `hack-local-variables' inside
`kill-all-local-variables', but setting only the variables marked
'permanent-local. The later call to `hack-local-variables' inside
`run-mode-hooks' may then only go through the remaining variables.

WDYT?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Phil Sainty <psainty@orcon.net.nz> writes:

> Just as `kill-all-local-variables' runs `change-major-mode-hook'
> before doing anything else, it could run another hook at the end
> before returning.

I think that may be a promising approach -- it'd allow modes to opt in
to "early file-locals".

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> What can be probably done is a call to `hack-local-variables' inside
> `kill-all-local-variables', but setting only the variables marked
> 'permanent-local. The later call to `hack-local-variables' inside
> `run-mode-hooks' may then only go through the remaining variables.

This area is notoriously brittle and complicated.  I worry that any
change in evaluation order is going to break lots of stuff for various
use cases.

So I'd really prefer a new way to specify "early variables", but I have
no idea how that'd look in practice.  Perhaps just a new thing in
.dir-local.el, making that backwards-compatible is a challenge in
itself.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This area is notoriously brittle and complicated.  I worry that any
> change in evaluation order is going to break lots of stuff for various
> use cases.

Agree.

> So I'd really prefer a new way to specify "early variables", but I have
> no idea how that'd look in practice.  Perhaps just a new thing in
> .dir-local.el, making that backwards-compatible is a challenge in
> itself.

Maybe something like 'early-local property instead of 'permanent-local?
It will not break the existing cases, while allowing the major modes or
users to mark the variables that are safe to load early.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> Maybe something like 'early-local property instead of 'permanent-local?
> It will not break the existing cases, while allowing the major modes or
> users to mark the variables that are safe to load early.

Hm, yes, that sounds like a possibility.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > So I'd really prefer a new way to specify "early variables", but I have
  > > no idea how that'd look in practice.  Perhaps just a new thing in
  > > .dir-local.el, making that backwards-compatible is a challenge in
  > > itself.

  > Maybe something like 'early-local property instead of 'permanent-local?
  > It will not break the existing cases, while allowing the major modes or
  > users to mark the variables that are safe to load early.

Could you spell out the behavior you propose `early-local' to have?
From those three lines, I am not sure what it would do.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > Maybe something like 'early-local property instead of 'permanent-local?
>   > It will not break the existing cases, while allowing the major modes or
>   > users to mark the variables that are safe to load early.
>
> Could you spell out the behavior you propose `early-local' to have?
> From those three lines, I am not sure what it would do.

'early-local will apply file/directory-local variables _before_ executing
the major-mode body. In contrast, the variables without this property
are applied _after_ executing the major-mode body.

Currently, all the file/directory-locals are applied after executing the
major-mode body.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > 'early-local will apply file/directory-local variables _before_ executing
  > the major-mode body. In contrast, the variables without this property
  > are applied _after_ executing the major-mode body.

I suppose you'll do something to protect those local variables from
`kill-all-local-variables' so it won't kill them.

Will it distinguish THIS major mode's early local variables from the
early local variables of the PREVIOUS major mode?  Those ought to be
killed with all the rest.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > 'early-local will apply file/directory-local variables _before_ executing
>   > the major-mode body. In contrast, the variables without this property
>   > are applied _after_ executing the major-mode body.
>
> I suppose you'll do something to protect those local variables from
> `kill-all-local-variables' so it won't kill them.

No. Not killing the _already existing_ local variables is what
'persistent-local is for. What I am proposing here is only relevant to
buffer-local and directory-local variables that are going to be loaded
anyway. The change will only affect _when_ they are loaded.

> Will it distinguish THIS major mode's early local variables from the
> early local variables of the PREVIOUS major mode?  Those ought to be
> killed with all the rest.

I do not propose to keep the local variable values from the previous
major mode. What I suggest is running a part of hack-local-variables
earlier. Either way, file/directory-local variables will be set in the
buffer.

Hope I clarified the confusion.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > 'early-local will apply file/directory-local variables _before_ executing
  > >   > the major-mode body. In contrast, the variables without this property
  > >   > are applied _after_ executing the major-mode body.
  > >
  > > I suppose you'll do something to protect those local variables from
  > > `kill-all-local-variables' so it won't kill them.

  > No. Not killing the _already existing_ local variables is what
  > 'persistent-local is for. What I am proposing here is only relevant to
  > buffer-local and directory-local variables that are going to be loaded
  > anyway. The change will only affect _when_ they are loaded.

That's the right thing to do.  However, your previous message seemed
to imply a problem, because it proposed loading these buffer-local
variables "before executing the major mode body", and that would be
before `kill-all-local-variables'.  That function would kill these new
bindings, unless it were prevented from doing so.

Now I think that was just my misundersanding.  So I am relieved.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> That's the right thing to do.  However, your previous message seemed
> to imply a problem, because it proposed loading these buffer-local
> variables "before executing the major mode body", and that would be
> before `kill-all-local-variables'.  That function would kill these new
> bindings, unless it were prevented from doing so.

I was referring to define-derived-mode macro, where
`kill-all-local-variables' is actually called before BODY argument. Of
course, as Lars pointed, some major modes may not use
`define-derived-mode' and the meaning of "body" is different.

The major modes that do not use define-derived-mode are actually
problematic. While we can expect `kill-all-local-variables' to be called
early even in such major modes, the mode symbol is not yet known by the
time `kill-all-local-variables' is called. I am not sure how to approach
the proposed 'early-local property for mode-specific
file/directory-local variables in such scenario.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> We have a variable org-startup-folded:
>
>    Non-nil means entering Org mode will switch to OVERVIEW.
>
> This variable controls the initial state of the Org buffer.
> When, non-nil, headings will be folded at the end of Org initialization.
> This folding happens inside org-mode specification and hence setting the
> variable after Org mode is loaded makes no difference.

The problem is not that the variables are set too late,it's that you
fold too early: the folding should not to take place from within
`org-mode` but instead `org-mode` should add the folding function to
`hack-local-variables-hook`.

What you request is a common need, and `hack-local-variables-hook` is
the standard answer.


        Stefan

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The problem is not that the variables are set too late,it's that you
> fold too early: the folding should not to take place from within
> `org-mode` but instead `org-mode` should add the folding function to
> `hack-local-variables-hook`.
>
> What you request is a common need, and `hack-local-variables-hook` is
> the standard answer.

I guess that it can be an option. Though such hook will also need to be
guarded against repetitive execution (if `hack-local-variables' is
called by some minor mode or a command).

However, pretty much all the BODY in Org major mode definition will need
to go into the hook. It feels awkward.

Will moving the whole major mode definition into
`hack-local-variables-hook' be safe?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The problem is not that the variables are set too late,it's that you
>> fold too early: the folding should not to take place from within
>> `org-mode` but instead `org-mode` should add the folding function to
>> `hack-local-variables-hook`.
>>
>> What you request is a common need, and `hack-local-variables-hook` is
>> the standard answer.
>
> I guess that it can be an option. Though such hook will also need to be
> guarded against repetitive execution (if `hack-local-variables' is
> called by some minor mode or a command).
>
> However, pretty much all the BODY in Org major mode definition will need
> to go into the hook. It feels awkward.

The body of a major mode should usually be limited to setting some
variables.  All the font-lock highlighting, the imenu scanning, the
syntax-propertizing, etc... is done afterwards (the later the better).

The act of folding is not part of "defining" the major mode in my
book :-)

> Will moving the whole major mode definition into
> `hack-local-variables-hook' be safe?

Define "safe".  I'm sure it'll cause problems in corner cases.
If those problems come down to the fact that `hack-local-variables-hook'
doesn't fit the bill, then we can look at fixing that.


        Stefan

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> However, pretty much all the BODY in Org major mode definition will need
>> to go into the hook. It feels awkward.
>
> The body of a major mode should usually be limited to setting some
> variables.  All the font-lock highlighting, the imenu scanning, the
> syntax-propertizing, etc... is done afterwards (the later the better).

One issue I envision is when unsafe variable dialogue is being
displayed. If font-locking is not setup prior to that, Emacs will try to
fontify the visible part of buffer when displaying the dialogue and
later font-lock settings may not fully apply.

See https://list.orgmode.org/orgmode/878ro8kqwv.fsf@localhost/T/#u

> The act of folding is not part of "defining" the major mode in my
> book :-)

Indeed. However, a lot more things in Org mode depend on user
customizations. For example, buffer-invisibility-spec may be changed
depending on user settings, including settings defined in file-local
variables.

>> Will moving the whole major mode definition into
>> `hack-local-variables-hook' be safe?
>
> Define "safe".  I'm sure it'll cause problems in corner cases.
> If those problems come down to the fact that `hack-local-variables-hook'
> doesn't fit the bill, then we can look at fixing that.

I was mostly asking if you are aware about any gotchas.
As I stated above, there is at least one gotcha with early fontification
when unsafe variable dialogue is being shown.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> However, pretty much all the BODY in Org major mode definition will need
>>> to go into the hook. It feels awkward.
>> The body of a major mode should usually be limited to setting some
>> variables.  All the font-lock highlighting, the imenu scanning, the
>> syntax-propertizing, etc... is done afterwards (the later the better).
> One issue I envision is when unsafe variable dialogue is being
> displayed. If font-locking is not setup prior to that, Emacs will try to
> fontify the visible part of buffer when displaying the dialogue and
> later font-lock settings may not fully apply.
>
> See https://list.orgmode.org/orgmode/878ro8kqwv.fsf@localhost/T/#u

I didn't quite understand the problem from that thread.

Your description above isn't very clear either  On the one hand you say
"font-locking is not setup prior to that" but on the other you say
"Emacs will try to fontify the visible part of buffer" which only
happens if font-lock is setup, AFAIK.

I suggest you open a bug report to get to the bottom of this.

>> The act of folding is not part of "defining" the major mode in my
>> book :-)
> Indeed.  However, a lot more things in Org mode depend on user
> customizations.  For example, buffer-invisibility-spec may be changed
> depending on user settings, including settings defined in file-local
> variables.

I don't see a problem with postpone that to `hack-local-variables-hook', no.
[ Other approaches exist, of course, such as monitoring those config
  vars and updating the other pieces of data that depend on them
  whenever needed.  ]

>>> Will moving the whole major mode definition into
>>> `hack-local-variables-hook' be safe?
>> Define "safe".  I'm sure it'll cause problems in corner cases.
>> If those problems come down to the fact that `hack-local-variables-hook'
>> doesn't fit the bill, then we can look at fixing that.
> I was mostly asking if you are aware about any gotchas.

Not really, no.  I have no doubt that there are various (in addition to
the one you mention above).


        Stefan

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>> Will moving the whole major mode definition into
>>>> `hack-local-variables-hook' be safe?
>>> Define "safe".  I'm sure it'll cause problems in corner cases.
>>> If those problems come down to the fact that `hack-local-variables-hook'
>>> doesn't fit the bill, then we can look at fixing that.
>> I was mostly asking if you are aware about any gotchas.
>
> Not really, no.  I have no doubt that there are various (in addition to
> the one you mention above).

Looking at some of the related code, I had a feeling of "déjà vu":
`hack-local-variables-hook' indeed suffers from various gotchas
(starting with the non-intuitive name and the fact that it only
applies to file-visiting buffers), and I think my recommendation was out
of date: the better answer is to use `after-change-major-mode-hook` the
`:after-hook` thingy of `define-derived-mode`.

And the remaining gotcha (for Org's support of Emacs-25) is that in
Emacs<26 these were run *before* `hack-local-variables` :-(


        Stefan

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Not really, no.  I have no doubt that there are various (in addition to
>> the one you mention above).
>
> Looking at some of the related code, I had a feeling of "déjà vu":
> `hack-local-variables-hook' indeed suffers from various gotchas
> (starting with the non-intuitive name and the fact that it only
> applies to file-visiting buffers)

Does it mean that "Major modes can use this to examine user-specified
local variables in order to initialize other data structure based on
them." statement in the docstring of `hack-local-variables-hook' should
be amended?

> , and I think my recommendation was out
> of date: the better answer is to use `after-change-major-mode-hook` the
> `:after-hook` thingy of `define-derived-mode`.

Will it run before or after org-mode-hook?

> And the remaining gotcha (for Org's support of Emacs-25) is that in
> Emacs<26 these were run *before* `hack-local-variables` :-(

This is not a problem. Org officially supports Emacs >=26. No more need
to keep Emacs 25 compatibility. See
https://orgmode.org/worg/org-maintenance.html#emacs-compatibility

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> See https://list.orgmode.org/orgmode/878ro8kqwv.fsf@localhost/T/#u
>
> I didn't quite understand the problem from that thread.
>
> Your description above isn't very clear either  On the one hand you say
> "font-locking is not setup prior to that" but on the other you say
> "Emacs will try to fontify the visible part of buffer" which only
> happens if font-lock is setup, AFAIK.
>
> I suggest you open a bug report to get to the bottom of this.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=58888

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ihor Radchenko [2022-10-30 04:01:08] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> Not really, no.  I have no doubt that there are various (in addition to
>>> the one you mention above).
>>
>> Looking at some of the related code, I had a feeling of "déjà vu":
>> `hack-local-variables-hook' indeed suffers from various gotchas
>> (starting with the non-intuitive name and the fact that it only
>> applies to file-visiting buffers)
>
> Does it mean that "Major modes can use this to examine user-specified
> local variables in order to initialize other data structure based on
> them." statement in the docstring of `hack-local-variables-hook' should
> be amended?

Good question, maybe it should be amended to point to
`after-change-major-mode-hook`?

>> , and I think my recommendation was out
>> of date: the better answer is to use `after-change-major-mode-hook` the
>> `:after-hook` thingy of `define-derived-mode`.
>
> Will it run before or after org-mode-hook?

It's run after setting file-local vars, which themselves are set after
running the mode hooks.

bug#57003: 28.1.90; Can local variables be loaded before loading major mode?

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> , and I think my recommendation was out
>>> of date: the better answer is to use `after-change-major-mode-hook` the
>>> `:after-hook` thingy of `define-derived-mode`.
>>
>> Will it run before or after org-mode-hook?
>
> It's run after setting file-local vars, which themselves are set after
> running the mode hooks.

This is unfortunate.

org-mode-hook is "Mode hook for Org mode, run after the mode was turned
on.". But we need to load file-local variables in order to turn on Org
mode + respect file-local settings. So, neither
hack-local-variables-hook nor :after-hook will solve the problem I
suggested solving in this discussion.

For Org mode, it is necessary to complete the setup before org-mode-hook
is loaded (I am not talking about folding; there are more critical parts
like parser setup). Otherwise, the existing third-party packages adding
things to org-mode-hook will be broken.

Outside Org mode peculiarities, the fact that
hack-local-variables-hook is run after *-mode-hook also means that
subtle bugs might be introduced into major modes that are utilizing
hack-local-variables-hook for important setup.

I conclude that my original idea herein about a new option to load local
variables early may be even more important than I initially thought.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>