bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Jimmy Yuen Ho Wong]

As a discussion from bug #63300, it appears this long standing
undocumented behavior of `custom-save-variable` is coming into conflict
with the introduction of `connection-local-*` variables being user
customizable and the fact that Tramp in Emacs 29 sets them on
load. Here's a scenario where the combination of these behaviors results
in one too many surprises:

0. (setf custom-file "~/.emacs.d/custom.el")
1. M-x load-library tramp (or install a package that transitively
requires tramp, without the user knowning)
2. Now `connection-local-profile-alist` and
`connection-local-criteria-alist` are set by
`hack-connection-local-variables-apply`.
3. M-x list-packages
4. Installs a new package
5. Now in addition to `package-selected-packages` being updated, 2
gigantic variables are also saved. Since these connection-local
variables are highly machine, application and connection dependent,
saving them into the custom file will make it very annoying to be shared
across multiple machines. This violates the principle of least
astonishment.

Expectation:

`custom-save-variable` should only save the value of one variable
regardless of whether a custom file exists.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Eli Zaretskii]

> From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> Date: Sun, 04 Jun 2023 13:36:30 +0100
> 
> 
> As a discussion from bug #63300, it appears this long standing
> undocumented behavior of `custom-save-variable` is coming into conflict
> with the introduction of `connection-local-*` variables being user
> customizable and the fact that Tramp in Emacs 29 sets them on
> load. Here's a scenario where the combination of these behaviors results
> in one too many surprises:
> 
> 0. (setf custom-file "~/.emacs.d/custom.el")
> 1. M-x load-library tramp (or install a package that transitively
> requires tramp, without the user knowning)
> 2. Now `connection-local-profile-alist` and
> `connection-local-criteria-alist` are set by
> `hack-connection-local-variables-apply`.
> 3. M-x list-packages
> 4. Installs a new package
> 5. Now in addition to `package-selected-packages` being updated, 2
> gigantic variables are also saved. Since these connection-local
> variables are highly machine, application and connection dependent,
> saving them into the custom file will make it very annoying to be shared
> across multiple machines. This violates the principle of least
> astonishment.

I think the connection-local variables should be simple variables,
initialized from corresponding user options.  Then Tramp could hack
the variables without fear of clobbering user customizations.

Michael, can this be done on emacs-29 safely enough?

> Expectation:
> 
> `custom-save-variable` should only save the value of one variable
> regardless of whether a custom file exists.

How is custom-save-variable involved in the above scenario?

And what is custom-save-variable? did you mean customize-save-variable
instead?  That one does save just one variable, the one you type at
the prompt.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Jimmy Wong]

[-- Attachment #1: Type: text/plain, Size: 2458 bytes --]

Yes sorry, I mean customize-save-variable, custom-save-variable doesn’t exist. There’s a branch in customize-save-variable that saves all previously updated variables to the custom file if it exists.

I just took a look at cus-edit.el, there appears to be no function that can surgically serialize just one variable value to the custom file. Fixing this the right way will probably involve changing all the places that call custom-save-all and still arguably result in a breaking change.
On 4 Jun 2023 at 1:55 PM +0100, Eli Zaretskii <eliz@gnu.org>, wrote:
> > From: Jimmy Yuen Ho Wong <wyuenho@gmail.com>
> > Date: Sun, 04 Jun 2023 13:36:30 +0100
> >
> >
> > As a discussion from bug #63300, it appears this long standing
> > undocumented behavior of `custom-save-variable` is coming into conflict
> > with the introduction of `connection-local-*` variables being user
> > customizable and the fact that Tramp in Emacs 29 sets them on
> > load. Here's a scenario where the combination of these behaviors results
> > in one too many surprises:
> >
> > 0. (setf custom-file "~/.emacs.d/custom.el")
> > 1. M-x load-library tramp (or install a package that transitively
> > requires tramp, without the user knowning)
> > 2. Now `connection-local-profile-alist` and
> > `connection-local-criteria-alist` are set by
> > `hack-connection-local-variables-apply`.
> > 3. M-x list-packages
> > 4. Installs a new package
> > 5. Now in addition to `package-selected-packages` being updated, 2
> > gigantic variables are also saved. Since these connection-local
> > variables are highly machine, application and connection dependent,
> > saving them into the custom file will make it very annoying to be shared
> > across multiple machines. This violates the principle of least
> > astonishment.
>
> I think the connection-local variables should be simple variables,
> initialized from corresponding user options. Then Tramp could hack
> the variables without fear of clobbering user customizations.
>
> Michael, can this be done on emacs-29 safely enough?
>
> > Expectation:
> >
> > `custom-save-variable` should only save the value of one variable
> > regardless of whether a custom file exists.
>
> How is custom-save-variable involved in the above scenario?
>
> And what is custom-save-variable? did you mean customize-save-variable
> instead? That one does save just one variable, the one you type at
> the prompt.

[-- Attachment #2: Type: text/html, Size: 3048 bytes --]

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Eli Zaretskii]

> Date: Sun, 4 Jun 2023 14:02:55 +0100
> From: Jimmy Wong <wyuenho@gmail.com>
> Cc: 63891@debbugs.gnu.org
> 
> Yes sorry, I mean customize-save-variable, custom-save-variable doesn’t exist. There’s a branch in
> customize-save-variable that saves all previously updated variables to the custom file if it exists.

I don't think I understand.  customize-save-variable saves only a
single variable: the one whose name you type, with the value you type.
Which branch there does more, and how can you invoke that branch?

> I just took a look at cus-edit.el, there appears to be no function that can surgically serialize just one
> variable value to the custom file.

customize-save-variable is that function.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Jimmy Wong]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

This branch: https://github.com/emacs-mirror/emacs/blob/emacs-29/lisp/cus-edit.el#L1109

No it does not saves only one variable to file, it only saves one variable to file if you have only modified one variable.

As a matter of fact, custom-variable-save, custom-variable-mark-to-reset-standard, custom-face-save, custom-face-mark-to-reset-standard and custom-group-save all have the same problem. They all call custom-save-all and they all dump all modified customizable variable values on file without regard to whether it’s a single variable, face or a group the user asked Emacs to save.
On 4 Jun 2023 at 2:23 PM +0100, Eli Zaretskii <eliz@gnu.org>, wrote:
> > Date: Sun, 4 Jun 2023 14:02:55 +0100
> > From: Jimmy Wong <wyuenho@gmail.com>
> > Cc: 63891@debbugs.gnu.org
> >
> > Yes sorry, I mean customize-save-variable, custom-save-variable doesn’t exist. There’s a branch in
> > customize-save-variable that saves all previously updated variables to the custom file if it exists.
>
> I don't think I understand. customize-save-variable saves only a
> single variable: the one whose name you type, with the value you type.
> Which branch there does more, and how can you invoke that branch?
>
> > I just took a look at cus-edit.el, there appears to be no function that can surgically serialize just one
> > variable value to the custom file.
>
> customize-save-variable is that function.

[-- Attachment #2: Type: text/html, Size: 1950 bytes --]

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Eli Zaretskii]

> Date: Sun, 4 Jun 2023 15:00:01 +0100
> From: Jimmy Wong <wyuenho@gmail.com>
> Cc: 63891@debbugs.gnu.org
> 
> This branch: https://github.com/emacs-mirror/emacs/blob/emacs-29/lisp/cus-edit.el#L1109

I don't know what that is.  I'm using the Emacs Git repository, the
emacs-29 branch.

> No it does not saves only one variable to file, it only saves one variable to file if you have only modified
> one variable.

That's not what I see.  I've modified several options using the
menu-bar's Options menu, then typed

  M-x customize-save-variable RET truncate-lines RET y

and saw that only truncate-lines was written to the custom file.

If you see something else, please show a complete recipe that
reproduces the behavior you see.

> As a matter of fact, custom-variable-save, custom-variable-mark-to-reset-standard,
> custom-face-save, custom-face-mark-to-reset-standard and custom-group-save all have the same
> problem. They all call custom-save-all and they all dump all modified customizable variable values on
> file without regard to whether it’s a single variable, face or a group the user asked Emacs to save.

I wasn't talking about custom-save-all -- that indeed saves all the
options customized in this session.  I was talking about
customize-save-variable, which prompts for a single variable and its
value, and saves only that single variable, at least in my testing.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Jimmy Wong]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

Ok you are right, the problem seems to be somewhere in Tramp on require that sets the saved-value symbol property of connection-local-profile-alist and connection-local-criteria-alist without them being written to file, even when enable-connection-local-variables is set to nil. When custom-save-all naively scans the obarray for symbols, sees these variables having saved-value set without going through the customize machinary, it assumes they are saved to a file and saves them again.
On 4 Jun 2023 at 3:25 PM +0100, Eli Zaretskii <eliz@gnu.org>, wrote:
> > Date: Sun, 4 Jun 2023 15:00:01 +0100
> > From: Jimmy Wong <wyuenho@gmail.com>
> > Cc: 63891@debbugs.gnu.org
> >
> > This branch: https://github.com/emacs-mirror/emacs/blob/emacs-29/lisp/cus-edit.el#L1109
>
> I don't know what that is. I'm using the Emacs Git repository, the
> emacs-29 branch.
>
> > No it does not saves only one variable to file, it only saves one variable to file if you have only modified
> > one variable.
>
> That's not what I see. I've modified several options using the
> menu-bar's Options menu, then typed
>
> M-x customize-save-variable RET truncate-lines RET y
>
> and saw that only truncate-lines was written to the custom file.
>
> If you see something else, please show a complete recipe that
> reproduces the behavior you see.
>
> > As a matter of fact, custom-variable-save, custom-variable-mark-to-reset-standard,
> > custom-face-save, custom-face-mark-to-reset-standard and custom-group-save all have the same
> > problem. They all call custom-save-all and they all dump all modified customizable variable values on
> > file without regard to whether it’s a single variable, face or a group the user asked Emacs to save.
>
> I wasn't talking about custom-save-all -- that indeed saves all the
> options customized in this session. I was talking about
> customize-save-variable, which prompts for a single variable and its
> value, and saves only that single variable, at least in my testing.

[-- Attachment #2: Type: text/html, Size: 2630 bytes --]

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> No it does not saves only one variable to file, it only saves one variable to file if you have only modified
>> one variable.
>
> That's not what I see.  I've modified several options using the
> menu-bar's Options menu, then typed
>
>   M-x customize-save-variable RET truncate-lines RET y
>
> and saw that only truncate-lines was written to the custom file.
>
> If you see something else, please show a complete recipe that
> reproduces the behavior you see.

custom-set-variables sets the saved-value property of the
variables. This is what is called in connection-local-set-profiles and
connection-local-set-profile-variables.

Best regards, Michael.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jimmy Wong <wyuenho@gmail.com>,  63891@debbugs.gnu.org
> Date: Tue, 06 Jun 2023 14:01:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> Hi Eli,
> 
> >> No it does not saves only one variable to file, it only saves one variable to file if you have only modified
> >> one variable.
> >
> > That's not what I see.  I've modified several options using the
> > menu-bar's Options menu, then typed
> >
> >   M-x customize-save-variable RET truncate-lines RET y
> >
> > and saw that only truncate-lines was written to the custom file.
> >
> > If you see something else, please show a complete recipe that
> > reproduces the behavior you see.
> 
> custom-set-variables

That's a different function.  I was talking about
customize-save-variable.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> >> No it does not saves only one variable to file, it only saves one variable to file if you have only modified
>> >> one variable.
>> >
>> > That's not what I see.  I've modified several options using the
>> > menu-bar's Options menu, then typed
>> >
>> >   M-x customize-save-variable RET truncate-lines RET y
>> >
>> > and saw that only truncate-lines was written to the custom file.
>> >
>> > If you see something else, please show a complete recipe that
>> > reproduces the behavior you see.
>>
>> custom-set-variables
>
> That's a different function.  I was talking about
> customize-save-variable.

I know. I just wanted to clarify, where the saved-value of other user
options is set.

And customize-save-variable, although invoked with just one variable,
saves *all* variables with a saved-value property, IIUC.

A recipe you have asked for could be

--8<---------------cut here---------------start------------->8---
(defcustom a nil
  "" :type 'boolean)

(defcustom b nil
  "" :type 'boolean)

(defcustom c nil
  "" :type 'boolean)

(custom-set-variables '(a t) '(b t))

;; Nothing happened so far in the init file.

(customize-save-variable 'c t)

;; Your init file contains then

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(a t)
 '(b t)
 '(c t)
 ...)
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

[Oops, resending to the right bug address]

Michael Albinus <michael.albinus@gmx.de> writes:


 > custom-set-variables sets the saved-value property of the
 > variables. This is what is called in connection-local-set-profiles and
 > connection-local-set-profile-variables.

Why do those functions use custom-set-variables, instead of
customize-set-variable?

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Michael Albinus]

Mauro Aranda <maurooaranda@gmail.com> writes:

Hi Mauro,

>> custom-set-variables sets the saved-value property of the
>> variables. This is what is called in connection-local-set-profiles and
>> connection-local-set-profile-variables.
>
> Why do those functions use custom-set-variables, instead of
> customize-set-variable?

This is what we have used before. Due to bug#62106 we cannot use
customize-set-variable.

Best regards, Michael.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

On 27/10/23 08:51, Michael Albinus wrote:
 > Mauro Aranda <maurooaranda@gmail.com> writes:
 >
 > Hi Mauro,
 >
 >>> custom-set-variables sets the saved-value property of the
 >>> variables. This is what is called in connection-local-set-profiles and
 >>> connection-local-set-profile-variables.
 >>
 >> Why do those functions use custom-set-variables, instead of
 >> customize-set-variable?
 >
 > This is what we have used before. Due to bug#62106 we cannot use
 > customize-set-variable.
 >

I see, thanks.  I'll study this more carefully.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

Hi Michael,

I looked at this more deeply, and I think I still don't understand
what's being asked of Custom in this use case.

First, let me just say that I'm aware of some problems with the
custom-save-all approach to modify the custom-file.  It was reported in
Bug#14150, but, while a different approach to fix Bug#14150 could also
solve this bug, I'm still not sure if the use case in files-x.el is a
supported one.

Both functions, connection-local-set-profile-variables and
connection-local-set-profiles modify 2 defcustoms, and want to tell
Custom that a change has happened.  The usual way to do that is to call
customize-set-variable, because the assumption is that the user used
some command provided by a package to modify the option.  And the
setting lasts for the session, of course.  But the surprise in Bug#62106
was that the users weren't requesting these changes in the options.  It
was done without a choice.

So, the code was changed to use custom-set-variables, which is used in
the custom-file and which means all the settings here should persist
from session to session.  So, in addition to modifying the user option
without a choice, the code then said that these modifications should be
saved.  That's even worse, which should show that custom-set-variables
is just the wrong tool here.  Of course, there's the workaround of
resetting saved-value to nil if possible.  But that just means that if
the user has a saved setting, he/she could possibly end up with all
settings added by a package too.  Of course, if Custom had other
approach for saving the settings, that would not happen, but it wouldn't
happen if the code weren't lying to Custom either.  And please note that
a similar workaround could be added if the code used
customize-set-variable still.

But here is my first question, if packages are going to be changing this
2 options without asking the user about it, why do the packages need to
lie to Custom saying that the user asked for that? Why don't just setq,
add-to-list or modify it some other way? At least that way Custom would
know the truth, the setting was changed outside of Customize.

That's why I don't understand what is the expectation about Custom here
(apart from being less naive when saving the custom-file).  The code is
modifying a user option and tells Custom that it was upon the user
request, when in fact it hasn't.

Finally, have you considered the approach of having the user option plus
another variable which packages should modify when desired? Then the
code could merge the user settings with the package settings.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Drew Adams]

> if packages are going to be changing this 2 options
> without asking the user about it, why do the packages
> need to lie to Custom saying that the user asked for
> that?  Why don't just setq, add-to-list or modify it
> some other way? At least that way Custom would know
> the truth, the setting was changed outside of Customize.
> 
> That's why I don't understand what is the expectation
> about Custom here (apart from being less naive when
> saving the custom-file).  The code is modifying a user
> option and tells Custom that it was upon the user
> request, when in fact it hasn't.
> 
> Finally, have you considered the approach of having
> the user option plus another variable which packages
> should modify when desired? Then the code could merge
> the user settings with the package settings.

Hi Mauro,

My 2 cents about such things -

1. Libraries can usefully modify user options.  They
just need to make users aware of this happening, so
that taking advantage of this is a user choice.

In particular, libraries can define commands whose
purpose includes changing option values - sometimes
even saving such changes immediately.  To me, this
isn't a no-no.  What is a no-no is changing an
option value behind a user's back, and a fortiori
saving such a change - big NO-NO.

Wrt your last paragraph above: That's the approach
that Emacs generally takes.  It's OK, though it's a
bit of a kludge.  More importantly, users can _want_
to modify an option value on the fly, as opposed to
modifying its non-option shadow/stand-in variable.

2. Emacs should make available features I noted in
previous posts, such as a function to consider a
change to an option value (by program) to not be a
change.  This lets code change a value but not
have Customize consider that a change has been
made - so the change won't be saved automatically
or reported as having occurred.

This is a _convenience_ for users, not an obstacle:
be able to change behavior that's usually governed
by an option, without having Customize barf or save
the changes.  In effect, it's being able to use an
option temporarily as if it were not an option.

3. Emacs should also make available the ability for
`defvar' (or a new macro) to use the features of
`defcustom'.  In particular, this includes :set and
:type, and it includes the ability to persist the
value.

The difference between this and `defcustom' is that
uch variables aren't recognized by Emacs as user
options.  Users can't use the Customize UI or
`set-variable' with such vars.  `C-h v' provides no
`customize' link, etc.
___

#3 is maybe the most relevant to the points you
raised, but I also wanted to mention #1 and #2.

I proposed #3 to emacs-devel@gnu.org back in 2009:

https://lists.gnu.org/archive/html/emacs-devel/2009-10/msg00668.html

I sent a patch for #3 as bug #27348 (closed by Lars,
saying that no one would want such a feature):

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27348

That patch, with a couple trivial tweaks, is still
valid against Emacs 29, I believe.

It adds keyword `:not-custom-var' to `defcustom'.
If its value is non-`nil' then the variable doesn't
satisfy `custom-variable-p', which means it's _not
available for interactive use_ (completion,
`set-variable', `apropos-user-option' output, etc.).
IOW, such a variable is for code more than for user
configuration.

The patch also defines macro `defvarc', which is
just `defmacro' with `:not-custom-var' set to `t'.

The `defcustom' keywords etc. could have just been
added to `defvar' for optional use.  I defined a
separate macro just to not interfere with any
existing uses of `defvar'.

In sum, this uncouples interactive customization
from the other features that Customize offers, in
particular, type-checking and persistence, and it
provides those features for non-option variables.

The ability to type-check, provide `:set' and
`:initialize' trigger functions, automatically
`:require' libraries, add links to doc, associate
with one or more `:groups', etc.  These are useful
things to be able to do with at least some defvars,
not just with defcustoms.

Similarly, the ability to persist non-option vars
in a user's custom file can be useful.  This alone
is a frequent question, to which the answer has
been `savehist-additional-variables', `desktop.el',
or Bookmark+ variable-list bookmarks.

The patch also includes a macro `with-user-vars',
which temporarily lets a set of variables be
customizable.  That is, it lets you treat a
`defvarc' variable as if it were a `defcustom'
option.  So if you want, you can use the Customize
UI to change a defvarc's value, or define commands
that use (e.g. complete) `defvarc' variable names.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

On 28/10/23 15:22, Drew Adams wrote:
 >> if packages are going to be changing this 2 options
 >> without asking the user about it, why do the packages
 >> need to lie to Custom saying that the user asked for
 >> that?  Why don't just setq, add-to-list or modify it
 >> some other way? At least that way Custom would know
 >> the truth, the setting was changed outside of Customize.
 >>
 >> That's why I don't understand what is the expectation
 >> about Custom here (apart from being less naive when
 >> saving the custom-file).  The code is modifying a user
 >> option and tells Custom that it was upon the user
 >> request, when in fact it hasn't.
 >>
 >> Finally, have you considered the approach of having
 >> the user option plus another variable which packages
 >> should modify when desired? Then the code could merge
 >> the user settings with the package settings.
 >
 > Hi Mauro,
 >
 > My 2 cents about such things -
 >
 > 1. Libraries can usefully modify user options.  They
 > just need to make users aware of this happening, so
 > that taking advantage of this is a user choice.
 >
 > In particular, libraries can define commands whose
 > purpose includes changing option values - sometimes
 > even saving such changes immediately.  To me, this
 > isn't a no-no.  What is a no-no is changing an
 > option value behind a user's back, and a fortiori
 > saving such a change - big NO-NO.

Of course, and we agree.  I sent a message yesterday asking why didn't
the code just used customize-set-variable, which is one of the functions
that Custom gives for libraries when they want to change the value of
some user option for the session, upon user request.  That's why I'm
raising the point that this change is happening behind the user's back.

But it seems to be the way that these particular user options,
connection-local-profile-alist and connection-local-criteria-alist, are
supposed to be used by packages.  IOW, it seems to me that this is not
specific to Tramp.  For example:

emacs -Q
(progn
   (require 'files-x)
   (setq old-connection-local-profile-alist connection-local-profile-alist)
   (require 'eshell)
   (equal old-connection-local-profile-alist 
connection-local-profile-alist))

That evaluates to nil.  This is not happening via some command. Of
course, the libraries might need to change the value to work correctly,
but that's why I'm asking what's the expectation about Custom after the
change, because telling Custom this happened upon user request is not
true.  Another bad consequence is this:

M-x customize-option RET connection-local-profile-alist

State shown is: SAVED and set

But of course, if you visit custom-file, you'll see no such reference to
connection-local-profile-alist.  This should show that this is not the
way custom-set-variables is supposed to be used.

 > Wrt your last paragraph above: That's the approach
 > that Emacs generally takes.  It's OK, though it's a
 > bit of a kludge.  More importantly, users can _want_
 > to modify an option value on the fly, as opposed to
 > modifying its non-option shadow/stand-in variable.

OK, that's something I haven't considered.

 > 2. Emacs should make available features I noted in
 > previous posts, such as a function to consider a
 > change to an option value (by program) to not be a
 > change.  This lets code change a value but not
 > have Customize consider that a change has been
 > made - so the change won't be saved automatically
 > or reported as having occurred.

I wonder what does this mean in terms of the possible states for a user
option.  AFAICT, none of the current possible states (STANDARD, SAVED,
CHANGED, SET) fit in your description.
STANDARD: No, standard value and current value don't match.
SAVED: No, that setting wasn't saved.
CHANGED: No, because you're saying that Custom shouldn't consider the
change.
SET: No, because the code needs to use customize-set-variable, but that
wouldn't be Custom ignoring the change.

 > 3. Emacs should also make available the ability for
 > `defvar' (or a new macro) to use the features of
 > `defcustom'.  In particular, this includes :set and
 > :type, and it includes the ability to persist the
 > value.
 >
 > The difference between this and `defcustom' is that
 > uch variables aren't recognized by Emacs as user
 > options.  Users can't use the Customize UI or
 > `set-variable' with such vars.  `C-h v' provides no
 > `customize' link, etc.

This sounds interesting.

 > I proposed #3 to emacs-devel@gnu.org back in 2009:
 >
 > https://lists.gnu.org/archive/html/emacs-devel/2009-10/msg00668.html
 >
 > I sent a patch for #3 as bug #27348 (closed by Lars,
 > saying that no one would want such a feature):
 >
 > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27348
 >
 > That patch, with a couple trivial tweaks, is still
 > valid against Emacs 29, I believe.
 >
 > It adds keyword `:not-custom-var' to `defcustom'.
 > If its value is non-`nil' then the variable doesn't
 > satisfy `custom-variable-p', which means it's _not
 > available for interactive use_ (completion,
 > `set-variable', `apropos-user-option' output, etc.).
 > IOW, such a variable is for code more than for user
 > configuration.
 >
 > The patch also defines macro `defvarc', which is
 > just `defmacro' with `:not-custom-var' set to `t'.
 >
 > The `defcustom' keywords etc. could have just been
 > added to `defvar' for optional use.  I defined a
 > separate macro just to not interfere with any
 > existing uses of `defvar'.
 >
 > In sum, this uncouples interactive customization
 > from the other features that Customize offers, in
 > particular, type-checking and persistence, and it
 > provides those features for non-option variables.
 >
 > The ability to type-check, provide `:set' and
 > `:initialize' trigger functions, automatically
 > `:require' libraries, add links to doc, associate
 > with one or more `:groups', etc.  These are useful
 > things to be able to do with at least some defvars,
 > not just with defcustoms.
 >
 > Similarly, the ability to persist non-option vars
 > in a user's custom file can be useful.  This alone
 > is a frequent question, to which the answer has
 > been `savehist-additional-variables', `desktop.el',
 > or Bookmark+ variable-list bookmarks.
 >
 > The patch also includes a macro `with-user-vars',
 > which temporarily lets a set of variables be
 > customizable.  That is, it lets you treat a
 > `defvarc' variable as if it were a `defcustom'
 > option.  So if you want, you can use the Customize
 > UI to change a defvarc's value, or define commands
 > that use (e.g. complete) `defvarc' variable names.

Thank you for the links and the explanation.  I'll take the time to read
them.

Just by reading your explanation, I don't think I understand if just by
using something like your description of 'defvarc' would solve the issue
of having to combine user's customizations with a package setting by
using two different variables.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Drew Adams]

>  > 2. Emacs should make available features I noted in
>  > previous posts, such as a function to consider a
>  > change to an option value (by program) to not be a
>  > change.  This lets code change a value but not
>  > have Customize consider that a change has been
>  > made - so the change won't be saved automatically
>  > or reported as having occurred.
> 
> I wonder what does this mean in terms of the
> possible states for a user option.  AFAICT,
> none of the current possible states (STANDARD,
> SAVED, CHANGED, SET) fit in your description.
>
> STANDARD: No, standard value and current value
>           don't match.
> SAVED:    No, that setting wasn't saved.
> CHANGED:  No, because you're saying that Custom
>           shouldn't consider the change.
> SET:      No, because the code needs to use
>           customize-set-variable, but that
>           wouldn't be Custom ignoring the change.

I use my library `cus-edit+.el', which changes
some of the `cus-edit.el' code.  See, e.g.,
function `custom-consider-variable-unchanged'.
Doc string:

  Consider this variable as being unchanged now.
  This does not save the current value; it just
  considers the value to be unchanged.  If no
  further changes are made to this variable, then
  after doing this, `customize-customize' will not
  display this variable, since it was considered
  unchanged.

The cus-edit+.el code is here:

http://www.emacswiki.org/emacs-en/download/cus-edit%2b.el

And here is some info about it:

http://www.emacswiki.org/CustomizingAndSaving#CustomizePlus

About considering some customized state to be
"unchanged", see this post to bug 19328 thread.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=19328#47

See this part of the post: "Dealing with Spurious
Changes, 3: Consider Unchanged."

This lets Customize know that the current values
(options or faces) are to be treated as if they
were saved, but without actually saving them to
your custom file.

That way, your custom file is not polluted with
things that you're not really concerned with, yet
you're not bothered by seeing such fictitious
changes show up each time you check for changes.

Unlike ignoring changes to certain preferences
(see below), and really saving current values,
`Consider Unchanged' isn't a persistent change.

You can use it any time to "reset" the change
counter for given preferences, so the current
change is considered the new base value (as if
it were saved), and any further changes you make
to them will then show up as changes, using
`customize-unsaved'.

The "ignoring changes" mentioned above is another
possibility (see "Dealing with Spurious Changes,
2: Ignore" in the same bug 19328 message).  It's
appropriate for some preferences that you might
change often and temporarily.  You can add such
options to a list, `customize-customized-ignore'.
The effect is to make `customize-unsaved' ignore
them.

I don't claim that what I did in `cus-edit+.el'
is the only or the best way to fix the problems
it addresses.  It might be a starting point for
someone with a better idea or understanding of
the custom code/behavior.

>  > 3. Emacs should also make available the ability for
>  > `defvar' (or a new macro) to use the features of
>  > `defcustom'.  In particular, this includes :set and
>  > :type, and it includes the ability to persist the
>  > value.
>  >
>  > The difference between this and `defcustom' is that
>  > uch variables aren't recognized by Emacs as user
>  > options.  Users can't use the Customize UI or
>  > `set-variable' with such vars.  `C-h v' provides no
>  > `customize' link, etc.
> 
> This sounds interesting.
> 
>  > I proposed #3 to emacs-devel@gnu.org back in 2009:
>  >
>  > https://lists.gnu.org/archive/html/emacs-devel/2009-10/msg00668.html
>  >
>  > I sent a patch for #3 as bug #27348 (closed by Lars,
>  > saying that no one would want such a feature):
>  >
>  > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27348
...
>  > In sum, this uncouples interactive customization
>  > from the other features that Customize offers, in
>  > particular, type-checking and persistence, and it
>  > provides those features for non-option variables.
>  >
>  > The ability to type-check, provide `:set' and
>  > `:initialize' trigger functions, automatically
>  > `:require' libraries, add links to doc, associate
>  > with one or more `:groups', etc.  These are useful
>  > things to be able to do with at least some defvars,
>  > not just with defcustoms.
>  >
>  > Similarly, the ability to persist non-option vars
>  > in a user's custom file can be useful.  This alone
>  > is a frequent question, to which the answer has
>  > been `savehist-additional-variables', `desktop.el',
>  > or Bookmark+ variable-list bookmarks.
>  >
>  > The patch also includes a macro `with-user-vars',
>  > which temporarily lets a set of variables be
>  > customizable.  That is, it lets you treat a
>  > `defvarc' variable as if it were a `defcustom'
>  > option.  So if you want, you can use the Customize
>  > UI to change a defvarc's value, or define commands
>  > that use (e.g. complete) `defvarc' variable names.
> 
> Thank you for the links and the explanation.  I'll take the time to read
> them.
> 
> Just by reading your explanation, I don't think I understand if just by
> using something like your description of 'defvarc' would solve the issue
> of having to combine user's customizations with a package setting by
> using two different variables.

I don't know what particular issues it might solve.
Its point is just to separate the ability to have
`defcustom' features (type-checking, initialization,
persistence, etc.) without interfering with a user's
use of the Customize UI etc.

HTH.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

On 28/10/23 23:20, Drew Adams wrote:
 >>  > 2. Emacs should make available features I noted in
 >>  > previous posts, such as a function to consider a
 >>  > change to an option value (by program) to not be a
 >>  > change.  This lets code change a value but not
 >>  > have Customize consider that a change has been
 >>  > made - so the change won't be saved automatically
 >>  > or reported as having occurred.
 >>
 >> I wonder what does this mean in terms of the
 >> possible states for a user option.  AFAICT,
 >> none of the current possible states (STANDARD,
 >> SAVED, CHANGED, SET) fit in your description.
 >>
 >> STANDARD: No, standard value and current value
 >>           don't match.
 >> SAVED:    No, that setting wasn't saved.
 >> CHANGED:  No, because you're saying that Custom
 >>           shouldn't consider the change.
 >> SET:      No, because the code needs to use
 >>           customize-set-variable, but that
 >>           wouldn't be Custom ignoring the change.
 >
 > I use my library `cus-edit+.el', which changes
 > some of the `cus-edit.el' code.  See, e.g.,
 > function `custom-consider-variable-unchanged'.
 > Doc string:
 >
 >   Consider this variable as being unchanged now.
 >   This does not save the current value; it just
 >   considers the value to be unchanged.  If no
 >   further changes are made to this variable, then
 >   after doing this, `customize-customize' will not
 >   display this variable, since it was considered
 >   unchanged.

'customize-customized', right?

 > The cus-edit+.el code is here:
 >
 > http://www.emacswiki.org/emacs-en/download/cus-edit%2b.el

I tried to visit this link with eww, but got 404 Not Found.

 > And here is some info about it:
 >
 > http://www.emacswiki.org/CustomizingAndSaving#CustomizePlus

Same here.

Anyway, I read custom-consider-variable-unchanged, and it looks to me
that it would have the same problem when saving an unrelated variable,
because of the way custom-save-all works.  I want to work on improving
the saving mechanism so that problem disappears, but ISTM that this
bug report is about a different bug: a misuse of custom-set-variables.

 > About considering some customized state to be
 > "unchanged", see this post to bug 19328 thread.
 >
 > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=19328#47

Thank you.  I read the "Dealing with Spurious Changes" section from
cus-edit+.el.  Part 2 and Part 3 sound interesting to me.  It seems to
me that adding an ignore mechanism would be useful, in general.

 > I don't claim that what I did in `cus-edit+.el'
 > is the only or the best way to fix the problems
 > it addresses.  It might be a starting point for
 > someone with a better idea or understanding of
 > the custom code/behavior.

It's certainly useful to me to know about your approach.

I guess at this point I need to hear from Michael to understand better
about this 2 defcustoms in particular, and what are the expectations
after modifying them.  I certainly wish the code goes back to using
customize-set-variable or something similar, rather than
custom-set-variables.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Michael Albinus]

Mauro Aranda <maurooaranda@gmail.com> writes:

Hi Mauro,

> I guess at this point I need to hear from Michael to understand better
> about this 2 defcustoms in particular, and what are the expectations
> after modifying them.  I certainly wish the code goes back to using
> customize-set-variable or something similar, rather than
> custom-set-variables.

I'm neutral to whatever is used. But applying customize-set-variable
as-it-is has the known drawbacks. If it is fixed in
customize-set-variable, or if there is another function, we could use it
for connection-local variables.

However, I don't work in the Customize area myself. Don't expect patches
from me.

Best regards, Michael.

bug#63891: 29.0.91; customize-save-variable should not save all variables if a custom file exists

[Mauro Aranda]

On 29/10/23 08:07, Michael Albinus wrote:
 > Mauro Aranda <maurooaranda@gmail.com> writes:
 >
 > Hi Mauro,
 >
 >> I guess at this point I need to hear from Michael to understand better
 >> about this 2 defcustoms in particular, and what are the expectations
 >> after modifying them.  I certainly wish the code goes back to using
 >> customize-set-variable or something similar, rather than
 >> custom-set-variables.
 >
 > I'm neutral to whatever is used. But applying customize-set-variable
 > as-it-is has the known drawbacks. If it is fixed in
 > customize-set-variable, or if there is another function, we could use it
 > for connection-local variables.

Yes, but that is because packages modify the user option without really
asking the user, as I've said in previous posts.  First thing to check
is if there's no way around doing that.

I see in Git history that not telling Custom about the change was used
before.  So I guess going back to that is not an option.

That leaves the alternative of using something like
customize-set-variable, but that it also tells Custom that it should
ignore the change, to not consider the option set by the user.

 > However, I don't work in the Customize area myself. Don't expect patches
 > from me.

That's OK.  I'll work on the Customize code to solve this issue, once I
figure out what's needed.