bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

Admittedly, this is probably not a problem that others will run into
often.  But it is quite annoying for me.

I think the problem was introduced in Emacs 27, but I'm not positive.

This is the problem:

If I update a user option and then save the new value, it is saved to my
`custom-file', as usual.  But thereafter my `custom-file' cannot be
loaded by Emacs 20, because of these two entries that have been updated:

 '(tramp-default-method "ftp" nil (tramp))
 '(tramp-verbose 9 nil (tramp))

I did not modify or ask to save those two options.  Their values remain
the same as they were.  What has happened is that Emacs has now added
"nil (tramp)", and those additional elements in the list make Emacs 20
choke, because library `tramp' does not exist in Emacs 20.

Signaling: (file-error "Cannot open load file" "tramp")
  require(tramp)
  mapcar(require (tramp))
  custom-set-variables(... (tramp-debug-buffer t)
                           (tramp-default-method "ftp" nil (tramp))
                           (tramp-verbose 9 nil (tramp)) ...)

But why must Emacs now add that "nil (tramp)" to my custom settings?
Must defining the option value require the library?

I have no other case of an option where such a require argument is
inserted.  And I need not use Tramp at all, for these options to get
updated this way.

Should Emacs (e.g. 27) be doing that systematically?  If it should, then
what's the best way for me to prevent it from doing that, for my use
case?

(In fact, in Emacs 20, the doc string for `custom-set-variables' does
not even mention the possibility of an entry having a 4th argument,
REQUEST, but the code for that function does handle it, as
`custom-requests'.)

In GNU Emacs 27.0.50 (build 3, x86_64-w64-mingw32)
 of 2018-03-21
Repository revision: e70d0c9e66d7a8609450b2889869d16aeb0363b5
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install -C 'CFLAGS=-O2 -static -g3''

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

> Admittedly, this is probably not a problem that others will run into
> often.  But it is quite annoying for me.
>
> I think the problem was introduced in Emacs 27, but I'm not positive.

I suspect it is also in Emacs 26.1, see below.

> This is the problem:
>
> If I update a user option and then save the new value, it is saved to my
> `custom-file', as usual.  But thereafter my `custom-file' cannot be
> loaded by Emacs 20, because of these two entries that have been updated:
>
>  '(tramp-default-method "ftp" nil (tramp))
>  '(tramp-verbose 9 nil (tramp))

In Tramp 2.3 (integrated in Emacs 26.1), the defcustoms have changed:

--8<---------------cut here---------------start------------->8---
(defcustom tramp-default-method
  ...
  :group 'tramp
  :type 'string
  :require 'tramp) ;; new in Tramp 2.3.
--8<---------------cut here---------------end--------------->8---

The additional arguments are due to the ":require 'tramp", which was
added that time.

> But why must Emacs now add that "nil (tramp)" to my custom settings?
> Must defining the option value require the library?

IIRC, there was a reason for this. Don't remember the details :-(

> Should Emacs (e.g. 27) be doing that systematically?  If it should, then
> what's the best way for me to prevent it from doing that, for my use
> case?

Maybe you put at the beginning of your custom file

--8<---------------cut here---------------start------------->8---
;; This is for Emacs < 22
(or (locate-library "tramp")
    (provide 'tramp))
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> Maybe you put at the beginning of your custom file
> (or (locate-library "tramp") (provide 'tramp))

Yes, I guess that would work and is better than nothing,
but it seems wrong to tell Emacs that something has
been provided that has not. ;-)

Worse than style or ugliness, though, is that there
might be code here or there that tests
(featurep 'tramp) and acts accordingly.  That code
surely will be led astray by such a hack.

I'd like to find a better way.

Seems like there should be a way for Tramp and
Customize to collaborate to produce a conditional
sexp that does not require tramp unconditionally.
E.g., one that tests (at time of evaluating
`custom-set-variables') whether Tramp is even
included in the Emacs version.

Or maybe somehow just use a soft-require - the
equivalent of (require 'tramp nil t)?  Would that
work for Tramp, if it were possible for
`custom-set-variables'?

Seems like something is missing from Emacs for
such cases.  A library wants to ensure that it is
loaded before evaluating a sexp that sets one of
its user options.  But if loading the library is
impossible then setting the variable is benign,
so a soft-require is all that is really needed.

(Or am I missing something?)

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> Maybe you put at the beginning of your custom file
>> (or (locate-library "tramp") (provide 'tramp))
>
> Yes, I guess that would work and is better than nothing,
> but it seems wrong to tell Emacs that something has
> been provided that has not. ;-)
>
> Worse than style or ugliness, though, is that there
> might be code here or there that tests
> (featurep 'tramp) and acts accordingly.  That code
> surely will be led astray by such a hack.

We're spaeking about Emacs < 22. How many such Emacsen are in use today,
which also use packages checking for Tramp?

> Seems like there should be a way for Tramp and
> Customize to collaborate to produce a conditional
> sexp that does not require tramp unconditionally.
> E.g., one that tests (at time of evaluating
> `custom-set-variables') whether Tramp is even
> included in the Emacs version.
>
> Or maybe somehow just use a soft-require - the
> equivalent of (require 'tramp nil t)?  Would that
> work for Tramp, if it were possible for
> `custom-set-variables'?
>
> Seems like something is missing from Emacs for
> such cases.  A library wants to ensure that it is
> loaded before evaluating a sexp that sets one of
> its user options.  But if loading the library is
> impossible then setting the variable is benign,
> so a soft-require is all that is really needed.

If there were such a possibility I would consider it. But it must
already exist for Emacsen < 22; I doubt we will change them if it's not
possible yet.

And given the low likelihood plus the existing workaround, I believe it
is not worth to spend time for this. Sorry.

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> >> Maybe you put at the beginning of your custom file
> >> (or (locate-library "tramp") (provide 'tramp))
> >
> > Yes, I guess that would work and is better than nothing,
> > but it seems wrong to tell Emacs that something has
> > been provided that has not. ;-)
> >
> > Worse than style or ugliness, though, is that there
> > might be code here or there that tests
> > (featurep 'tramp) and acts accordingly.  That code
> > surely will be led astray by such a hack.
> 
> We're spaeking about Emacs < 22. How many such Emacsen are in use today,
> which also use packages checking for Tramp?

Yes, I said at the outset:

  Admittedly, this is probably not a problem that others
  will run into often.  But it is quite annoying for me.

I end up trying to remember to do this immediately after
saving any (!) user option in Emacs 26+: Open my
`custom-file' and delete the offending parts of the sexp.
Otherwise, when I later open Emacs 20 I get the error.

But when you say "which also use packages checking for
Tramp", what it really comes down to is being able to
use the same `custom-file' (which has only stuff that is
inserted automatically by Emacs) with multiple Emacs
releases (in this case including Emacs 20).

> > Seems like there should be a way for Tramp and
> > Customize to collaborate to produce a conditional
> > sexp that does not require tramp unconditionally.
> > E.g., one that tests (at time of evaluating
> > `custom-set-variables') whether Tramp is even
> > included in the Emacs version.
> >
> > Or maybe somehow just use a soft-require - the
> > equivalent of (require 'tramp nil t)?  Would that
> > work for Tramp, if it were possible for
> > `custom-set-variables'?
> >
> > Seems like something is missing from Emacs for
> > such cases.  A library wants to ensure that it is
> > loaded before evaluating a sexp that sets one of
> > its user options.  But if loading the library is
> > impossible then setting the variable is benign,
> > so a soft-require is all that is really needed.
> 
> If there were such a possibility I would consider it.

Can we perhaps consider this bug report as a request
for such an enhancement?

> But it must already exist for Emacsen < 22;

No, I don't think so.  (But I'm not sure what you mean.)
It would be enough that such a mechanism exists for
Emacs 26+ or whatever.

> I doubt we will change them if it's not possible yet.

Isn't it already possible to DTRT here (use a soft-require
instead of a hard-require)?

Can't Tramp, for example, just use a :set, or an :initialize,
function that does a soft-require, instead of using :require,
which puts a hard-require into `custom-file'?

Something like this, for example:

(defcustom tramp-verbose 3
  "..."
  :group 'tramp
  :type 'integer
  :set (lambda (sym val)
         (when (require 'tramp nil t)
           (custom-set-default sym val)))
  :initialize 'custom-initialize-set)

> And given the low likelihood plus the existing workaround,
> I believe it is not worth to spend time for this. Sorry.

I think the defcustom above would fit the bill, and it
would require no time to implement.  It just sets the
variable normally, but only if Tramp can first be loaded.

It does nothing is Tramp is not available.
(boundp 'tramp-verbose) returns nil in that case.

(Am I missing something?)

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

> But when you say "which also use packages checking for
> Tramp", what it really comes down to is being able to
> use the same `custom-file' (which has only stuff that is
> inserted automatically by Emacs) with multiple Emacs
> releases (in this case including Emacs 20).

Yes. But again, we're speaking about "multiple Emacs releases" including
releases prior 22. And I don't believe this is used so much these days.
The recent Tramp itself is backward compatible up to Emacs 24 only.

>> If there were such a possibility I would consider it.
>
> Can we perhaps consider this bug report as a request
> for such an enhancement?

Sure. I keep it open, at least as long we're discussing.

>> But it must already exist for Emacsen < 22;
>
> No, I don't think so.  (But I'm not sure what you mean.)

I mean that if we need another mechanism but ":require 'tramp" in the
defcustom, that mechanism must exist also in such old releases.

> Something like this, for example:
>
> (defcustom tramp-verbose 3
>   "..."
>   :group 'tramp
>   :type 'integer
>   :set (lambda (sym val)
>          (when (require 'tramp nil t)
>            (custom-set-default sym val)))
>   :initialize 'custom-initialize-set)

This results in 

--8<---------------cut here---------------start------------->8---
Loading tramp...
Recursive load: "/net/ford/albinus/src/tramp/lisp/tramp.elc", "/net/ford/albinus/src/tramp/lisp/tramp.elc", "/net/ford/albinus/src/tramp/lisp/tramp.elc", "/net/ford/albinus/src/tramp/lisp/tramp.elc", "/net/ford/albinus/src/tramp/lisp/tramp.elc"
--8<---------------cut here---------------end--------------->8---

> I think the defcustom above would fit the bill, and it
> would require no time to implement.  It just sets the
> variable normally, but only if Tramp can first be loaded.
>
> It does nothing is Tramp is not available.
> (boundp 'tramp-verbose) returns nil in that case.
>
> (Am I missing something?)

A working setup? :-)

Well, maybe I try something like this next days, as time permits. And
I'm always willing to test your proposals.

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> > (defcustom tramp-verbose 3
> >   "..."
> >   :group 'tramp
> >   :type 'integer
> >   :set (lambda (sym val)
> >          (when (require 'tramp nil t)
> >            (custom-set-default sym val)))
> >   :initialize 'custom-initialize-set)
> 
> This results in
> 
> Loading tramp...
> Recursive load: "/net/ford/albinus/src/tramp/lisp/tramp.elc",
> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
> "/net/ford/albinus/src/tramp/lisp/tramp.elc"

Hm, indeed.  Maybe :intialize needs to not try to set it here
(i.e., during loading).

Maybe :initialize needs to test (featurep 'tramp) or something?

Or maybe some other value of :initialize is appropriate?
`custom-initialize-default'? 
`custom-initialize-safe-set'?
`custom-initialize-safe-default'?

Dunno.  I'm no expert on loading and defcustom initialization.

> Well, maybe I try something like this next days, as time permits. And
> I'm always willing to test your proposals.

Thanks for taking a look at this.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> > (defcustom tramp-verbose 3
>> >   "..."
>> >   :group 'tramp
>> >   :type 'integer
>> >   :set (lambda (sym val)
>> >          (when (require 'tramp nil t)
>> >            (custom-set-default sym val)))
>> >   :initialize 'custom-initialize-set)
>> 
>> This results in
>> 
>> Loading tramp...
>> Recursive load: "/net/ford/albinus/src/tramp/lisp/tramp.elc",
>> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
>> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
>> "/net/ford/albinus/src/tramp/lisp/tramp.elc",
>> "/net/ford/albinus/src/tramp/lisp/tramp.elc"
>
> Hm, indeed.  Maybe :intialize needs to not try to set it here
> (i.e., during loading).
>
> Maybe :initialize needs to test (featurep 'tramp) or something?
>
> Or maybe some other value of :initialize is appropriate?
> `custom-initialize-default'? 
> `custom-initialize-safe-set'?
> `custom-initialize-safe-default'?
>
> Dunno.  I'm no expert on loading and defcustom initialization.

I will play along those lines:

--8<---------------cut here---------------start------------->8---
  :set (lambda (sym val)
	 (eval-after-load 'tramp
           '(custom-set-default sym val)))
  :initialize 'custom-initialize-set)
--8<---------------cut here---------------end--------------->8---

Tomorrow. Tonight, I'm running out of time.

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> I will play along those lines:
> 
>   :set (lambda (sym val)
> 	    (eval-after-load 'tramp '(custom-set-default sym val)))
>   :initialize 'custom-initialize-set)
> 
> Tomorrow. Tonight, I'm running out of time.

Yes, maybe that will do the job.  Thanks for trying.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> Tomorrow. Tonight, I'm running out of time.
>
> Yes, maybe that will do the job.  Thanks for trying.

I've checked all the defcustoms in Tramp. Looks, like only some few need
the :require attribute in their spec, likely I have added this attribute
too eagerly to *all* defcustoms ('cos "it doesn't hurt"). I have removed
it from the majority of defcustoms, lets see how it works.

Pushed to master.

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> I've checked all the defcustoms in Tramp. Looks, like only some few need
> the :require attribute in their spec, likely I have added this attribute
> too eagerly to *all* defcustoms ('cos "it doesn't hurt"). I have removed
> it from the majority of defcustoms, lets see how it works.
> 
> Pushed to master.

Thanks for doing this, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> I've checked all the defcustoms in Tramp. Looks, like only some few need
>> the :require attribute in their spec, likely I have added this attribute
>> too eagerly to *all* defcustoms ('cos "it doesn't hurt"). I have removed
>> it from the majority of defcustoms, lets see how it works.
>> 
>> Pushed to master.
>
> Thanks for doing this, Michael.

Will you check next time? Or shall I close the bug, and you'll reopen if
there are still problems?

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> >> Pushed to master.
> > Thanks for doing this, Michael.
> 
> Will you check next time? Or shall I close the bug, and you'll reopen if
> there are still problems?

Will this fix be in Emacs 26?

In any case, I won't have a way to check it until I have access to
a Windows binary that includes the fix.

I think you can close this bug now.  If I see a problem later then
I can ask you to reopen it.

Thx.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Version: 27.1

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

> Will this fix be in Emacs 26?

No. Emacs 26.1 is in code freeze. But the fix will be in Tramp 2.4.0,
planned for release end of June.

> In any case, I won't have a way to check it until I have access to
> a Windows binary that includes the fix.
>
> I think you can close this bug now.  If I see a problem later then
> I can ask you to reopen it.

Closed. If needed, you could remind me to backport this to Emacs 26.2.

> Thx.

Best regards, Michael.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Drew Adams]

> Closed. If needed, you could remind me to backport this to Emacs 26.2.

Yes, I think it's important to backport it, if possible.
Please do that.

The problem didn't exist before Emacs 26 (which isn't
released yet).  If it gets fixed for 26.2 and later then
no one will ever encounter it, most likely.

bug#31558: 27.0; `custom-file' settings messed up by Emacs 27

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>> Closed. If needed, you could remind me to backport this to Emacs 26.2.
>
> Yes, I think it's important to backport it, if possible.
> Please do that.
>
> The problem didn't exist before Emacs 26 (which isn't
> released yet).  If it gets fixed for 26.2 and later then
> no one will ever encounter it, most likely.

Backported to the emacs-26 branch.