bug#9712: 24.0.50; doc about byte-compiling `defcustom'

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

Put this in a file foo.el, and byte-compile the file:
 
(defcustom titi (eval-when-compile `(,(kbd "S-<tab>")))
  "jjjjjjjjjjjj"
  :type '(repeat sexp) :group 'edit)
 
(defvar toto (eval-when-compile `(,(kbd "S-<tab>")))
 "kkkkkkkkkk")
 
This is what the byte-compiled file shows:
 
#@14 jjjjjjjjjjjj\x1f
(custom-declare-variable 'titi '(eval-when-compile
  `(,(kbd "S-<tab>"))) '(#$ . 509) :type '(repeat sexp) :group 'edit)
#@12 kkkkkkkkkk\x1f
(defvar toto '([S-tab]) (#$ . 647))
 
Here's the bug (a doc bug), as I see it: Where in the doc is this
behavior described?  Where does it say that the STANDARD arg to
defcustom is not evaluated by the byte compiler, even when you use
`eval-when-compile'?
 
(elisp) Variable Definitions says this, but it's about all I can find:
 
"Evaluating the `defcustom' form evaluates STANDARD, but does not
necessarily install the standard value."
 
And of course the doc for `eval-when-compile', (elisp) `Eval During
Compile', says "The result of evaluation by the compiler becomes a
constant which appears in the compiled program."
 
I see it documented nowhere that the STANDARD arg to defcustom appears
unevaluated in byte-compiled code.
 
 
 
In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2011-09-19 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.5) --no-opt'
 

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

Is this maybe more than a doc bug?  In bytecomp.el there is this:

(defun byte-compile-file-form-custom-declare-variable (form)
  (when (byte-compile-warning-enabled-p 'callargs)
    (byte-compile-nogroup-warn form))
  (push (nth 1 (nth 1 form)) byte-compile-bound-variables)
  ;; Don't compile the expression because it may be displayed to the user.
  ;; (when (eq (car-safe (nth 2 form)) 'quote)
  ;;   ;; (nth 2 form) is meant to evaluate to an expression, so if we have the
  ;;   ;; final value already, we can byte-compile it.
  ;;   (setcar (cdr (nth 2 form))
  ;;           (byte-compile-top-level (cadr (nth 2 form)) nil 'file)))

Seems like this non-evaluation is intended, but sounds like the designer is not
sure that's a good idea (?).  What does "because it may be displayed to the
user" mean?  Is the idea that the user would see the byte-compiled value instead
of the code that evaluates to that value?  If so, why is that bad?

At any rate, there is at least a doc bug.  In particular, someone who wraps the
`defcustom' VALUE in `eval-when-compile' will mistakenly expect the evaluation
result to be placed into the byte code as a constant.  S?he will not expect that
VALUE is not evaluated at byte-compile time, even when inside
`eval-when-compile'.

A consequence of the current behavior is that you really cannot use macro calls
in VALUE, unless you are sure that the macro definition will be available also
at load time (i.e., byte-compilation time is not enough).  I ran into this using
macro `kbd' in VALUE.

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

> A consequence of the current behavior is that you really 
> cannot use macro calls in VALUE, unless you are sure that
> the macro definition will be available also
> at load time (i.e., byte-compilation time is not enough).  I 
> ran into this using macro `kbd' in VALUE.

To clarify: using macro `icicle-kbd', which is similar to `kbd'.  The problem
presumably would not occur for `kbd' itself because it is in `subr.el', which is
always loaded.

E.g., put this in file throw-1.el:
(defcustom foo (eval-when-compile (toto 4))
  "jjjj" :type 'integer :group 'edit)

Put this in file throw-2.el, and evaluate it to define `toto':
(defmacro toto (n) `,n)

Byte-compile throw-1.el.

emacs -Q
M-x load-file throw-1.elc

Error: Symbol's function definition is void: toto.

Bummer.  I would think that byte-compiling would expand and evaluate macros, so
that the file defining the macros would not be needed at load/runtime if the
byte-compiled file is available.  I would expect that to be the case even
without the `eval-when-compile' (which is a specific directive to the
byte-compiler to eval and replace with the value).

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

If, as the comment in the code suggests, the aim in commenting out the code
(that would otherwise let the byte-compiler eval the defcustom value and assign
the result as a constant in the compiled code) is to avoid having users see the
byte-compiled value in Customize buffers, then why not solve both problems
instead of choosing one or the other?

Byte-compiling should, as is everywhere claimed (e.g., the doc), replace the
VALUE with its evaluation, as a constant, in the byte-compiled code.  But
nothing prevents saving the Lisp sexp that evaluates to that constant.  And
nothing then prevents making that saved sexp available to the user in Customize.
We could even let the user see both: the value and the sexp that eval'd to it.

And most evaluation of a defcustom value leads to a normal Lisp object that
users are used to seeing anyway (string, number,...).

Was there some other reason, besides not having the user see the sexp's value,
for not letting the byte-compiler treat the defcustom VALUE normally?

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Stefan Monnier]

retitle 9712 defcustom's initial value expression is not byte-compiled
thanks

> This is what the byte-compiled file shows:
 
> #@14 jjjjjjjjjjjj\x1f
> (custom-declare-variable 'titi '(eval-when-compile
>   `(,(kbd "S-<tab>"))) '(#$ . 509) :type '(repeat sexp) :group 'edit)
> #@12 kkkkkkkkkk\x1f
> (defvar toto '([S-tab]) (#$ . 647))
 
> Here's the bug (a doc bug), as I see it: Where in the doc is this
> behavior described?  Where does it say that the STANDARD arg to
> defcustom is not evaluated by the byte compiler, even when you use
> `eval-when-compile'?
 
The bug is not in the doc but in the behavior.


        Stefan

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

> The bug is not in the doc but in the behavior.

That was my feeling, as well.  But I did not expect agreement, based on the
commented code.

The bug has been there for multiple major Emacs releases now - a decade or so,
and appears to have been introduced intentionally.  If there is no plan to fix
it soon, perhaps the info that is in the comments should at least be conveyed to
users.

Alternatively, the code could be uncommented and byte-compiling be let free to
do what is advertised that it does.  The downside would be showing users the
constant values in Customize, but that seems like less of a disfunction than
what we have now.

As it stands now, the behavior is stable (many years), but it is not what is
intended and it does not correspond to the behavior implied by the doc.

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Stefan Monnier]

>> The bug is not in the doc but in the behavior.
> That was my feeling, as well.  But I did not expect agreement, based on the
> commented code.

> The bug has been there for multiple major Emacs releases now -
> a decade or so, and appears to have been introduced intentionally.
> If there is no plan to fix it soon, perhaps the info that is in the
> comments should at least be conveyed to users.

The introduction of lexical-binding makes this issue more serious
because this code does not obey the lexical-binding variable (and
escapes the sanity checks of cconv.el, hence occasionally hiding
problems when converting to lexical-binding).

> Alternatively, the code could be uncommented and byte-compiling be let free to
> do what is advertised that it does.  The downside would be showing users the
> constant values in Customize, but that seems like less of a disfunction than
> what we have now.

We need to solve the display issue some other way, yes.


        Stefan

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

> > The bug has been there for multiple major Emacs releases now -
> > a decade or so, and appears to have been introduced intentionally.
> > If there is no plan to fix it soon, perhaps the info that is in the
> > comments should at least be conveyed to users.
> 
> The introduction of lexical-binding makes this issue more serious
> because this code does not obey the lexical-binding variable (and
> escapes the sanity checks of cconv.el, hence occasionally hiding
> problems when converting to lexical-binding).

So will this be fixed in Emacs 24.1 then?  +1

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Stefan Monnier]

>> > The bug has been there for multiple major Emacs releases now -
>> > a decade or so, and appears to have been introduced intentionally.
>> > If there is no plan to fix it soon, perhaps the info that is in the
>> > comments should at least be conveyed to users.
>> The introduction of lexical-binding makes this issue more serious
>> because this code does not obey the lexical-binding variable (and
>> escapes the sanity checks of cconv.el, hence occasionally hiding
>> problems when converting to lexical-binding).
> So will this be fixed in Emacs 24.1 then?  +1

No.


        Stefan

bug#9712: 24.0.50; doc about byte-compiling `defcustom'

[Drew Adams]

> >> > The bug has been there for multiple major Emacs releases
> >> > now - a decade or so, and appears to have been introduced 
> >> > intentionally.  If there is no plan to fix it soon,
> >> > perhaps the info that is in the comments should at least
> >> > be conveyed to users.
> >> >
> >> The introduction of lexical-binding makes this issue more serious
> >> because this code does not obey the lexical-binding variable (and
> >> escapes the sanity checks of cconv.el, hence occasionally hiding
> >> problems when converting to lexical-binding).
> >
> > So will this be fixed in Emacs 24.1 then?  +1
> 
> No.

Why not, especially since 24.1 introduces generalized lexical binding, which
"makes this issue more serious"?  What's the hurry to shove 24.1 out the door?

And it seems like if you insist on only going for halfway measures in 24.1 then
it would be better for users to see the constant value in Customize (no big
deal) that it is for byte-compilation not to work properly and for the doc to be
contradicted in several places (defcustom, eval-when-compile,...).