`read--expression' and `read-minibuffer'

`read--expression' and `read-minibuffer'

[Drew Adams]

Function `read--expression' was added in Emacs 24.4.

Can someone please explain why this is an "internal" function
(two hyphens in the name)?  Why wouldn't users want to use
this to read a Lisp expression from the minibuffer?

It seems like `read--expression' should now be the usual way
to read and return a Lisp expression, and that it should NOT
be considered "internal".  No?

On the other hand, we still also have the longstanding
(non-internal) function `read-minibuffer' for reading and
returning a Lisp "object".

It's not clear to me how a Lisp object differs from a Lisp
sexp.  (And the doc of `interactive' even calls what "x"
reads a "Lisp expression", not a "Lisp object".)

Well, sure, there are Lisp objects (e.g. byte-compiled
functions) that are not readable expressions.  But those are
not readable anyway, AFAIK.  `read-minibuffer', like
`read--expression', just reads expressions, AFAICT.

The difference between `read-minibuffer' and `read--expression'
is this: The former uses `minibuffer-local-map', so it has no
special handling of Lisp expressions and no completion.  The
latter uses `read-expression-map', and it provides Lisp-sexp
completion.

Why isn't `read--expression' treated as a user function, like
`read-minibuffer', instead of being considered "internal"?

And `read-minibuffer' is still used for `interactive' with
spec "x".  Why isn't `read--expression' used for that now?

How about promoting `read--expression' to `read-expression'
and using it for `(interactive "x")'?  Nothing wrong with
also keeping `read-minibuffer' (though it's name is not great),
but why continue to use it in preference to `read--expression'?

`read-minibuffer' is still used in several places, but I
don't see why those places wouldn't benefit from using
`read--expression' instead.

Am I missing something?  I ask because I read Lisp sexps
from the minibuffer, and I don't (yet) see any reason to
use `read-minibuffer' instead of "internal" function
`read--expression'.  I'd like to learn which to use and why.
Thx.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> Well, sure, there are Lisp objects (e.g. byte-compiled
> functions) that are not readable expressions.  But those are
> not readable anyway, AFAIK.

Lists of symbols are Lisp object, are perfectly readable, but shouldn't
use `read--expression'.  `read--expression' is to read the subset of
S-exps which represent an Elisp expression (something that would be
meaningful to pass to `eval', for example).

> Why isn't `read--expression' treated as a user function, like
> `read-minibuffer', instead of being considered "internal"?

Can't remember the details, but I think it's because it's easier to make
it "external" than the reverse.

> And `read-minibuffer' is still used for `interactive' with
> spec "x".  Why isn't `read--expression' used for that now?

Not sure either.  Probably just a conservative choice.


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> > Well, sure, there are Lisp objects (e.g. byte-compiled
> > functions) that are not readable expressions.  But those are
> > not readable anyway, AFAIK.
> 
> Lists of symbols are Lisp object, are perfectly readable, but shouldn't
> use `read--expression'.

I don't see why not.

> `read--expression' is to read the subset of
> S-exps which represent an Elisp expression (something that would be
> meaningful to pass to `eval', for example).

I don't see why you think that what it reads is limited, or could
or should be limited, to expressions that can be eval'd.

(read--expression "Expression: ")
(1 2 frou frou)

=> (1 2 frou frou)

> > Why isn't `read--expression' treated as a user function, like
> > `read-minibuffer', instead of being considered "internal"?
> 
> Can't remember the details, but I think it's because it's easier to make
> it "external" than the reverse.

How about making it external now?  It is a useful general-purpose
function.

> > And `read-minibuffer' is still used for `interactive' with
> > spec "x".  Why isn't `read--expression' used for that now?
> 
> Not sure either.  Probably just a conservative choice.

Maybe consider it?  I don't see any risk.  The main difference
is that it uses a more useful keymap for reading expressions.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> I don't see why you think that what it reads is limited, or could
> or should be limited, to expressions that can be eval'd.

Because it is tailored for that use case.  The kind of completion it
provides for instance assumes the S-exp is an Elisp expression.


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> > I don't see why you think that what it reads is limited, or
> > could or should be limited, to expressions that can be eval'd.
> 
> Because it is tailored for that use case.  The kind of completion
> it provides for instance assumes the S-exp is an Elisp expression.

Of course it is an Elisp expression.  It does not follow that
the Elisp expression that has been read will then be _evaluated_.

A list such as (2 bookmark-alist) or (forward-char forward-line)
is an Elisp exression, and the function and variable names in
these can be constructed using completion.  Neither list is
one that you would then try to evaluate.

And even for what you might consider a more typical case, such
as expression (previous-single-property-change (point) 'foo),
there is nothing that obliges the expression to be evaluated.

Completion can be used and useful when constructing an Elisp
expression to be read, regardless of whether the expression
that is read will be evaluated, or can be evaluated without
error.

What a Lisp program does with a Lisp expression is up to that
program.  Zillions of actions are possible, other than just
evaluation.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> What a Lisp program does with a Lisp expression is up to that
> program.  Zillions of actions are possible, other than just
> evaluation.

Great.


        Stefan

Re: `read--expression' and `read-minibuffer'

[Davis Herring]

>> Because it is tailored for that use case.  The kind of completion
>> it provides for instance assumes the S-exp is an Elisp expression.
>
> Of course it is an Elisp expression.  It does not follow that
> the Elisp expression that has been read will then be _evaluated_.

In this thread, Stefan is defining an "Elisp expression" to be precisely 
an S-exp that is intended for `eval'.

> Completion can be used and useful when constructing an Elisp
> expression to be read, regardless of whether the expression
> that is read will be evaluated, or can be evaluated without
> error.

So a variant of `read--expression' could be written that does a subset 
of that completion appropriate to the more general context of entering 
an S-exp.  It might for instance be a flag passed to `read--expression'. 
  This answers your original question: it's internal because we are 
still discovering the use cases and desirable interface for it.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> >> Because it is tailored for that use case.  The kind of completion
> >> it provides for instance assumes the S-exp is an Elisp expression.
> >
> > Of course it is an Elisp expression.  It does not follow that
> > the Elisp expression that has been read will then be _evaluated_.
> 
> In this thread, Stefan is defining an "Elisp expression" to be
> precisely an S-exp that is intended for `eval'.

He may be.  But that does not really correspond to the effect of
`read--expression'.  There is nothing in its behavior that limits
what is read to expressions that are intended to be evaluated.

> > Completion can be used and useful when constructing an Elisp
> > expression to be read, regardless of whether the expression
> > that is read will be evaluated, or can be evaluated without
> > error.
> 
> So a variant of `read--expression' could be written that does a subset
> of that completion appropriate to the more general context of entering
> an S-exp.  It might for instance be a flag passed to `read--expression'.

See above.  What is read and returned is an arbitrary sexp, not
one that is supposedly intended only for evaluation.

Perhaps you are referring to the fact that after `(' a symbol is
completed against only function names, instead of against both
function and variable names?

That's not a real problem, but that's the only place/case where I
can see an argument for a "variant" behavior (completion against
both function and variable names at that position too).

Or perhaps you are referring to the fact that `read--expression'
runs `eval-expression-minibuffer-setup-hook' functions at the
outset?

If you want to bother creating a variant for completing both
function and variable symbols, and for not running that
eval-intended hook, I don't have a problem with that.

The point is to get a reasonably behaving read function for
sexps that is not branded "internal".

The non-"internal" function `read-minibuffer' is not very
reasonable, and is essentially a vestige, at this point.

And there is little sense in otherwise asking users to roll
their own, e.g., using just `read-from-minibuffer' with
`read-expression-map' and `read-expression-history'.

We can do better, and better is already available:
`read--expression'.  Some such function should now be
available non-"internally".

If you want to (1) rename the current `read--expression'
to `read-expression-for-eval' and (2) create a variant
`read-expression' (which is for both eval and not),
please go ahead.

Or if you want to roll those together, distinguishing the
behaviors by an optional arg, please go ahead.

> This answers your original question: it's internal because we are
> still discovering the use cases and desirable interface for it.

1. That's not a good reason to make it "internal".  If you want
   to learn more about use cases and improving the interface then
   make it non-"internal".  Users are best placed to tell you
   about both of those things.

   The function has been available for almost 3 years "internally".
   Do you really think more time to just ponder it "internally" is
   what is called for, to "discover the use cases and desirable
   interface"?

2. That "reason" applies to ALL new functions.  It's hardly a
   solid reason.  It's not a reason why _this_ function should
   be "internal".

3. The same (#2) applies to the only other reason given so far
   (by Stefan), which was that it is easier to make an existing
   function become non-"internal" than it is to make it become
   "internal".  That "reason" too applies to ALL new functions.

What would _you_ choose, to read an arbitrary Lisp expression
in the minibuffer?  Would you really choose the non-"internal"
function `read-minibuffer'?  Not if you wanted to help the
users whose input was being read.  Or would you use just
`read-from-minibuffer', passing `read-expression-map' and
`read-expression-history'?

(`read-expression-map' and `read-expression-history' have been
around since the beginning (Emacs 20, anyway), for reading sexps.)

Time to plant this little one outside, in the wide world beyond
the hothouse/incubator, I'd say.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> That's not a real problem, but that's the only place/case where I
> can see an argument for a "variant" behavior (completion against
> both function and variable names at that position too).

For S-exps in general, we can't provide much completion, and we
certainly shouldn't provide completion against function names and
variable names.

An S-exp is like an XML tree.  An Elisp expression is like a XHTML tree
(i.e. it's a particular kind of XML that is expected to follow
a particular schema).  Yes, it's also an XML tree, but if all you know
is that you're typing an XML tree, providing any completion based on
XHTML identifiers is just bogus.

> The non-"internal" function `read-minibuffer' is not very
> reasonable, and is essentially a vestige, at this point.

Ah, so that's your complaint.  Why don't you make a bug-report about
that one, pointing out exactly what are its shortcomings.  But do
remember: it's a function meant for S-exps, i.e. random data represented
in a form suitable for the Lisp reader, with no assumption like an
expectation that it be related to Elisp code.

> We can do better, and better is already available:
> `read--expression'.  Some such function should now be
> available non-"internally".

Now I'm confused.  If you want a function to read an Elisp expression,
indeed read--expression might be your solution.  Then make a bug report
asking for it to be renamed.

> If you want to (1) rename the current `read--expression' to
> `read-expression-for-eval' and (2) create a variant `read-expression'
> (which is for both eval and not), please go ahead.

I have no idea what a "read-expression not for eval" would mean or look like.

`read--expression' already couldn't care less whether you'll pass the
result to `eval' or not.  All it cares about is that it provides useful
behavior (eldoc, completion, indentation) under the assumption that the
user is typing something that is (more or less) valid Elisp code, rather
than random data.

> 1. That's not a good reason to make it "internal".

I already said that it's internal for no particular reason.  So at this
point I'll just PLONK


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

I wasn't as clear as I should have been. 

> > That's not a real problem, but that's the only place/case where I
> > can see an argument for a "variant" behavior (completion against
> > both function and variable names at that position too).
> 
> For S-exps in general, we can't provide much completion, and we
> certainly shouldn't provide completion against function names and
> variable names.
> 
> An S-exp is like an XML tree.  An Elisp expression is like a XHTML tree
> (i.e. it's a particular kind of XML that is expected to follow
> a particular schema).  Yes, it's also an XML tree, but if all you know
> is that you're typing an XML tree, providing any completion based on
> XHTML identifiers is just bogus.

Your analogy is OK.  I wouldn't speak of an "Elisp expression" here,
since to me that just means any expression - sexp that you can
construct with Emacs Lisp.  But now I know what you mean.  It's not
about any intention to evaluate; your speaking of that was throwing
me off.

I was really asking only about getting the extra hyphen removed
from `read--expression'. ;-)  Use cases I have for it do involve
sometimes immediately evaluating the sexp read, but sometimes they
involve putting such sexps together to form larger sexps (which
sometimes might be evaluated at some point).

I don't have a problem with the completion that `read-expression'
provides, and in many such use cases the names to complete are
indeed names of program parts - in particular names of functions
and variables.  This, in spite of the fact that the sexps read
need not be evaluable without error.

For more general sexps - your XML (S-exp) level of generality,
there is completion against an obarray.  And in some intermediate
cases `dabbrev' completion (names from one or more buffers) can
be helpful.  When Elisp buffers are involved, it too can be about
reading bits of potential code.

In this thread I was really only talking about `read--expression'
and what it does.  What it does is useful also to ordinary Emacs
user-programmers, not just "internally".

> > The non-"internal" function `read-minibuffer' is not very
> > reasonable, and is essentially a vestige, at this point.
> 
> Ah, so that's your complaint.  Why don't you make a bug-report about
> that one, pointing out exactly what are its shortcomings.  But do
> remember: it's a function meant for S-exps, i.e. random data represented
> in a form suitable for the Lisp reader, with no assumption like an
> expectation that it be related to Elisp code.

See above.  Any Lisp sexp can be related to Elisp code.  Any data
can be a piece of some code.  But yes, I was interested here in
what `read--expression' does already.

> > We can do better, and better is already available:
> > `read--expression'.  Some such function should now be
> > available non-"internally".
> 
> Now I'm confused.  If you want a function to read an Elisp expression,
> indeed read--expression might be your solution.  Then make a bug report
> asking for it to be renamed.
> 
> > If you want to (1) rename the current `read--expression' to
> > `read-expression-for-eval' and (2) create a variant `read-expression'
> > (which is for both eval and not), please go ahead.
> 
> I have no idea what a "read-expression not for eval" would mean or look
> like.

It's what Davis was trying to distinguish `read--expression' as:
reading an expression "intended for evaluation" (paraphrasing).
Likewise, you.  You both tried to distinguish what it does as
reading an expression that might be evaluated (even though it can
also read expressions that cannot, without raising an error).

Creating variants of the behavior was his idea.  I'm fine with
the current behavior (which I don't associate directly with
evaluation, but which I recognize provides completion against
known functions and variables).

> `read--expression' already couldn't care less whether you'll
> pass the result to `eval' or not.

Thank you.  That's what I was saying.  It is you (and Davis) who
used "evaluation" to characterize it.

> All it cares about is that it provides useful
> behavior (eldoc, completion, indentation) under the assumption that the
> user is typing something that is (more or less) valid Elisp code, rather
> than random data.

Even that assumption is too strong (see above), but I don't know a
great way to characterize it.  I'd say that the completion against
existing function (after `(' ) and variable names sums it up, but
I don't have a name for that.  "Expression" doesn't say more than
"sexp", to me - it's just as general.

> > 1. That's not a good reason to make it "internal".
> 
> I already said that it's internal for no particular reason.

Not really.  You said that it's internal because it's easier to
later make a name non-internal than the reverse.  Which is true,
but which is not a great reason, in itself: it would apply to
every new function.

I submitted bug report #24386 to get rid of the "internal"
characterization.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> Your analogy is OK.  I wouldn't speak of an "Elisp expression" here,
> since to me that just means any expression - sexp that you can
> construct with Emacs Lisp.

What would you call it then?
What did you expect an "Elisp expression" to mean when contrasted to an
"S-exp"?  I really can't understand how you could have gotten confused.

> See above.  Any Lisp sexp can be related to Elisp code.  Any data
> can be a piece of some code.

That's irrelevant.  'int b*;' and 'a[b' can appear in a C program, but
they're not C expressions.  Similarly, ((a . 2) (b . 3)) is *not* an
Elisp expression, regardless of whether it can appear inside an
Elisp expression.

> It's what Davis was trying to distinguish `read--expression' as:
> reading an expression "intended for evaluation" (paraphrasing).

I don't think I talked about "expression" intended for evaluation but
"S-exp" intended for evaluation.

For me the name "expression" already tends to have a connotation of
being a piece of code that can be evaluated (e.g. in contexts where
you'd also see related concepts like declaration, instruction, data,
...), so I use "S-exp" to make it clear I'm only talking about the
XML-like syntactic level independent from any associated semantic.

> Likewise, you.  You both tried to distinguish what it does as
> reading an expression that might be evaluated (even though it can
> also read expressions that cannot, without raising an error).

How else would you describe the difference between
- a thingy that uses the Elisp reader syntax and whose semantics we know
  nothing about.
- that same thingy but knowing that it's a chunk of Elisp code.
?

> I don't have a name for that.  "Expression" doesn't say more than
> "sexp", to me - it's just as general.

If we confuse those two concepts somewhere, it's a bug.

>> > 1. That's not a good reason to make it "internal".
>> I already said that it's internal for no particular reason.
> Not really.  You said that it's internal because it's easier to
> later make a name non-internal than the reverse.  Which is true,
> but which is not a great reason, in itself: it would apply to
> every new function.

We violently agree: since it would apply to every function, it's not
a particular reason.


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> > Your analogy is OK.  I wouldn't speak of an "Elisp expression"
> > here, since to me that just means any expression - sexp that
> > you can construct with Emacs Lisp.
> 
> What would you call it then?
> What did you expect an "Elisp expression" to mean when contrasted to an
> "S-exp"?  I really can't understand how you could have gotten confused.

You spoke of intention to evaluate, as I said.  I didn't,
and still don't, see how such an intention enters into what
`read--expression' does.  I don't even know what such
intention means, operationally.

I can see how the Lisp _reader_ does not accept certain Lisp
objects, so such errors raised can be considered syntax errors,
just as trying to read an isolated comma or period.

But if you bring in evaluation to further classify Lisp
objects then please tell us exactly how those that you
call "expression" differ with respect to evaluation.

And after that, please bring in the notion of an "intention"
to evaluate.

Finally, please relate all of that to what `read--expression'
actually does, which is what I was/am interested in.

I think you'll find that you'll be going in circles and
waving your hands.  I would be too, if I were trying to
nail down such things.

> > See above.  Any Lisp sexp can be related to Elisp code.  Any data
> > can be a piece of some code.
> 
> That's irrelevant.  'int b*;' and 'a[b' can appear in a C program, but
> they're not C expressions.  Similarly, ((a . 2) (b . 3)) is *not* an
> Elisp expression, regardless of whether it can appear inside an
> Elisp expression.

Adding emphasis doesn't make it any more evident, I'm afraid.
(And Lisp is not C.)

You haven't defined Elisp expression, except to say that it
is a sexp that you intend to evaluate.  Is (foo 2) an Elisp
expression?  Even if `foo' is not a defined function?  Or if
it is defined but evaluating (foo 2) raises an error?

What does this tell you?

(read--expression "Sexp: ")
Type: ((a . 2) (b . 3))

Does `read--expression' complain that what it read here is
not an "Elisp expression"?  (Nope.)

I'll not quibble over the (as yet undefined and unclear)
terms you use.  You asked why the confusion.  I've told you.

I said from the beginning that I was talking about what
`read--expression' reads and returns.

> > It's what Davis was trying to distinguish `read--expression' as:
> > reading an expression "intended for evaluation" (paraphrasing).
> 
> I don't think I talked about "expression" intended for evaluation
> but "S-exp" intended for evaluation.

Perhaps.  I identified the two; mea culpa.

But I do see now that (elisp) `Intro Eval' defines "expression"
and "form" as "a Lisp object that is intended for evaluation".
So I can see where you're coming from.

(I don't see there, either, how such "intention" is defined,
however.  To me, this is as vague as your use of it.)

On the other hand, it adds: "It is sometimes referred to as
an "S-expression" or "sexp", but we generally do not use this
terminology in this manual."

It is standard Lisp terminology (from McCarthy's first paper,
even, IIRC), but that's no matter.  At least you can see, I
hope, my "confusion": the two are indeed the same, even in
Emacs parlance.  (And AFAICT, they correspond to what
`read--expression' reads and returns.)

Apparently, in Emacs-Speak, they are the same.  They are
distinguished from the more general category of Lisp "objects".

I gather that the latter, or at least some of the latter, are
what you have been calling "S-exps".

> For me the name "expression" already tends to have a connotation of
> being a piece of code that can be evaluated (e.g. in contexts where
> you'd also see related concepts like declaration, instruction, data,
> ...), so I use "S-exp" to make it clear I'm only talking about the
> XML-like syntactic level independent from any associated semantic.

I gathered that, but it was not clear at the outset.  And
your terms "expression" and "S-exp" apparently correspond to
what Emacs calls "expression" and Lisp "object", respectively,
whereas it says that  "expression" and "S-exp" are the same
(and that the manual does not use "S-exp"/"sexp" - but Emacs
does elsewhere). 

> > Likewise, you.  You both tried to distinguish what it does as
> > reading an expression that might be evaluated (even though it can
> > also read expressions that cannot, without raising an error).
> 
> How else would you describe the difference between
> - a thingy that uses the Elisp reader syntax and whose semantics
>   we know nothing about.
> - that same thingy but knowing that it's a chunk of Elisp code.
> ?

Define "knowing that it's a chunk of Elisp code".  Does it
mean that `eval' does not raise an error?  That it does not
raise certain kinds of error?

Is any thing acceptable to the Lisp reader a potential
"chunk of Elisp code"?  Why not?

> > I don't have a name for that.  "Expression" doesn't
> > say more than "sexp", to me - it's just as general.

And for Emacs too, apparently.  On this, at least, Emacs
and I apparently speak with the same terms.

> If we confuse those two concepts somewhere, it's a bug.

At any rate, my interest was with what `read--expression'
reads and returns.  Call that what you like.

And `read--expression' reads and returns the Lisp object
((a . 2) (b . 3)) just fine - a thing which you apparently
exclude from your class of "expressions" but include in
in your class of "sexps".  So for you, it seems that
`read--expression' does not necessarily read an expression.

> >> > 1. That's not a good reason to make it "internal".
> >> I already said that it's internal for no particular reason.
> > Not really.  You said that it's internal because it's easier to
> > later make a name non-internal than the reverse.  Which is true,
> > but which is not a great reason, in itself: it would apply to
> > every new function.
> 
> We violently agree: since it would apply to every function,
> it's not a particular reason.

I don't think we disagree, but we have at least been
confusing each other with words.  Getting better, I hope.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> You spoke of intention to evaluate, as I said.

When people write code, they intend for it to be evaluated at some
point, since that's the whole difference between "code" and "data".


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> When people write code, they intend for it to be evaluated at some
> point, since that's the whole difference between "code" and "data".

No kidding.

RE: `read--expression' and `read-minibuffer'

[Herring, Davis]

[Many variations of the following points snipped.]

> You spoke of intention to evaluate, as I said.  I didn't,
> and still don't, see how such an intention enters into what
> `read--expression' does.  I don't even know what such
> intention means, operationally.

It affects completion, as has already been stated: given

(comment-c

as a _form_ (hopefully less ambiguous than "expression"), the only completion is `comment-choose-indent'; as a general s-exp, `comment-column' is probably more likely.

> But if you bring in evaluation to further classify Lisp
> objects then please tell us exactly how those that you
> call "expression" differ with respect to evaluation.

They have a structure that could possibly evaluate without error?  Of course, nothing can check that an evaluation completes without error (without actually evaluating it, which might never complete).  So `read--expression' doesn't bother checking at all.

> Does `read--expression' complain that what it read here is
> not an "Elisp expression"?  (Nope.)

It is better to interpret `read--expression' not as "read and return a form" but rather as "read an s-exp from the user, helping them type a form".  It really is a question of intent: the caller intends to get a form, so they make it easy for the user (who is hopefully also intending to enter a form) to do so.

["they" are "expression"/"form" and "sexp"]
> Apparently, in Emacs-Speak, they are the same.  They are
> distinguished from the more general category of Lisp "objects".

Be careful with the map and the territory: a string for the Lisp reader isn't itself a (Lisp) object unless it also exists as a string object in Lisp.  An object results from a read of an s-exp; the words "expression" and "form" are commonly applied both to the object read (if it "seems" evaluable) and to the read syntax for it, but the word "object" is not.

Davis

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> > You spoke of intention to evaluate, as I said.  I didn't,
> > and still don't, see how such an intention enters into what
> > `read--expression' does.  I don't even know what such
> > intention means, operationally.
> 
> It affects completion, as has already been stated: given
> (comment-c as a _form_ (hopefully less ambiguous than "expression"),

(No less ambiguous.)

> the only completion is `comment-choose-indent'; as a general s-exp, 
> `comment-column' is probably more likely.

All of that has already been said, by each of us.  I said it at
the outset.  It's clear that the completion it provides is for
function names after `('.  No one disputes that.

What `read--expression' reads and returns says nothing about any
particular intention to evaluate.  And what it reads and returns 
is not limited by the (handy, for some expressions) completion it
provides.  And even that completion says nothing about the
intended use of what is read, in particular about any intention
to evaluate it.

This has all been said before in this thread.

> > But if you bring in evaluation to further classify Lisp
> > objects then please tell us exactly how those that you
> > call "expression" differ with respect to evaluation.
> 
> They have a structure that could possibly evaluate without error?

Is that really how you want to define "expression"?  So `(/ 3 0)'
is not an expression, _because_ it raises an arithmetic error?
Or do you want to backtrack a bit, and settle on just syntax
errors?  Or on some other kind of error?  See my previous message
about this.

> Of course, nothing can check that an evaluation completes without error
> (without actually evaluating it, which might never complete).  So `read--
> expression' doesn't bother checking at all.

Of course.  And where does that leave your attempt to define
"expression" in terms of evaluation non-error?

And what makes you assume that it would if it could, that is,
that it even cares only about expressions that could be evaluated
without error?  Why not accept it for what it is - characterize
it by its actual behavior (which is to read any syntactically
valid sexp).

What it does is similar to what `read' does.  The difference is
that it reads from the minibuffer, providing handy completion,
in particular for some kinds of expressions.

And note that the doc of `read' too talks about reading a "Lisp
expression".  It does not exclude `((a . b) (c . d))' or any
other syntactically valid sexp from the class of "expressions".

In fact, the behavior of `read' - the behavior of the Lisp reader,
_defines_ syntactic validity.  It defines what is an "expression",
aka "sexp", and what is not, I think.

> > Does `read--expression' complain that what it read here is
> > not an "Elisp expression"?  (Nope.)
> 
> It is better to interpret `read--expression' not as "read and return a
> form" but rather as "read an s-exp from the user, helping them type a
> form".

So you say.  But why is it better to interpret things as you
choose to interpret them?  A better interpretation is based on
what the function actually does, I think: what it reads without
error and what it returns.

> It really is a question of intent: the caller intends to get a
> form, so they make it easy for the user (who is hopefully also
> intending to enter a form) to do so.  ["they" are
> "expression"/"form" and "sexp"]

So you say, with nothing but hand-waving to back it up.  That's
presumptuous about what a caller wants to get from the user and
what it wants to do with what it gets.

Instead, look at what the function actually reads (without error)
and what it actually returns.  There is no such limitation in its
_behavior_ as what you propose/intend.

`read-expression' is handy for reading lots of sexps that do not
fit your characterization of intent - simply because that's what
it actually does: reads and returns a sexp.

> > Apparently, in Emacs-Speak, they are the same.  They are
> > distinguished from the more general category of Lisp "objects".
> 
> Be careful with the map and the territory: a string for the Lisp reader
> isn't itself a (Lisp) object unless it also exists as a string object in
> Lisp.  An object results from a read of an s-exp; the words "expression"
> and "form" are commonly applied both to the object read (if it "seems"
> evaluable) and to the read syntax for it, but the word "object" is not.

We're not (so far, but if you insist then we can also go down that
path) distinguishing representation from represented, here.

Is a sexp or an expression text on paper, characters in a buffer
or in a string?  Is it something that only _represents_ a Lisp
object?  Or is it a Lisp object?

Is what is read by `read--expression' (or by any read) a
sexp/expression?  Or is that what is returned by it?  Or both?
Is this a string: "abc"?  Or does a string result only from
reading that text?

It's not necessary to make this representation/represented
distinction _here_, but if you insist, then go for it.  Stake
out clearly what is representation and what is represented, and
relate what you come up with to the behavior of `read-expression'.

_My_ question is only about `read--expression'.  To me, it is
handy for ordinary user-programmers, not just for Emacs developers,
and it should not be considered "internal".

To me, even though it provides function-name completion after `('
and both function- and variable-name completion otherwise, it is
useful not only for reading function applications and other forms
that can be evaluated without error, but also other sexps that
cannot.

You can limit yourself to using it only for the former, if you
like, but that's not what the function does, so its users are not
limited to any such "intention".

Re: `read--expression' and `read-minibuffer'

[Davis Herring]

>> It affects completion, as has already been stated: given
>> (comment-c as a _form_ (hopefully less ambiguous than "expression"),
>
> (No less ambiguous.)

Why not?  It's defined in the Emacs Lisp Reference Manual, and it's not 
a generic term of art from parsing in a variety of contexts.

> All of that has already been said, by each of us.  I said it at
> the outset.  It's clear that the completion it provides is for
> function names after `('.  No one disputes that.

And thus a user who was expecting to enter the cons (comment-column . 
40) would be surprised, would they not, when the completion chose 
something else?

Isn't that enough of a reason to acknowledge the function's 
predisposition towards reading forms, not general values?

> And what it reads and returns is not limited by the (handy, for
> some expressions) completion it provides.

No one is questioning its _capacity_ to accept other data from the user. 
  (But if capacity is all that matters, you might as well use a normal 
`read-from-minibuffer' and call `read' on the result yourself.)

> And even that completion says nothing about the
> intended use of what is read, in particular about any intention
> to evaluate it.

Nonsense -- that choice of completion mechanism, coupled with the choice 
of the caller to use a function that provides it, indicates plainly that 
the code's author meant for a form to be entered.  The only escape is 
the notion that the author used the function, despite a semantic 
mismatch, because there was no better choice available or because they 
didn't know about it.  Either of those constitutes a bug -- the former 
of laziness, and the latter of ignorance.

>> They have a structure that could possibly evaluate without error?
>
> Is that really how you want to define "expression"?  So `(/ 3 0)'
> is not an expression, _because_ it raises an arithmetic error?
> Or do you want to backtrack a bit, and settle on just syntax
> errors?  Or on some other kind of error?  See my previous message
> about this.

No backtracking: I already said "could possibly evaluate".  No dotted 
list (e.g., (/ 3 . 0)) can ever evaluate.  You can also determine that 
the list (3 0 /) will not evaluate successfully without any semantic 
understanding, because 3 is not a function.  Somewhat less reliably, we 
can determine that the list (comment-column 40) will fail under 
evaluation because `comment-column' is void as a function.

Rice's theorem, of course, denies most progress beyond that point: you 
can try to count arguments, for example, but as has been discussed 
recently that is fraught with peril in the case of advice-like function 
wrappers.

We nonetheless can offer a rigorous definition of a form such that all 
expressions which evaluate without error are forms, even though not all 
forms evaluate without error (and some of them have variable behavior).

> And what makes you assume that it would if it could, that is,
> that it even cares only about expressions that could be evaluated
> without error?  Why not accept it for what it is - characterize
> it by its actual behavior (which is to read any syntactically
> valid sexp).

That's like characterizing Emacs as a compiler because someone chooses 
to write an object file in it with C-q.  The tailoring of the tool, not 
the result of its use in any particular instance, defines its purpose.

> And note that the doc of `read' too talks about reading a "Lisp
> expression".  It does not exclude `((a . b) (c . d))' or any
> other syntactically valid sexp from the class of "expressions".

In that context, "expression" is set against "object" and plainly refers 
to the textual representation.  (This is why I switched to the word "form"!)

> A better interpretation is based on
> what the function actually does, I think: what it reads without
> error and what it returns.

"Actually does" is a poor choice.  `universal-argument' "actually 
returns" the "exit function" created by `set-transient-map'.  But it 
doesn't document that behavior, because we prefer the flexibility of 
changing it to the (marginal) utility to the caller of 
`universal-argument'.  (For example, in Emacs 23 it returns t; should 
the current implementation contain a stray "t" to retain that behavior?)

> So you say, with nothing but hand-waving to back it up.  That's
> presumptuous about what a caller wants to get from the user and
> what it wants to do with what it gets.

What I have offered to "back it up" appears minuscule only in comparison 
to the redundant text arguing about it (much of it snipped here).

> `read-expression' is handy for reading lots of sexps that do not
> fit your characterization of intent - simply because that's what
> it actually does: reads and returns a sexp.

So you find the (undefined) behavior of its current implementation 
useful.  What if you start relying on it and then someone else changes 
it to do the sanity checks I discuss above (because, say, they want to 
get better error messages than the evaluator gives, or they want to stop 
the user from exiting the minibuffer with invalid input)?

That's why we have the documented contract -- to let us know whether you 
were in error to rely on it for "off-label" uses, or whether the 
implementer was in error to alter it.  We may choose that contract at 
our whim -- and at our peril.  There's no reason to suppose that the 
answer is "whatever it currently implements, exactly", and less than no 
reason to choose "help the user type a form, but do not check its 
validity in any way in case the caller is expecting something else".

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> >> a _form_ (hopefully less ambiguous than "expression"),
> >
> > (No less ambiguous.)
> 
> Why not?  It's defined in the Emacs Lisp Reference Manual, and it's
> not a generic term of art from parsing in a variety of contexts.

I already addressed that.  Again: In the Elisp manual, node
`Intro Eval', "form" is given as a synonym for "expression".
And both are mentioned as synonyms for both "S-expression" and
"sexp" (which the manual generally avoids using).

> > All of that has already been said, by each of us.  I said it at
> > the outset.  It's clear that the completion it provides is for
> > function names after `('.  No one disputes that.
> 
> And thus a user who was expecting to enter the cons (comment-column .
> 40) would be surprised, would they not, when the completion chose
> something else?

Unless `comment-column' has also been given a function definition
in the user's session, completion of `comment-c' after `(' would
not provide `comment-column' as a candidate.  This has all been
said multiple times now.

But a user could type `(2 commment-c', `TAB' to complete to
`(2 comment-column', and then remove the 2.  For a long name that
is not memorized, such completion can help.

And for a variable or function name in the current buffer, `M-/'
and `C-M-/' provide completion anywhere, including just after `('.
Again, for a long name, such completion can help, regardless of
whether the sexp read is evaluable without error.

And using completion or not, a user can certainly enter
`(comment-column . 4)', and that dotted list is what is
returned by `read--expression'.

This has all been covered already in this thread.

Why do you reply to:

  It's clear that the completion it provides is for
  function names after `('.  No one disputes that.

by belaboring that point, as if you are divulging it?  TAB
completion of a name after `(' is function-name completion.
Yes, and so what?  `read--expression' is not _limited_ to
reading only function names after `('.

> Isn't that enough of a reason to acknowledge the function's
> predisposition towards reading forms, not general values?

Predisposition for completing a function name after `('.
It is not limited to reading list sexps that only have a
function-name car.

> > And what it reads and returns is not limited by the (handy,
> > for some expressions) completion it provides.
> 
> No one is questioning its _capacity_ to accept other data
> from the user.

And no one is questioning that it provides function-name (only)
completion after `('.

> (But if capacity is all that matters, you might as well use a normal
> `read-from-minibuffer' and call `read' on the result yourself.)

That's the point of this thread.  No.  `read--expression' is
handier for many sexps, including some that are not lists with
function-name cars.

It's not black-&-white.  Each read function, according to what
it actually does, has its own advantages and use cases.

The point (#1) is that the useful uses of `read--expression' are
not limited to reading lists with function-name cars.  In some
contexts, completion of function and variable names can be useful
for reading other sexps.

And the point (#2) is that the usefulness of `read--expression'
is not limited to Emacs developers - it should not be considered
"internal".

> > And even that completion says nothing about the intended use of
> > what is read, in particular about any intention to evaluate it.
> 
> Nonsense -- that choice of completion mechanism, coupled with the choice
> of the caller to use a function that provides it, indicates plainly that
> the code's author meant for a form to be entered.

It's not about the author of `read--expression'.  It's about its
users and their intentions.  Why particular code reads a sexp -
any sexp, is up to it.  You cannot assume that it will then be
evaluated by that code.  And that stands for any read function,
including `read--expression'.

> >> They have a structure that could possibly evaluate without error?
> >
> > Is that really how you want to define "expression"?  So `(/ 3 0)'
> > is not an expression, _because_ it raises an arithmetic error?
> > Or do you want to backtrack a bit, and settle on just syntax
> > errors?  Or on some other kind of error?  See my previous message
> > about this.
> 
> No backtracking: I already said "could possibly evaluate".  No
> dotted list (e.g., (/ 3 . 0)) can ever evaluate.
> You can also determine that the list (3 0 /) will not evaluate
> successfully without any semantic understanding, because 3 is
> not a function.  Somewhat less reliably, we
> can determine that the list (comment-column 40) will fail under
> evaluation because `comment-column' is void as a function.

But none of that is what `read--expression' does.  It happily reads
(/ 3 . 0), (3 0 /), (comment-column 40), and (comment-column . 40).

What it does not accept (it raises an error) is a syntax error,
i.e., a Lisp _read_ error, such as an isolated `.' or `,'.

What it does is read and return a Lisp sexp, and not only one
that `eval' can handle without error.

And what if your code even _does_ intend to evaluate a sexp read,
but only after its symbol car gets defined?  Will your "improved"
`read--expression' be right in not accepting (foo 42) because, at
the moment of reading, `foo' is not yet defined?  So much for its
guessing intention of evaluation.

> > And note that the doc of `read' too talks about reading a "Lisp
> > expression".  It does not exclude `((a . b) (c . d))' or any
> > other syntactically valid sexp from the class of "expressions".
> 
> In that context, "expression" is set against "object" and plainly refers
> to the textual representation.  (This is why I switched to the word
> "form"!)

(Again ((elisp) `Intro Eval'), for Emacs, "form"="expression".)

And yes, of course it is setting "expression" against "object".
That's what a Lisp read function does, and `read--expression' is
no exception.

In the language of Common Lisp (CLTL2, 22.1.1), the Lisp reader

   "accept[s] characters, interpret[s] them as the printed
    representation of a Lisp object, and construct[s] and
    return[s] such an object."

> > A better interpretation is based on what the function actually
> > does, I think: what it reads without error and what it returns.
> 
> "Actually does" is a poor choice.

Info that helps the user about what `read--expression' does:

. It reads a Lisp sexp from the minibuffer.
. For symbol names, it provides completion using `TAB'.
. For a symbol name after `(', only function names are completed
  using `TAB'.  (Function in the wide sense: macro, special-form,
  function.)
. For a symbol name anywhere else, function and variable names
  are completed using `TAB'.

(And `M-/' and `C-M-/' complete to names from the current buffer.)

> > `read-expression' is handy for reading lots of sexps that do not
> > fit your characterization of intent - simply because that's what
> > it actually does: reads and returns a sexp.
> 
> So you find the (undefined) behavior of its current implementation
> useful.  
>
> What if you start relying on it and then someone else changes
> it to do the sanity checks I discuss above (because, say, they want to
> get better error messages than the evaluator gives, or they want to stop
> the user from exiting the minibuffer with invalid input)?

We'll cross that bridge when we come to it.  For now, the only
request has been to remove the extra hyphen and not consider it
"internal", please.

> That's why we have the documented contract

Sorry, I must have missed that "documented contract".
Where is it? What does `C-h f read--expression' tell you?
"Not documented."

You said that the behavior is undefined.  So I say look at what it does.

> -- to let us know whether you were in error to rely on it for
> "off-label" uses, 

Sorry, where's the label?  ("Not documented.")

> or whether the implementer was in error to alter it.  We may choose
> that contract at our whim -- and at our peril.  There's no reason to
> suppose that the answer is "whatever it currently implements, exactly",

What reason is there to suppose that the nonexistent contract
would say, if it existed, that the answer is "whatever Davis says"?

> and less than no reason to choose "help the user type a form,
> but do not check its validity in any way in case the caller is
> expecting something else".

In the absence of any "documented contract", we can go by what
the function does.  Or we can go by what you happen to think
the function really wants to do but does not do.

So far, I'm content with what `read--expression' does and does
not do.  But it should be considered just as useful for an
ordinary Emacs user-programmer as it is for Emacs developers:
not "internal".

Re: `read--expression' and `read-minibuffer'

[Michael Heerdegen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > That's not a real problem, but that's the only place/case where I
> > can see an argument for a "variant" behavior (completion against
> > both function and variable names at that position too).
>
> For S-exps in general, we can't provide much completion, and we
> certainly shouldn't provide completion against function names and
> variable names.
>
> An S-exp is like an XML tree.  An Elisp expression is like a XHTML tree
> (i.e. it's a particular kind of XML that is expected to follow
> a particular schema).  Yes, it's also an XML tree, but if all you know
> is that you're typing an XML tree, providing any completion based on
> XHTML identifiers is just bogus.

This seems a bit like a pseudo problem to me.  The difficulty to define
and distinguish the two cases indicates that.

We also don't have two emacs lisp modes.  Emacs Lisp mode doesn't turn
off completion for the sub-s-exps of a s-exp that are not expressions.
We try to provide a context dependent behavior as good as we can.  If I
want to write random data, I just don't use completion, simply because
it's not useful.

Personally I use emacs-lisp-mode in all minibuffers that prompt for an
elisp-whatever.  Apart from completion, some other familiar keys are
useful as well.  And font locking, paredit, rainbow-delimiters...

And I'm not sure how large the part of corner cases is.  Like this one:
In el-search.el, I prompt for a pcase pattern.  These are typically not
expressions, but it is useful to have completion there.  Completion is
not useful at every position, but that's not the case for expressions as
well (well, they can contain pcase patterns...).

Apart from that, I would say, let's wait with concepts like "whether a
s-exp is an expression or not is undefined until you try to evaluate it"
until we have quantum computers :-)


Regards,

Michael.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

> This seems a bit like a pseudo problem to me.  The difficulty to define
> and distinguish the two cases indicates that.

Not at all.  When a piece of code needs the user to type in an alist, it
uses read-minibuffer, when it needs the user to type in some kind of
chunk of code, it uses read--expression.

Both can be used to type in the same set of values, but the intention
behind is very different and providing eldoc support when typing in an
alist wouldn't make much sense in general, whereas it will usually be
meaningful in read--expression.


        Stefan

Re: `read--expression' and `read-minibuffer'

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Not at all.  When a piece of code needs the user to type in an alist,

Is this a real-life example?  In this special case, a key+value multi
prompt would be much more comfortable than a prompt for an alist, I
think.  If the alist is longer, I would prepare it in scratch and copy
it into the minibuffer anyway.  About what use cases of entering
non-expression s-exps are we talking here?

And maybe it often makes sense when symbol names appear in the alist,
who knows?  It's worse to have no completion in these cases than having
a completion that is just not useful in the other cases.

Or the programmer may even want to enable the user to enter an
expression that evaluates to the alist he should specify (maybe the user
wants to refer to some other alist some symbol is bound to, or he had
entered before)...then we are in the other case again.


Michael.

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

>> Not at all.  When a piece of code needs the user to type in an alist,
> Is this a real-life example?  In this special case, a key+value multi
> prompt would be much more comfortable than a prompt for an alist, I
> think.

Could be, but that's what read-minibuffer does: it reads an S-exp
without knowing anything more about what the S-exp will be used for.
It's never the best choice, indeed.


        Stefan

Re: `read--expression' and `read-minibuffer'

[Stefan Monnier]

>>> Because it is tailored for that use case.  The kind of completion
>>> it provides for instance assumes the S-exp is an Elisp expression.
>> Of course it is an Elisp expression.  It does not follow that
>> the Elisp expression that has been read will then be _evaluated_.
> In this thread, Stefan is defining an "Elisp expression" to be precisely an
> S-exp that is intended for `eval'.

Not necessarily directly for `eval', but yes an Elisp expression as
opposed to random S-exp data.  E.g. when reading an alist, you shouldn't
use read--expression (whereas, when reading an expression that will
evaluate to an alist, read--expression is probably the right thing to
use).

> So a variant of `read--expression' could be written that does a subset
> of that completion appropriate to the more general context of entering
> an S-exp.

For any random S-exp data, we already have `read-minibuffer', so I don't
see any point in changing read--expression to cover that case.


        Stefan

RE: `read--expression' and `read-minibuffer'

[Drew Adams]

> >>> Because it is tailored for that use case.  The kind of completion
> >>> it provides for instance assumes the S-exp is an Elisp expression.
> >> Of course it is an Elisp expression.  It does not follow that
> >> the Elisp expression that has been read will then be _evaluated_.
> > In this thread, Stefan is defining an "Elisp expression" to be precisely
> > an S-exp that is intended for `eval'.
> 
> Not necessarily directly for `eval', but yes an Elisp expression as
> opposed to random S-exp data.  E.g. when reading an alist, you shouldn't
> use read--expression (whereas, when reading an expression that will
> evaluate to an alist, read--expression is probably the right thing to
> use).
> 
> > So a variant of `read--expression' could be written that does a subset
> > of that completion appropriate to the more general context of entering
> > an S-exp.
> 
> For any random S-exp data, we already have `read-minibuffer', so I don't
> see any point in changing read--expression to cover that case.

Completion of parts of the data.  Just as in the function-application
sexp case.  There just is assumption that the first symbol after `('
should complete against only function symbols.