should read-file-name not respect text properties in its input string?

should read-file-name not respect text properties in its input string?

[Drew Adams]

Back in 2007 (thread "display-completion-list should not strip text
properties"), I argued that completion candidates should be able to have text
properties and the result read should retain those properties. Richard OK'd that
change.

I think the same thing should hold for `read-file-name', not just
`completing-read': If you provide input that has text properties, the file name
returned should retain those properties. IOW, stripping properties should not be
part of the job of reading file names.

That's the behavior I've had in Icicles. But I've discovered that some other
code (GNUS, it seems) naively chokes if `read-file-name' returns a string that
has text properties. So I've backed off in Icicles - I now strip properties in
my version of `read-file-name', to be compatible. Dommage.

I'd still like to pose the question here, however. Shouldn't `read-file-name'
just return the string that you choose, including any text properties it might
have? If we made that change, then some code (e.g. GNUS) that expects there are
no properties might need to be fixed to first remove all properties.

There might not be any obvious need for a file-name string that has properties,
but neither is there any a priori reason to strip properties that might be
included. An application might well want to take advantage of properties here in
some way.

The place where a user reported a problem with this was GNUS or mml (it seems)
simply printing (into an XML attribute value) the file name that was read,
without first stripping any properties. That is, it uses the wrong Lisp print
function, producing crap like this:

<#part type="text/plain"
       filename=#("~/test.txt" 0 10 (auto-composed t))
       disposition=attachment description=test>
<#/part>

I would think that a program that creates XML code would take the trouble of
verifying such data before sticking it in, but, ahem, this is the real world.
;-)

To me, it's the job of any caller that wants no properties to strip them out;
that shouldn't be the job of `read-file-name'. Anyone else feel that
`read-file-name' should remain general and agnostic wrt the presence of text
properties in the string it reads? 

Re: should read-file-name not respect text properties in its input string?

[Stefan Monnier]

> Back in 2007 (thread "display-completion-list should not strip text
> properties"), I argued that completion candidates should be able to
> have text properties and the result read should retain those
> properties. Richard OK'd that change.

IIUC completion candidate's properties are preserved when displayed in
*Completions* now.  So part of it is done.

I think that to make read-file-name preserve properties, you'd need to
make substitute-in-file-name preserve properties.  It's doable, but it's
a non-trivial amount of work and it's not clear how useful that would
be, so ... patches welcome, especially together with some compelling
example showing why it's useful (rather than some vague principle).


        Stefan

RE: should read-file-name not respect text properties in its input string?

[Drew Adams]

> > Back in 2007 (thread "display-completion-list should not strip text
> > properties"), I argued that completion candidates should be able to
> > have text properties and the result read should retain those
> > properties. Richard OK'd that change.
> 
> IIUC completion candidate's properties are preserved when displayed in
> *Completions* now.  So part of it is done.
> 
> I think that to make read-file-name preserve properties, you'd need to
> make substitute-in-file-name preserve properties.  It's 
> doable, but it's
> a non-trivial amount of work and it's not clear how useful that would
> be, so ... patches welcome, especially together with some compelling
> example showing why it's useful (rather than some vague principle).

I already said I don't have any particular example in mind. The logic and
reasons are essentially the same as for `completing-read', however - see the
referenced 2007 thread.

Actually, however, I just realized that the situation is different from what I
thought.

In the thread about `display-completion-list', the request I made, and the aim,
was not just that `display-completion-list' be able to display candidates that
have text properties, but also that `completing-read' return such a chosen
candidate, with its properties. IOW, that `completing-read' would respect text
properties in the user's input, not just in the completion candidates, returning
the possibly propertized string that the user chooses.

Apparently, that part was never implemented - `completing-read' still strips
text properties. And, since `read-file-name' is now implemented using
`completing-read', so does `read-file-name'.

The good news is that if what was requested and OK'd last year gets implemented,
then it will also hold for `read-file-name'. I can see that because I have a
version of `completing-read' that preserves text properties, and when I call
`read-file-name' in Emacs 23, it too preserves text properties.

IOW, AFAICT, there is no need to modify `substitute-in-file-name' or
`read-file-name'. It is only `completing-read' that needs to be fixed to allow
this. The rest will follow.

The reason I didn't see this at first is that Icicles uses a version of
`completing-read' that DTRT: doesn't strip text properties. So `read-file-name'
also doesn't strip them, which led to the problem with the GNUS code that
blindly print the propertized string as a Lisp #("..."...) form. I thought the
problem was `read-file-name', but it is the vanilla `completing-read' that is
the only problem.

I hope you will consider making the change to `completing-read', so it returns
whatever string the user inputs, including its text properties, if any.

Re: should read-file-name not respect text properties in its input string?

[Stefan Monnier]

> I already said I don't have any particular example in mind. The logic and
> reasons are essentially the same as for `completing-read', however - see the
> referenced 2007 thread.

Could you sumarize it or give an URL for it?
The main problem I have to understand how this could be used is that if
the user types the text, it won't get the properties from the completion
candidates since the completion was never involved.

So do you want the properties from the completion candidates or from the
minibuffer text?  If from the minibuffer text (the only reliable
choice), what kind of properties would these be, who would put
them there?


        Stefan

RE: should read-file-name not respect text properties in its input string?

[Drew Adams]

> > I already said I don't have any particular example in mind. 
> > The logic and reasons are essentially the same as for
> > `completing-read', however - see the
> > referenced 2007 thread.
> 
> Could you sumarize it or give an URL for it?

Irrelevant, as I said, because `read-file-name' will automatically work if the
job is finished for `completing-read'.

> The main problem I have to understand how this could be used 
> is that if the user types the text, it won't get the properties
> from the completion candidates since the completion was never
> involved.

There are different ways to choose a candidate, and different ways to provide
input to `completing-read'. You can choose with the mouse. You can yank
propertized text in the minibuffer. And TAB completion too can put the
propertized candidate in the minibuffer.

> So do you want the properties from the completion candidates 
> or from the minibuffer text?

Whatever is in the minibuffer when the user ends completion (RET). If the user
chooses one of the candidates (however it is chosen), then it should be used,
together with its properties. If the user instead enters something else
(non-candidate with lax completion), whatever s?he enters should be used,
together with its properties.

> If from the minibuffer text (the only reliable
> choice), what kind of properties would these be, who would put
> them there?

If the input text comes from choosing a candidate (e.g. mouse-2, TAB
completion), then whatever properties the candidate has. If the input comes from
the user yanking propertized text, then whatever properties that yanked text
has. Even if the user uses facemenu in the minibuffer to change the face of text
s?he inputs, and then types text with that face, then whatever properties that
text has.

However the input text happens to become propertized, it should retain its
properties when it is returned from `completing-read' (and so from
`read-file-name' also).

RE: should read-file-name not respect text properties in its inputstring?

[Drew Adams]

I said:

> Whatever is in the minibuffer when the user ends completion 
> (RET). If the user chooses one of the candidates (however
> it is chosen), then it should be used, together with its
> properties.

However, note that `choose-completion-string' removes the `mouse-face' property
that was added to candidates for use in *Completions*.

Re: should read-file-name not respect text properties in its input string?

[Stefan Monnier]

>> > I already said I don't have any particular example in mind. 
>> > The logic and reasons are essentially the same as for
>> > `completing-read', however - see the
>> > referenced 2007 thread.
>> 
>> Could you sumarize it or give an URL for it?

> Irrelevant, as I said, because `read-file-name' will automatically
> work if the job is finished for `completing-read'.

I did read what you wrote, so I meant a summary about completing-readm
not about read-file-name.

>> If from the minibuffer text (the only reliable choice), what kind of
>> properties would these be, who would put them there?

> If the input text comes from choosing a candidate (e.g. mouse-2, TAB
> completion), then whatever properties the candidate has. If the input
> comes from the user yanking propertized text, then whatever properties
> that yanked text has. Even if the user uses facemenu in the minibuffer
> to change the face of text s?he inputs, and then types text with that
> face, then whatever properties that text has.

> However the input text happens to become propertized, it should retain
> its properties when it is returned from `completing-read' (and so from
> `read-file-name' also).

So, again, what kind of properties would these be?


        Stefan

RE: should read-file-name not respect text properties in its input string?

[Drew Adams]

> So, again, what kind of properties would these be?

Uh, again, text properties. Period.





Here's one mail from the 2007 thread, which lists possible uses of a few
properties:

http://lists.gnu.org/archive/html/emacs-devel/2007-01/msg00807.html

However, a priori, any text properties at all. An application might add its own,
original properties and treat them in different ways. Properties can even
distinguish candidates that otherwise have the same string (Icicles uses this
possibility in some contexts). There are lots of possible uses - use your
imagination.

It's easy enough for a caller to strip properties. And it's easy enough for a
caller to strip completion candidates of properties from the outset. No reason
for `completing-read' to systematically strip properties from the result. Let
those callers that know they don't want such info throw it away - don't throw it
away for them.

The preservation of properties that are already there does not force anyone to
use properties. Just like the possibility of using properties in a buffer.

Re: should read-file-name not respect text properties in its input string?

[Stefan Monnier]

>> So, again, what kind of properties would these be?
> Uh, again, text properties. Period.

> Here's one mail from the 2007 thread, which lists possible uses of a few
> properties:

> http://lists.gnu.org/archive/html/emacs-devel/2007-01/msg00807.html

This lists uses of properties in completion candidates (typically when
displayed in *Completions*).  As mentioned, this should already
work now.  We're discussing a different issue, which is the text
properties of the value returned by `completing-read'.

I'm not fundamentally opposed to this change, but I strongly suspect
that any code that relies on those properties will be very brittle since
there is very little control over them.  So, if you can come up with
actual scenarios where those properties could be useful/important, maybe
we can make sure not only that they're returned but that they are
returned reliably.  If you can't come up with such a scenario, that
casts doubts on the usefulness of the requested change.


        Stefan

RE: should read-file-name not respect text properties in its input string?

[Drew Adams]

> > http://lists.gnu.org/archive/html/emacs-devel/2007-01/msg00807.html
> 
> This lists uses of properties in completion candidates (typically when
> displayed in *Completions*).  As mentioned, this should already
> work now.  We're discussing a different issue, which is the text
> properties of the value returned by `completing-read'.
> 
> I'm not fundamentally opposed to this change, but I strongly suspect
> that any code that relies on those properties will be very 
> brittle since there is very little control over them.

No idea what that means. I've used this feature for years, and nothing has
broken.

But I don't really care. I don't want to get into a tug of war over this.

> So, if you can come up with
> actual scenarios where those properties could be 
> useful/important, maybe
> we can make sure not only that they're returned but that they are
> returned reliably.  If you can't come up with such a scenario, that
> casts doubts on the usefulness of the requested change.

I've no desire to belabor this. I'm not selling anything. If you don't want to
do it, please don't.

The general argument is that `completing-read' should be lossless; it should be
only about choosing, not losing info. Why? Why not? There is no reason that it
should lose info.

You agree that it should accept a set of rich structures, display them, and let
you choose from them. So far so good. Why should what you get back be only a
stripped down string instead of the whole structure you chose, including
properties that might not be visible? It's not only about rich display (color
etc.); it's also about choosing possibly complex objects. It's not only visible
properties that are useful.

Using properties lets an application pass along additional information about a
candidate choice, in addition to the string text - not just to *Completions* but
to the `completing-read' caller as the return value. The info can group/classify
candidates, order them, or do anything else you want.

I mentioned that I put that to advantage in Icicles in various contexts. I
associate different kinds of info with candidates, sometimes even candidates
whose appearance is the same: tag info, buffer locations, Info structure,
bookmark info, annotations, internal representations, and so on. 

Are there other ways to associate additional info with string candidates,
besides using text properties? Sure, and I use some of them too. But why not
provide this capability?

If the general rationale (don't lose info) doesn't convince you, and the
suggestions of uses don't inspire you, fuggedabowdit.

Re: should read-file-name not respect text properties in its input string?

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>> So, if you can come up with
>> actual scenarios where those properties could be 
>> useful/important, maybe
>> we can make sure not only that they're returned but that they are
>> returned reliably.  If you can't come up with such a scenario, that
>> casts doubts on the usefulness of the requested change.
>
> I've no desire to belabor this. I'm not selling anything. If you don't want to
> do it, please don't.
>
> The general argument is that `completing-read' should be lossless; it
> should be only about choosing, not losing info. Why? Why not? There is
> no reason that it should lose info.
>
> You agree that it should accept a set of rich structures, display
> them, and let you choose from them.

Uh, _I_ don't agree.  At best, this may be argued for completing-read
with neither nil or confirm-only as REQUIRE-MATCH.  Other than that,
user input can be passed into it that need not match anything in the
completion list.  Then it does not make sense that sometimes information
will be picked from the completions, sometimes not.

So what if there text properties in it, and the user input contains
different text properties?  Is that supposed to be a non-match?  Are the
user input text properties to be thrown away?  Why would you want to
throw the user input information rather than the completion information
away?

> So far so good. Why should what you get back be only a stripped down
> string instead of the whole structure you chose, including properties
> that might not be visible?  It's not only about rich display (color
> etc.); it's also about choosing possibly complex objects. It's not
> only visible properties that are useful.

So why would you throw the properties in the user entry away when they
did not come about by completion, but the user input matches a
completion candidate?

> Using properties lets an application pass along additional information
> about a candidate choice, in addition to the string text - not just to
> *Completions* but to the `completing-read' caller as the return
> value. The info can group/classify candidates, order them, or do
> anything else you want.

Why would it do that if the information was not entered by completion,
but merely matches a completion candidate by chance?

> Are there other ways to associate additional info with string
> candidates, besides using text properties? Sure, and I use some of
> them too. But why not provide this capability?

Because no consistent behavior has been proposed so far?

> If the general rationale (don't lose info) doesn't convince you, and
> the suggestions of uses don't inspire you, fuggedabowdit.

So why lose the info from the keyboard entry and replace it with the
info of a completion candidate in case there is one?

If the completion candidates are not even consulted (like when
REQUIRE-MATCH is nil and TAB never is hit), why would I paste over the
user's text properties?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: should read-file-name not respect text properties in its input string?

[Drew Adams]

> > You agree that it should accept a set of rich structures, display
> > them, and let you choose from them.
> 
> Uh, _I_ don't agree.

"You" meant Stefan, to whom I replied. Richard OK'd that feature, and it's part
of Emacs. Take up your disagreement with Emacs.

> At best, this may be argued for completing-read
> with neither nil or confirm-only as REQUIRE-MATCH.  Other than that,
> user input can be passed into it that need not match anything in the
> completion list.

Read all of what I wrote. With lax completion your input need not match a
candidate, of course.

> Then it does not make sense that sometimes 
> information will be picked from the completions, sometimes not.

If an application depends on picking up additional information from a candidate,
other than just the stripped string, then it will use strict completion. (It
could conceivably somehow expect the user to input the appropriate propertized
text, but that would be unlikely.) 

If an app uses lax completion and user input doesn't match any candidate, then
the app had better not depend on any information that is not in the input.

> So what if there text properties in it, and the user input contains
> different text properties?  Is that supposed to be a 
> non-match?  

Matching is unchanged. Emacs uses ordinary string matching for completion: any
text properties present are ignored for matching. 

There is nothing new in this. `all-completions' and so on have worked with
propertized strings for a long time. And `display-completion-list' has enabled
matching against propertized candidates for a year.

What would be new would be to return the matched candidate, properties and all.

> Are the user input text properties to be thrown away?  Why would you want to
> throw the user input information rather than the completion 
> information away?

See previous. Emacs completion matching always ignores properties, regardless of
which string they belong to (candidate or input).

Nothing that is kept today would be thrown away. This is not about throwing away
information; it's about not throwing away information.

> > So far so good. Why should what you get back be only a stripped down
> > string instead of the whole structure you chose, including 
> > properties that might not be visible?  It's not only about rich
> > display (color etc.); it's also about choosing possibly complex
> > objects. It's not only visible properties that are useful.
> 
> So why would you throw the properties in the user entry away when they
> did not come about by completion, but the user input matches a
> completion candidate?

Never said that.

> > Using properties lets an application pass along additional 
> > information about a candidate choice, in addition to the string
> > text - not just to *Completions* but to the `completing-read'
> > caller as the return value. The info can group/classify
> > candidates, order them, or do anything else you want.
> 
> Why would it do that if the information was not entered by completion,
> but merely matches a completion candidate by chance?

See above. If an app depends on additional info that is provided in completion
candidates, then it will use strict completion.
 
> > Are there other ways to associate additional info with string
> > candidates, besides using text properties? Sure, and I use some of
> > them too. But why not provide this capability?
> 
> Because no consistent behavior has been proposed so far?

Forget it, David.

> > If the general rationale (don't lose info) doesn't convince you, and
> > the suggestions of uses don't inspire you, fuggedabowdit.
> 
> So why lose the info from the keyboard entry and replace it with the
> info of a completion candidate in case there is one?

If your input matches only one candidate, then that candidate is placed in the
minibuffer, and it is what is returned. That is nothing new. The difference is
that if the candidate has properties they are not lost.

If your input matches no candidate and completion is lax, then whatever is in
the minibuffer when you hit RET is returned - however it got in the minibuffer
(yanking, typing, completing plus editing,...). If your input for some reason
has text properties, they are included in the returned string.

Today, in neither case does the returned string have text properties.

> If the completion candidates are not even consulted (like when
> REQUIRE-MATCH is nil and TAB never is hit), why would I paste over the
> user's text properties?

You wouldn't. I never said that would happen.

Anyway, no sense arguing. Don't add this to Emacs.