substitute-in-file-name and "$"

substitute-in-file-name and "$"

[Michael Albinus]

Hello,

In file name operations, the character "$" is masked by "$$" in order
to pass substitute-in-file-name handling. Unfortunately, this works
only onces, because after the first call of substitute-in-file-name,
"$$" reduces to "$".

There is no rule that substitute-in-file-name must be called only
once. For example, if you use partial-completion-mode, the filename in
the minibuffer will be passed through substitute-in-file-name by
PC-do-completion. Later file name operations which call internally
substitute-in-file-name will fail therefore.

This happens permanently in tramp-smb.el, where "$" is a part of file
names of default shares.

As workaround, I've written a defadvice for PC-do-completion which
requotes "$", but this is far from being perfect. Better would be a
general solution.

What I'ld like to know: Is it a general rule, that
substitute-in-file-name can be called only once for a given file name?
Then it should be documented in the Elisp manual, and it is a bug in
PC-do-completion. Or is it a more general design problem?

Best regards, Michael.

Re: substitute-in-file-name and "$"

[Luc Teirlinck]

Michael Albinus wrote:

   There is no rule that substitute-in-file-name must be called only
   once.

Even if this rule is not stated explicitly, this just seems to be
common sense.  It would appear that this is just a special case of the
general rule that one should avoid bugs.

   For example, if you use partial-completion-mode, the filename in
   the minibuffer will be passed through substitute-in-file-name by
   PC-do-completion. Later file name operations which call internally
   substitute-in-file-name will fail therefore.

To me, it would seem obvious that this is a bug in `PC-do-completion'.
It can not handle filenames containing a literal `$'.

You say "for example".  Are you aware of tons of similar bugs
throughout the Emacs code or is `PC-do-completion' an isolated
example?

   What I'ld like to know: Is it a general rule, that
   substitute-in-file-name can be called only once for a given file name?
   Then it should be documented in the Elisp manual, and it is a bug in
   PC-do-completion. Or is it a more general design problem?

It might actually be good to mention this potential trap in the Elisp
manual.  However, this is a matter of avoiding bugs, not a matter of
following some stylistic "rule".

Sincerely,

Luc.

Re: substitute-in-file-name and "$"

[Michael Albinus]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Michael Albinus wrote:
>
>    There is no rule that substitute-in-file-name must be called only
>    once.
>
> Even if this rule is not stated explicitly, this just seems to be
> common sense.  It would appear that this is just a special case of the
> general rule that one should avoid bugs.

At least the examples in the Elisp manual give the feeling that
substitute-in-file-name could be called without special attention. The
trap with the "$" char isn't obvious.

>    For example, if you use partial-completion-mode, the filename in
>    the minibuffer will be passed through substitute-in-file-name by
>    PC-do-completion. Later file name operations which call internally
>    substitute-in-file-name will fail therefore.
>
> To me, it would seem obvious that this is a bug in `PC-do-completion'.
> It can not handle filenames containing a literal `$'.

It takes any file name you offer. But at least it should perform a
special handling for "$" (requoting), as Stefan mentioned already in
the original thread in gnu.emacs.help.

> You say "for example".  Are you aware of tons of similar bugs
> throughout the Emacs code or is `PC-do-completion' an isolated
> example?

At least I've done the same bug in tramp-smb.el (corrected). No, for
the time being that's the only case I know. But I fear that there are
other cases, because occasionally I get such reports for tramp-smb I
couldn't catch the origin. I'll investigate harder.

>    What I'ld like to know: Is it a general rule, that
>    substitute-in-file-name can be called only once for a given file name?
>    Then it should be documented in the Elisp manual, and it is a bug in
>    PC-do-completion. Or is it a more general design problem?
>
> It might actually be good to mention this potential trap in the Elisp
> manual.  However, this is a matter of avoiding bugs, not a matter of
> following some stylistic "rule".

Not only a stylistic "rule" but a design constraint. So it is worth a
note in the manual.

And, btw, if substitute-in-file-name can be called only once, how do I
know it where to perform this call? And what shall the others do, if
they would like to perform substitute-in-file-name?  Requote "$"? Or
apply an optional parameter (to be introduced), which forces this?

> Sincerely,
>
> Luc.

Best regards, Michael.

Re: substitute-in-file-name and "$"

[Luc Teirlinck]

Michael Albinus wrote:

   At least the examples in the Elisp manual give the feeling that
   substitute-in-file-name could be called without special attention. The
   trap with the "$" char isn't obvious.

None of the examples in the Elisp manual, (elisp)File Name Expansion,
involves Elisp variables.  Even when working with variables, it would
seem that you can call (substitute-in-file-name filename) 1001 times
without danger (it has no side effects).  Danger arises from stuff
like:

1. (setq filename (substitute-in-file-name filename))

or

2. (my-favorite-file-function (substitute-in-file-name filename))

The Elisp manual could point out that (1) should only be done if the
user really knows what he is doing.  Usually, using the return value
of (substitute-in-file-name filename) only at the time it is actually
needed (and _not_ changing filename itself at that time) or doing:

(1b) (setq substituted-filename (substitute-in-file-name filename))

will be safer.

In as far as (2) is concerned, the documentation string of
`my-favorite-file-function' should clearly point out what operations
it is going to perform on filename.  If not, that is a bug in the
documentation string.  In that case, the Elisp user could try to
figure it out by experimentation and file a bug report.  If
`my-favorite-file-function' is going to pass filename on to other
functions it calls, it should exercise a similar care itself.

   And, btw, if substitute-in-file-name can be called only once, how do I
   know it where to perform this call? And what shall the others do, if
   they would like to perform substitute-in-file-name?  Requote "$"? Or
   apply an optional parameter (to be introduced), which forces this?

Do you have examples where there is a real need to do (1) instead of
(1b)?  I have a hard time thinking of examples in the abstract, let
alone of examples occurring in actual situations.  Unquoting and then
requoting seems like a senseless circular operation to me.

Sincerely,

Luc.

Re: substitute-in-file-name and "$"

[Miles Bader]

On Sun, Jul 06, 2003 at 01:16:44AM +0200, Michael Albinus wrote:
> At least the examples in the Elisp manual give the feeling that
> substitute-in-file-name could be called without special attention. The
> trap with the "$" char isn't obvious.

Consider that the environment-variable's _value_ could contain a `$', so it
seems actually fairly obvious that you should only call
substitute-in-file-name once...

-Miles
-- 
Next to fried food, the South has suffered most from oratory.
  			-- Walter Hines Page

Re: substitute-in-file-name and "$"

[Kim F. Storm]

Miles Bader <miles@gnu.org> writes:

> On Sun, Jul 06, 2003 at 01:16:44AM +0200, Michael Albinus wrote:
> > At least the examples in the Elisp manual give the feeling that
> > substitute-in-file-name could be called without special attention. The
> > trap with the "$" char isn't obvious.
> 
> Consider that the environment-variable's _value_ could contain a `$', so it
> seems actually fairly obvious that you should only call
> substitute-in-file-name once...

Maybe substitute-in-file-name could put a `substitute-in-file-name'
property on the return value and check for that property on its
argument (and do nothing if it is present).

A later call to substitute-in-file-name should then skip over parts of
its argument string which has that property (and only substitute in
parts which are not previously substituted.

That is still not completely fool-proof if the result of
substitute-in-file-name is manipulated in some way which removes the
property and then later passed to substitute-in-file-name again.


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: substitute-in-file-name and "$"

[Kai Großjohann]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> In as far as (2) is concerned, the documentation string of
> `my-favorite-file-function' should clearly point out what operations
> it is going to perform on filename.

IMHO this is dangerous.  Now you are saying that docstrings should
specify "I invoke substitute-in-file-name" or "I do not invoke
substitute-in-file-name".  Tomorrow, docstrings should say "I
invoke/do not invoke mumblefoo".

Clearly, down that path lies madness.

I think that some policy is called for.  Alas, I don't have the
foggiest what that policy ought to look like.

Another alternative is to change things such that
substitute-in-file-name is idempotent.
-- 
~/.signature

Re: substitute-in-file-name and "$"

[Miles Bader]

On Sun, Jul 06, 2003 at 02:20:35PM +0200, Kai Gro?johann wrote:
> Now you are saying that docstrings should specify "I invoke
> substitute-in-file-name" or "I do not invoke substitute-in-file-name".
> 
> Clearly, down that path lies madness.

Hardly.

Whether or not a function calls substitute-in-file-name makes a fairly
substantial difference in the result of the function.  Since the docstring is
supposed to say _what the function does_, it should surely mention if it
expands environment variables or not (even if substitute-in-file-name were
changed to be idempotent).

Since most functions don't do such expansion, of course, only the ones that
do need mention it.

-Miles
-- 
In New York, most people don't have cars, so if you want to kill a person, you
have to take the subway to their house.  And sometimes on the way, the train
is delayed and you get impatient, so you have to kill someone on the subway.
  [George Carlin]

Re: substitute-in-file-name and "$"

[Michael Albinus]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> None of the examples in the Elisp manual, (elisp)File Name Expansion,
> involves Elisp variables.  Even when working with variables, it would
> seem that you can call (substitute-in-file-name filename) 1001 times

That's right.

> without danger (it has no side effects).  Danger arises from stuff
> like:
>
> 1. (setq filename (substitute-in-file-name filename))
>
> or
>
> 2. (my-favorite-file-function (substitute-in-file-name filename))
>
> The Elisp manual could point out that (1) should only be done if the
> user really knows what he is doing.  Usually, using the return value
> of (substitute-in-file-name filename) only at the time it is actually
> needed (and _not_ changing filename itself at that time) or doing:
>
> (1b) (setq substituted-filename (substitute-in-file-name filename))
>
> will be safer.

That's right, too.

> In as far as (2) is concerned, the documentation string of
> `my-favorite-file-function' should clearly point out what operations
> it is going to perform on filename.  If not, that is a bug in the
> documentation string.  In that case, the Elisp user could try to
> figure it out by experimentation and file a bug report.  If
> `my-favorite-file-function' is going to pass filename on to other
> functions it calls, it should exercise a similar care itself.

That wouldn't be possible. You know what `my-favorite-file-function'
is doing the default case. You don't know what's going on with an
alternative implementation within another file name handler. Would you
like to say for every file name operation "substitute-in file-name is
called" (or not)? And even if you do it, you cannot control, because
it could be called indirectly, within another file name function you
use.

>    And, btw, if substitute-in-file-name can be called only once, how do I
>    know it where to perform this call? And what shall the others do, if
>    they would like to perform substitute-in-file-name?  Requote "$"? Or
>    apply an optional parameter (to be introduced), which forces this?
>
> Do you have examples where there is a real need to do (1) instead of
> (1b)?  I have a hard time thinking of examples in the abstract, let
> alone of examples occurring in actual situations.  Unquoting and then
> requoting seems like a senseless circular operation to me.

`PC-do-completion' must do it, because the result is shown in the
minibuffer.

Or a file name like this: "$NEXT_HOP:/share$$", where $NEXT_HOP has
the value "/smb:next.hop.com". At least case (2) would be needed,
because Tramp file name handler could be activated only after applying
substitute-in-file-name to that file name.

> Sincerely,
>
> Luc.

Best regards, Michael.

Re: substitute-in-file-name and "$"

[Michael Albinus]

storm@cua.dk (Kim F. Storm) writes:

> Maybe substitute-in-file-name could put a `substitute-in-file-name'
> property on the return value and check for that property on its
> argument (and do nothing if it is present).
>
> A later call to substitute-in-file-name should then skip over parts of
> its argument string which has that property (and only substitute in
> parts which are not previously substituted.
>
> That is still not completely fool-proof if the result of
> substitute-in-file-name is manipulated in some way which removes the
> property and then later passed to substitute-in-file-name again.

A simpler approach would be: expand environment variables if
possible. Don't worry if you cannot expand.

The masquing with "$$" wouldn't be necessary this case. But I don't
know whether there are other drawbacks with this approach.

Best regards, Michael.

Re: substitute-in-file-name and "$"

[Luc Teirlinck]

Michael Albinus wrote:

   `PC-do-completion' must do it, because the result is shown in the
   minibuffer.

   Or a file name like this: "$NEXT_HOP:/share$$", where $NEXT_HOP has
   the value "/smb:next.hop.com". At least case (2) would be needed,
   because Tramp file name handler could be activated only after applying
   substitute-in-file-name to that file name.

You can always double up all $'s if your function only has access to
an already substituted filename.  (Putting a /: at the beginning would
be easier, but that could probably cause remote filenames to be
considered local.)  All functions have to make clear in their
documentation string what they are going to do to file names (if
anything) and in which form they return them, if non-standard.  If
they do not, that is a bug.  This is not madness.  It is completely
obvious.  Functions need to say what they do.  People who use
functions need to know exactly what they do, not more or less what
they do.  It is usually easier to be able to find this out by reading
documentation strings than by carefully reading through every single
line of code of the function itself and of all indirectly called
functions.

   A simpler approach would be: expand environment variables if
   possible. Don't worry if you cannot expand.

   The masquing with "$$" wouldn't be necessary this case. But I don't
   know whether there are other drawbacks with this approach.

What if somebody has an environment variable FOO expanding to bar and
another file named $FOO?  Several operating systems, including GNU and
Unix, allow people to essentially use any file names of their
choosing, no matter how perverse.  Programs have to be able to handle
that, even though that might make life complicated.

Sincerely,

Luc.

Re: substitute-in-file-name and "$"

[Kai Großjohann]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> What if somebody has an environment variable FOO expanding to bar and
> another file named $FOO?  Several operating systems, including GNU and
> Unix, allow people to essentially use any file names of their
> choosing, no matter how perverse.  Programs have to be able to handle
> that, even though that might make life complicated.

IMHO, the best approach is to have substitute-in-file-name return
"$$" for dollar signs in file names and to have all file primitives
grok this notation.

However, it suffers from being backward-incompatible.

Maybe the primitives could grok single dollar signs, too, then the
incompatibility would be restricted to file names with two adjacent
dollar signs.  Let's hope these are not too common.
-- 
~/.signature

Re: substitute-in-file-name and "$"

[Stefan Monnier]

> A simpler approach would be: expand environment variables if
> possible. Don't worry if you cannot expand.

That's already the case (in Emacs-CVS).


	Stefan

Re: substitute-in-file-name and "$"

[Stefan Monnier]

> > > At least the examples in the Elisp manual give the feeling that
> > > substitute-in-file-name could be called without special attention. The
> > > trap with the "$" char isn't obvious.
> > 
> > Consider that the environment-variable's _value_ could contain a `$', so it
> > seems actually fairly obvious that you should only call
> > substitute-in-file-name once...
> 
> Maybe substitute-in-file-name could put a `substitute-in-file-name'
> property on the return value and check for that property on its
> argument (and do nothing if it is present).

This definitely qualifies as over-engineering.
substitute-in-file-name should be used very rarely anyway (in terms of
number of times it shows up in source code), so it's not a big deal.
Most uses of substitute-in-file-name are hidden inside read-file-name.
I'm curious to know which other cases exist.

substitute-in-file-name is basically the function that turns "user-typed"
file names into their "internal" representation, so obviously it should
be applied exactly once.  As for having a filename that's made up of a
user-typed part and an internal part, that's also most likely a bug: the
user-typed part should be passed through substitute-in-file-name before
being combined with the internal part (hopefully with expand-file-name).


	Stefan

Re: substitute-in-file-name and "$"

[Richard Stallman]

    What I'ld like to know: Is it a general rule, that
    substitute-in-file-name can be called only once for a given file name?

Yes, I think that is the idea.  I will document that.

    Then it should be documented in the Elisp manual, and it is a bug in
    PC-do-completion. Or is it a more general design problem?

I think this is a bug in PC-do-completion.  Would someone like
to fix it?

Re: substitute-in-file-name and "$"

[Stefan Monnier]

> Richard Stallman <rms@gnu.org> writes:
> 
> >     What I'ld like to know: Is it a general rule, that
> >     substitute-in-file-name can be called only once for a given file name?
> > 
> > Yes, I think that is the idea.  I will document that.
> > 
> >     Then it should be documented in the Elisp manual, and it is a bug in
> >     PC-do-completion. Or is it a more general design problem?
> > 
> > I think this is a bug in PC-do-completion.  Would someone like
> > to fix it?
> 
> I think ido could have similar problems related to completion (it must
> expand things like $DIR to be able to present the user with the possible
> completions in that directory).  How do you suggest it is fixed?
> 
> I see only one possibility -- perform the substitution to make the
> completion list, but don't substitute it in the actual file-name read
> (as it does now).
> 
> Is that the correct approach?

The way read-file-name-internal does it, the filename is passed to
substitute-in-file-name (to get a filename that you can pass to file
operations) and before returning the result all $ are doubled (using
double_dollars in src/fileio.c).

But for ido you probably want to do it differently because you can't
wait for an explicit TAB or SPC from the user.
Problem is that it can be difficult to relate the output of
substitute-in-file-name to its input.


	Stefan

Re: substitute-in-file-name and "$"

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     What I'ld like to know: Is it a general rule, that
>     substitute-in-file-name can be called only once for a given file name?
> 
> Yes, I think that is the idea.  I will document that.
> 
>     Then it should be documented in the Elisp manual, and it is a bug in
>     PC-do-completion. Or is it a more general design problem?
> 
> I think this is a bug in PC-do-completion.  Would someone like
> to fix it?

I think ido could have similar problems related to completion (it must
expand things like $DIR to be able to present the user with the possible
completions in that directory).  How do you suggest it is fixed?

I see only one possibility -- perform the substitution to make the
completion list, but don't substitute it in the actual file-name read
(as it does now).

Is that the correct approach?


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: substitute-in-file-name and "$"

[Richard Stallman]

    Maybe substitute-in-file-name could put a `substitute-in-file-name'
    property on the return value and check for that property on its
    argument (and do nothing if it is present).

I don't like this idea.  I think it is simple enough to decide which
functions should call it, and which ones should redouble $ after
calling it.  That is simpler, so let's do that.

Re: substitute-in-file-name and "$"

[Michael Albinus]

"Stefan Monnier" <monnier+gnu/emacs@cs.yale.edu> writes:

> substitute-in-file-name should be used very rarely anyway (in terms of
> number of times it shows up in source code), so it's not a big deal.
> Most uses of substitute-in-file-name are hidden inside read-file-name.
> I'm curious to know which other cases exist.

Just grepped Emacs 21.3 tree (sorry, no CVS access behind company firewall):

lisp/calendar/diary-lib.el
lisp/comint.el
lisp/complete.el
lisp/emulation/vip.el
lisp/emulation/viper-cmd.el
lisp/emulation/viper-util.el
lisp/ffap.el
lisp/files.el
lisp/gud.el
lisp/info.el
lisp/mail/mailabbrev.el
lisp/mail/rmail.el
lisp/novice.el
lisp/obsolete/rnews.el
lisp/play/fortune.el
lisp/shell.el
lisp/term.el
lisp/woman.el
lisp/xscheme.el
src/fileio.c
src/lread.c
src/minibuf.c

And, of course, Tramp.

Re: substitute-in-file-name and "$"

[Stefan Monnier]

> > substitute-in-file-name should be used very rarely anyway (in terms of
> > number of times it shows up in source code), so it's not a big deal.
> > Most uses of substitute-in-file-name are hidden inside read-file-name.
> > I'm curious to know which other cases exist.
> 
> Just grepped Emacs 21.3 tree (sorry, no CVS access behind company firewall):

Yes, I also grepped, but that doesn't tell me what these uses are for.


	Stefan

Re: substitute-in-file-name and "$"

[Michael Albinus]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Michael Albinus wrote:
>
>    `PC-do-completion' must do it, because the result is shown in the
>    minibuffer.
>
>    Or a file name like this: "$NEXT_HOP:/share$$", where $NEXT_HOP has
>    the value "/smb:next.hop.com". At least case (2) would be needed,
>    because Tramp file name handler could be activated only after applying
>    substitute-in-file-name to that file name.
>
> You can always double up all $'s if your function only has access to
> an already substituted filename.  (Putting a /: at the beginning would

But you even don't know that. My original problem came from the fact
that, if a user applies partial completion, the file name is already
substituted, and in the other case it isn't.

Meanwhile, I tend to regard it as a bug in `PC-do-completion', like
the others.

> be easier, but that could probably cause remote filenames to be
> considered local.)  All functions have to make clear in their
> documentation string what they are going to do to file names (if
> anything) and in which form they return them, if non-standard.  If
> they do not, that is a bug.  This is not madness.  It is completely
> obvious.  Functions need to say what they do.  People who use
> functions need to know exactly what they do, not more or less what
> they do.  It is usually easier to be able to find this out by reading
> documentation strings than by carefully reading through every single
> line of code of the function itself and of all indirectly called
> functions.

For file name operations, this would be even an implementation
restriction whether they can apply substitute-in-file-name, or not.

More general, it might be said under which circumstances it is
harmless to apply substitute-in-file-name (for local cases, where the
result of substitution isn't returned by the function), and when not.

> Sincerely,
>
> Luc.

Re: substitute-in-file-name and "$"

[Michael Albinus]

"Stefan Monnier" <monnier+gnu/emacs@cs.yale.edu> writes:

> Yes, I also grepped, but that doesn't tell me what these uses are for.

Hmm. I've grepped now for both substitute-in-file-name and
file-name-handler-alist in order to get better candidates. ffap.el
seems to be a problem.

Touched a file '\$abc' in my home directory. Started "emacs-21.3 -q
--no-init-file -f ffap-bindings" (no Tramp, or so). Loaded home
directory in dired, point at file '$abc.el'. After "C-x C-f <TAB>" I
get the error message

    ffap-read-file-or-url-internal: Substituting nonexistent environment variable "abc"

> 	Stefan

Best regards, Michael.

Re: substitute-in-file-name and "$"

[Kevin Rodgers]

Kim F. Storm wrote:

> Maybe substitute-in-file-name could put a `substitute-in-file-name'
> property on the return value and check for that property on its
> argument (and do nothing if it is present).


That is an excellent solution.


> A later call to substitute-in-file-name should then skip over parts of
> its argument string which has that property (and only substitute in
> parts which are not previously substituted.
> 
> That is still not completely fool-proof if the result of
> substitute-in-file-name is manipulated in some way which removes the
> property and then later passed to substitute-in-file-name again.

But as long as file-name-directory, file-name-nondirectory, expand-file-name,

etc. preserved text properties, the programmer would have to intentionally

circumvent the feature.  Unfortunately, those functions seem to remove text
properties from their result (in the Emacs 21.3 release).

-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: substitute-in-file-name and "$"

[Richard Stallman]

    But as long as file-name-directory, file-name-nondirectory,
    expand-file-name, etc. preserved text properties, the programmer
    would have to intentionally circumvent the feature.
    Unfortunately, those functions seem to remove text properties from
    their result (in the Emacs 21.3 release).

It would be quite a pain to make them preserve text properties
in general.  Anyway, I think we already have the right solution
to this problem.

Re: substitute-in-file-name and "$"

[Richard Stallman]

    Touched a file '\$abc' in my home directory. Started "emacs-21.3 -q
    --no-init-file -f ffap-bindings" (no Tramp, or so). Loaded home
    directory in dired, point at file '$abc.el'. After "C-x C-f <TAB>" I
    get the error message

	ffap-read-file-or-url-internal: Substituting nonexistent environment variable "abc"

    > 	Stefan

Would someone like to work on fixing ffap?