bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

[Eli Zaretskii <eliz@gnu.org>]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 36490@debbugs.gnu.org,  erik_hahn@gmx.de
> Date: Tue, 09 Jul 2019 19:00:13 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Sorry, that's not what I meant to say.  I meant to say how would a
> > Lisp program know whether (expand-file-name "~/") means the home
> > directory or a directory whose name is literally "~"?
> 
> Well, we have documented that in expand-file-name "~/" means the home
> directory, and I have no problems with that.

Documenting a problem doesn't necessarily solve it.  E.g., it is also
documented that you must quote file names with special characters, but
you still raised the objection that the "~" use case makes that "odd".

I'm saying that the "~/" use case is "odd" as well, and for the same
reasons.

> "~/" isn't something you'll ever get from functions like
> directory-files

That's sheer luck, because:

  (file-name-as-directory "~")
    => "~/"

So just running "~" through an innocent API gives you a "magic"
directory name (if you consider "~" not "magic" by itself).  How is
this different from the "odd" use case where one must quote "~" to
avoid its interpretation as the home directory?  Who can guarantee
that some day directory-files-recursively will not want to do
something like the above?  If it does, we will be right back at the
same problem.

I say we should fix this problem in a way that isn't fragile, and
doesn't crucially depend on what the current code does or avoids
doing.

> > Btw, stuff like (expand-file-name "foo/~/") already does what you
> > want, so the problem is only with the leading '~', and can be avoided
> > if we avoid that situation.  IOW, why should this example:
> >
> >   (expand-file-name "~" "/tmp/")
> >      => "/home/larsi"
> >
> > determine how directory-files-recursively behaves?
> 
> expand-file-name's use case is to (basically) concatenate a directory
> name and a file name, but it's used instead of concat because nobody
> wants to care about whether the directory name has a trailing slash or
> not.

Ah, but when the file name begins with a "~", the "concatenation" does
more than what meets the eye.

> That's basically the use case for expand-file-name, and using it has
> avoided a lot of basic concatenation problems over the years (because
> Emacs allows sloppy handling of directory file names in most
> situations).

I think this is a simplification.  It ignores the fact that
expand-file-name interprets ~/, it ignores the fact that it does
arbitrary stuff for "remote" file names, it ignores the fact that on
Windows it prepends the drive letter if there isn't one already, etc.
IOW, expand-file-name is concatenation-like, but it has a few tricks
up its sleeve, and in this case the trick works against us.  We need
to disable that trick to support files and directories whose names
begin with a literal "~".  I see no way around that.