bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Drew Adams]

In all Emacs releases prior to Emacs 26, if you use command `dired' with
an input directory name that has wildcards and ends with a slash,
e.g. c:/foo/bar/*b*/, the command simply ignores the trailing slash and
correctly gives you a listing of all files and dirs in c:/foo/bar/ whose
names contain a b character.

Starting with Emacs 26, such input raises an error.  IMO it should not.

Other than that, the error message is anyway inappropriate: "No files
matching regexp".  There's absolutely no regep involved.  *b* is a glob
pattern, not a regexp - and so is *b*/, for that matter.

In general, the character / in a glob pattern cannot be matched by a
wildcard.  E.g., Wikipedia says "Normally, the path separator character
(/ on Linux/Unix, MacOS, etc. or \ on Windows) will never be matched."
and Linux man page glob(7) says "A '/' in a pathname cannot be matched
by a '?' or '*' wildcard, or by a range like "[.-0]"."

I think the change in behavior (raising an error) is wrong.  Emacs prior
to Emacs 26 was correct in ignoring a trailing slash in this context.
Please revert the code that introduced this regression.

In GNU Emacs 28.2 (build 2, x86_64-w64-mingw32)
 of 2022-09-13 built on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19044
System Description: Microsoft Windows 10 Pro (v10.0.2009.19044.2364)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> Date: Sat, 14 Jan 2023 22:28:36 +0000
> 
> In all Emacs releases prior to Emacs 26, if you use command `dired' with
> an input directory name that has wildcards and ends with a slash,
> e.g. c:/foo/bar/*b*/, the command simply ignores the trailing slash and
> correctly gives you a listing of all files and dirs in c:/foo/bar/ whose
> names contain a b character.
> 
> Starting with Emacs 26, such input raises an error.  IMO it should not.
> 
> Other than that, the error message is anyway inappropriate: "No files
> matching regexp".

I fixed the message to say "No files matching wildcard" instead.

> In general, the character / in a glob pattern cannot be matched by a
> wildcard.  E.g., Wikipedia says "Normally, the path separator character
> (/ on Linux/Unix, MacOS, etc. or \ on Windows) will never be matched."
> and Linux man page glob(7) says "A '/' in a pathname cannot be matched
> by a '?' or '*' wildcard, or by a range like "[.-0]"."

This is incorrect.  A wildcard like "*b*/" should expand to the list
of directories whose names match "*b*", whereas "*b*" should expand to
the list of files _and_ directories with matching names.  This is how
Dired behaves on Posix platforms (where such an expansion is done by
the 'ls' program), and we want a similar behavior with ls-lisp.

So I've now made ls-lisp on the emacs-29 branch behave like that: a
wildcard that ends in a slash is expanded to the list of matching
directories.  And with that, I'm closing this bug.

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Drew Adams]

> > In all Emacs releases prior to Emacs 26, if you use command `dired'
> > with an input directory name that has wildcards and ends with a slash,
> > e.g. c:/foo/bar/*b*/, the command simply ignores the trailing slash and
> > correctly gives you a listing of all files and dirs in c:/foo/bar/
> > whose names contain a b character.
> >
> > Starting with Emacs 26, such input raises an error.  IMO it should not.
> >
> > Other than that, the error message is anyway inappropriate: "No files
> > matching regexp".
> 
> I fixed the message to say "No files matching wildcard" instead.
> 
> > In general, the character / in a glob pattern cannot be matched by a
> > wildcard.  E.g., Wikipedia says "Normally, the path separator character
> > (/ on Linux/Unix, MacOS, etc. or \ on Windows) will never be matched."
> > and Linux man page glob(7) says "A '/' in a pathname cannot be matched
> > by a '?' or '*' wildcard, or by a range like "[.-0]"."
> 
> This is incorrect.  A wildcard like "*b*/" should expand to the list
> of directories whose names match "*b*", whereas "*b*" should expand to
> the list of files _and_ directories with matching names.  This is how
> Dired behaves on Posix platforms (where such an expansion is done by
> the 'ls' program), and we want a similar behavior with ls-lisp.

Excellent.  That's in fact what I was naively expecting
or hoping for.

But googling for info about what such a glob pattern
should match, I couldn't find anything that supported
such an interpretation.  Could you maybe point me to a
source that covers this - so I can read more and point
others to it?  If you don't have the time to find that
then don't worry about it.  Thx.

> So I've now made ls-lisp on the emacs-29 branch behave like that: a
> wildcard that ends in a slash is expanded to the list of matching
> directories.  And with that, I'm closing this bug.

Sounds good.  Thx.

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "60819-done@debbugs.gnu.org" <60819-done@debbugs.gnu.org>
> Date: Sun, 15 Jan 2023 17:04:05 +0000
> 
> > > In general, the character / in a glob pattern cannot be matched by a
> > > wildcard.  E.g., Wikipedia says "Normally, the path separator character
> > > (/ on Linux/Unix, MacOS, etc. or \ on Windows) will never be matched."
> > > and Linux man page glob(7) says "A '/' in a pathname cannot be matched
> > > by a '?' or '*' wildcard, or by a range like "[.-0]"."
> > 
> > This is incorrect.  A wildcard like "*b*/" should expand to the list
> > of directories whose names match "*b*", whereas "*b*" should expand to
> > the list of files _and_ directories with matching names.  This is how
> > Dired behaves on Posix platforms (where such an expansion is done by
> > the 'ls' program), and we want a similar behavior with ls-lisp.
> 
> Excellent.  That's in fact what I was naively expecting
> or hoping for.
> 
> But googling for info about what such a glob pattern
> should match, I couldn't find anything that supported
> such an interpretation.  Could you maybe point me to a
> source that covers this - so I can read more and point
> others to it?  If you don't have the time to find that
> then don't worry about it.  Thx.

I'm not sure this should be documented in some special way.  Since a
wildcard match cannot cross a slash, it follows that a slash can only
match itself.  So you are looking for files whose names end in a
slash, and those by convention are directories, both in Emacs and with
GNU Coreutils.  Cf directory-name-p and file-name-as-directory.

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Visuwesh]

[ஞாயிறு ஜனவரி 15, 2023] Drew Adams wrote:

>> > In general, the character / in a glob pattern cannot be matched by a
>> > wildcard.  E.g., Wikipedia says "Normally, the path separator character
>> > (/ on Linux/Unix, MacOS, etc. or \ on Windows) will never be matched."
>> > and Linux man page glob(7) says "A '/' in a pathname cannot be matched
>> > by a '?' or '*' wildcard, or by a range like "[.-0]"."
>> 
>> This is incorrect.  A wildcard like "*b*/" should expand to the list
>> of directories whose names match "*b*", whereas "*b*" should expand to
>> the list of files _and_ directories with matching names.  This is how
>> Dired behaves on Posix platforms (where such an expansion is done by
>> the 'ls' program), and we want a similar behavior with ls-lisp.
>
> Excellent.  That's in fact what I was naively expecting
> or hoping for.
>
> But googling for info about what such a glob pattern
> should match, I couldn't find anything that supported
> such an interpretation.  Could you maybe point me to a
> source that covers this - so I can read more and point
> others to it?  If you don't have the time to find that
> then don't worry about it.  Thx.

From OpenBSD's glob(7) manpage [1],

    Note that when matching a pathname, the path separator ‘/’, is not
    matched by a ‘?’, or ‘*’, character or by a “[..]” sequence. Thus,
    /usr/*/*/X11 would match /usr/X11R6/lib/X11 and
    /usr/X11R6/include/X11 while /usr/*/X11 would not match
    either. Likewise, /usr/*/bin would match /usr/local/bin but not
    /usr/bin.

1. http://man.openbsd.org/glob.7

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "60819-done@debbugs.gnu.org"
>  <60819-done@debbugs.gnu.org>
> Date: Sun, 15 Jan 2023 23:15:54 +0530
> 
> >From OpenBSD's glob(7) manpage [1],
> 
>     Note that when matching a pathname, the path separator ‘/’, is not
>     matched by a ‘?’, or ‘*’, character or by a “[..]” sequence. Thus,
>     /usr/*/*/X11 would match /usr/X11R6/lib/X11 and
>     /usr/X11R6/include/X11 while /usr/*/X11 would not match
>     either. Likewise, /usr/*/bin would match /usr/local/bin but not
>     /usr/bin.

Thanks, but that's not the issue at hand.  What Drew wanted to see was
an explicit wording to the effect that a trailing slash makes the
wildcard match only directories.

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Drew Adams]

(Restoring the bug-thread removed from cc.)

> >> >From OpenBSD's glob(7) manpage [1],
> >>  Note that when matching a pathname, the path separator ‘/’, is not
> >>  matched by a ‘?’, or ‘*’, character or by a “[..]” sequence. Thus,
> >>  /usr/*/*/X11 would match /usr/X11R6/lib/X11 and
> >>  /usr/X11R6/include/X11 while /usr/*/X11 would not match
> >>  either. Likewise, /usr/*/bin would match /usr/local/bin but not
> >>  /usr/bin.
> >
> > Thanks, but that's not the issue at hand.  What Drew wanted to see was
> > an explicit wording to the effect that a trailing slash makes the
> > wildcard match only directories.
> 
> Doesn't it follow from the quoted text?
> If * doesn't match a /, then it can't match a directory.  / is not a
> valid character in a filename so dir*/ would only match directories that
> starts with `dir'.

That it follows wasn't, and isn't, obvious to me.
(And I cited similar text from other sources, so
clearly I'd read such descriptions.)

> so dir*/ would only match directories that
  ^^
> starts with `dir'.

"So"?  I don't see how that follows.  Why would
one suppose that it matches directory names at
all?  The glob doc says that `/' in a glob
pattern delimits pattern segments that match
file-name components - nothing more.  What text
says that a directory name that matches a glob
pattern ends with `/'?

*b*a matches file names that include `a' after
what *b* matches.  But `a' is a char that _can_
be matched by a glob pattern, including by a
wildcard.

Can `/'?  It can't match a wildcard, at least.
But can a literal `/' in a glob pattern match a
`/' that's in a file-name component itself (i.e.,
in the text between the directory separators,
which for Emacs are `/')?  Is such a component
even possible?

I suppose so, but to include `/' in a file-name
component that char would have had to be escaped
when creating the file whose name includes it.
Or some other, equivalent means would have had
to be employed.

I'm no expert on whether this is even possible,
or how one might do it (including within Emacs,
and `touch' apparently won't do it).  But let's
assume you _can_ do it: somehow embed `/' in a
file-name component, so it's _part_ of the file
name.  And let's assume you can even do that
at the end of the file name: have the last
file-name component have `/' as its last char.

In such a (rare) case I can see how *b*/ would
match a file name whose last char is `/'.  To
me, that's the only way in which the text you
cited (and the similar text I cited) could be
saying that a glob pattern with `/' chars in
it could actually match those chars against '/'
chars embedded in a file name itself.

But I don't think this rare possibility (if it
is a possibility) is what Eli's talking about.
I don't think he's talking about `/' characters
embedded in a file name.

A `/' at the end of an Emacs absolute file name
isn't within any file-name component (unless -
see previous paragraphs, for a hypothetical
possibility).

I think Eli is saying that for Emacs such a `/'
is part of a directory's file name, i.e., what
(elisp) `Directory Names" calls the “directory
file name”.  I understand this to mean what
function `file-name-as-directory' returns: the
file name considered as directory, which Emacs
writes with a `/' at the end.  (Per POSIX etc.)

To me it wasn't obvious that a glob pattern that
ends with `/' imposes a `file-name-as-directory'
interpretation on candidate matches (but that's
exactly what I wanted Emacs's handling of globs
to do).  I do think it would help for the doc to
point this out, if that's what's meant.  I do.

But I understand that for you (and Eli?) this is
considered obvious.  I'll just say that if so,
then maybe it's a bit odd that the until-now
bugged behavior (existing since Day One or at
least as far back as Emacs 20): (1) existed and
(2) hadn't been reported as a bug.  Those facts
suggest to me that this hasn't been obvious at
all.  Apparently the ls-lisp code never thought
to implement it, and no one noticed that, or at
least never thought it was a bug.

So I still think some mention in the doc would
help.  But in any case, I'm happy to see the
bug fixed, including the faulty error msg.

That's really what I cared about.  And I'm glad
to have learned a thing or two (though I wonder
about the possibility of embedding `/' chars in
file-name components).

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Visuwesh]

[ஞாயிறு ஜனவரி 15, 2023] Drew Adams wrote:

> (Restoring the bug-thread removed from cc.)

[ I intended to remove the "-done" mail but missed that the non-done
  email was in the CCs.  ]

>> >> >From OpenBSD's glob(7) manpage [1],
>> >>  Note that when matching a pathname, the path separator ‘/’, is not
>> >>  matched by a ‘?’, or ‘*’, character or by a “[..]” sequence. Thus,
>> >>  /usr/*/*/X11 would match /usr/X11R6/lib/X11 and
>> >>  /usr/X11R6/include/X11 while /usr/*/X11 would not match
>> >>  either. Likewise, /usr/*/bin would match /usr/local/bin but not
>> >>  /usr/bin.
>> >
>> > Thanks, but that's not the issue at hand.  What Drew wanted to see was
>> > an explicit wording to the effect that a trailing slash makes the
>> > wildcard match only directories.
>> 
>> Doesn't it follow from the quoted text?
>> If * doesn't match a /, then it can't match a directory.  / is not a
>> valid character in a filename so dir*/ would only match directories that
>> starts with `dir'.
>
> That it follows wasn't, and isn't, obvious to me.
> (And I cited similar text from other sources, so
> clearly I'd read such descriptions.)
>
>> so dir*/ would only match directories that
>   ^^
>> starts with `dir'.
>
> "So"?  I don't see how that follows.  Why would
> one suppose that it matches directory names at
> all?  The glob doc says that `/' in a glob
> pattern delimits pattern segments that match
> file-name components - nothing more.  What text
> says that a directory name that matches a glob
> pattern ends with `/'?
>
> [...]
>
> Can `/'?  It can't match a wildcard, at least.
> But can a literal `/' in a glob pattern match a
> `/' that's in a file-name component itself (i.e.,
> in the text between the directory separators,
> which for Emacs are `/')?  Is such a component
> even possible?
>
> I suppose so, but to include `/' in a file-name
> component that char would have had to be escaped
> when creating the file whose name includes it.
> Or some other, equivalent means would have had
> to be employed.
>
> I'm no expert on whether this is even possible,
> or how one might do it (including within Emacs,
> and `touch' apparently won't do it).  But let's
> assume you _can_ do it: somehow embed `/' in a
> file-name component, so it's _part_ of the file
> name.  And let's assume you can even do that
> at the end of the file name: have the last
> file-name component have `/' as its last char.

'/' is not a valid file-name component in POSIX systems [1].  There's no
way to "escape" the forward slash.

> In such a (rare) case I can see how *b*/ would
> match a file name whose last char is `/'.  To
> me, that's the only way in which the text you
> cited (and the similar text I cited) could be
> saying that a glob pattern with `/' chars in
> it could actually match those chars against '/'
> chars embedded in a file name itself.
>
> But I don't think this rare possibility (if it
> is a possibility) is what Eli's talking about.
> I don't think he's talking about `/' characters
> embedded in a file name.
>
> A `/' at the end of an Emacs absolute file name
> isn't within any file-name component (unless -
> see previous paragraphs, for a hypothetical
> possibility).

I understand that Eli isn't talking about / embedded in filenames but
the answer to your original question came naturally from glob(7).  At
least to me, it was an obvious conclusion.

> I think Eli is saying that for Emacs such a `/'
> is part of a directory's file name, i.e., what
> (elisp) `Directory Names" calls the “directory
> file name”.  I understand this to mean what
> function `file-name-as-directory' returns: the
> file name considered as directory, which Emacs
> writes with a `/' at the end.  (Per POSIX etc.)
>
> To me it wasn't obvious that a glob pattern that
> ends with `/' imposes a `file-name-as-directory'
> interpretation on candidate matches (but that's
> exactly what I wanted Emacs's handling of globs
> to do).  I do think it would help for the doc to
> point this out, if that's what's meant.  I do.

Please see above.  In my mind, it is not that glob patterns impose a
restriction to match only directories when the patterns with /, it is
more so that a wildcard pattern cannot / is why `dir*/' only matches
directories that start with "dir".  IOW the restriction is implicit.

> But I understand that for you (and Eli?) this is
> considered obvious.  I'll just say that if so,
> then maybe it's a bit odd that the until-now
> bugged behavior (existing since Day One or at
> least as far back as Emacs 20): (1) existed and
> (2) hadn't been reported as a bug.  Those facts
> suggest to me that this hasn't been obvious at
> all.  Apparently the ls-lisp code never thought
> to implement it, and no one noticed that, or at
> least never thought it was a bug.

Personally, I don't use ls-lisp and I am not a frequent user of glob
patterns but I remember reading this "gotcha" somewhere and promptly
checked the manpage which was a definitive enough documentation in my
mind hence my initial message.

1. https://pubs.opengroup.org/onlinepubs/007904875/basedefs/xbd_chap03.html#tag_03_169

bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26

[Eli Zaretskii]

> From: Visuwesh <visuweshm@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "60819-done@debbugs.gnu.org"
>  <60819-done@debbugs.gnu.org>
> Date: Mon, 16 Jan 2023 09:43:06 +0530
> 
> > To me it wasn't obvious that a glob pattern that
> > ends with `/' imposes a `file-name-as-directory'
> > interpretation on candidate matches (but that's
> > exactly what I wanted Emacs's handling of globs
> > to do).  I do think it would help for the doc to
> > point this out, if that's what's meant.  I do.
> 
> Please see above.  In my mind, it is not that glob patterns impose a
> restriction to match only directories when the patterns with /, it is
> more so that a wildcard pattern cannot / is why `dir*/' only matches
> directories that start with "dir".  IOW the restriction is implicit.

This is not really as clear-cut as you say.  For example, try

  C-x d emacs/src/bitmaps/ RET

as opposed to

  C-x d emacs/src/bitmaps*/ RET

bug#60819: closed (Re: bug#60819: 28.2; `ls-lisp.el' regression introduced in Emacs 26)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I think the fix should go into `file-expand-wildcards` instead.
> E.g. with the patch below.
> [ Not sure it's worth pushing it onto `emacs-29`, OTOH.  ]
> Any objection?

Pushed to `master`.


        Stefan