bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Drew Adams]

The behavior seems wrong.  If so, the regression was introduced in
Emacs 24.5.

emacs -Q

I evaluate these two sexps:

(file-accessible-directory-p
"D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")

(directory-files
"D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")

The first returns t.  In Emacs prior to Emacs 24.5, the second returns
nil.  In Emacs 24.5 and later I get this error:

Debugger entered--Lisp error: (file-error "Opening directory"
 "Permission denied"
 "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
* directory-files(
"D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001"
 FULL
 "^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*")

Evaluating `file-attributes' for the same file returns this:

(t 1 37786 513
   (20885 54199 0 0)
   (20885 54199 0 0)
   (20885 54199 0 0)
   0 "drwxrwxrwx" t 0 506428215)

In GNU Emacs 25.0.50.1 (i686-pc-mingw32)
 of 2015-07-31 on LEG570
Bzr revision: 8d332aeccab2208e6c6bd434738565e6abf12043
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --host=i686-pc-mingw32 --enable-checking=yes,glyphs'

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Eli Zaretskii]

> Date: Tue, 25 Aug 2015 10:48:10 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> The behavior seems wrong.  If so, the regression was introduced in
> Emacs 24.5.

No, Emacs 24.5 fixed a bug that existed in previous versions.

> emacs -Q
> 
> I evaluate these two sexps:
> 
> (file-accessible-directory-p
> "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> 
> (directory-files
> "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> 
> The first returns t.  In Emacs prior to Emacs 24.5, the second returns
> nil.

The latter result was a clear bug.

> In Emacs 24.5 and later I get this error:
> 
> Debugger entered--Lisp error: (file-error "Opening directory"
>  "Permission denied"
>  "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> * directory-files(
> "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001"
>  FULL
>  "^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*")

This is the correct behavior in this case: you cannot access that
directory (it is private to another user).  Emacs 24.4 and before
would silently return nil there, which is obviously bogus (there's no
directory, except the root, without at least 2 entries, so what the
heck does nil mean here?).

I'm guessing that you cannot display that directory's contents from
the Windows Explorer as well.

> Evaluating `file-attributes' for the same file returns this:
> 
> (t 1 37786 513
>    (20885 54199 0 0)
>    (20885 54199 0 0)
>    (20885 54199 0 0)
>    0 "drwxrwxrwx" t 0 506428215)

What does the following return?

  (file-extended-attributes "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")

Anyway, Windows has an elaborate file access permissions mechanism
that is not reflected in the Posix-style APIs used by
file-accessible-directory-p and file-attributes.  So you have this
inconsistency.  That is unfortunate, but since Windows doesn't have
any reliable API that would allow us to perform the accessibility
test, we cannot really improve that situation.

IOW, on Windows you can never be sure you can access a file or
directory until you actually try.  Fortunately, there are only a few
directories on a typical Windows system where you really cannot access
files.  So this problem is rarely seen in practice.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Drew Adams]

> > In Emacs 24.5 and later I get this error:
> > Debugger entered--Lisp error: (file-error "Opening directory"
> >  "Permission denied"
> >  "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> 
> This is the correct behavior in this case: you cannot access that
> directory (it is private to another user).

Then why does (file-accessible-directory-p
"D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
return t?  The doc for that function says that it returns t if
I can open the directory.

> > Evaluating `file-attributes' for the same file returns this:
> >
> > (t 1 37786 513
> >    (20885 54199 0 0)
> >    (20885 54199 0 0)
> >    (20885 54199 0 0)
> >    0 "drwxrwxrwx" t 0 506428215)
> 
> What does the following return?
> 
>   (file-extended-attributes "D:/$RECYCLE.BIN/S-1-5-21-3120201979-
> 235963886-2582836866-1001")

((acl . "O:S-1-5-21-3138815620-4253048750-3916773603-37786G:S-1-5-21-3120201979-235963886-2582836866-513D:P(A;OICI;FA;;;BA)(A;OICI;FA;;;SY)(A;OICI;FA;;;S-1-5-21-3120201979-235963886-2582836866-1001)")
 (selinux-context nil nil nil nil))

What does that tell you?  Does it say who the mysterious other user
is, to whom the directory is private?

> Anyway, Windows has an elaborate file access permissions mechanism
> that is not reflected in the Posix-style APIs used by
> file-accessible-directory-p and file-attributes.  So you have this
> inconsistency.  That is unfortunate, but since Windows doesn't have
> any reliable API that would allow us to perform the accessibility
> test, we cannot really improve that situation.

I see.  What would you suggest, in that case, for filtering out
such directories, so the error is not raised?

I can wrap the call in `ignore-errors' or equivalent, I suppose.
But at that point it might be too late (depending on the context).

And given this "inconsistency", don't you think this gotcha should be
mentioned in the doc - of `file-accessible-directory-p', for example?
That predicate would seem to be unusable as a general test for access
to a directory.  IOW, what it claims it does is hardly what it does,
apparently.

And perhaps we need another predicate that actually does what this
one says it does, by trying to access the directory?  Somewhat akin
to `file-remote-p' versus `ffap-file-remote-p'.

> IOW, on Windows you can never be sure you can access a file or
> directory until you actually try.  Fortunately, there are only a few
> directories on a typical Windows system where you really cannot
> access files.  So this problem is rarely seen in practice.

Can you characterize those "few directories"?  Perhaps they can be
described in the doc.  Or perhaps `file-accessible-directory-p' can
be taught to return nil for them.

---

BTW, the doc string of `file-accessible-p' says nothing about what
it returns if file FILENAME does NOT name a directory you can open.
It should, I think, say what it does (returns nil, raises an error,
writes to the the Secretary General of the UN, whatever).

Similarly, the manual does not say explicitly that it returns nil
if the dir is not accessible, but one can surmise that from the
fact that it shows an example where it returns nil and the text
says that this implies that attempting to access the directory
raises an error.  Still, like the doc string, it could be clearer.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Eli Zaretskii]

> Date: Tue, 25 Aug 2015 11:34:54 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21346@debbugs.gnu.org
> 
> > > In Emacs 24.5 and later I get this error:
> > > Debugger entered--Lisp error: (file-error "Opening directory"
> > >  "Permission denied"
> > >  "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> > 
> > This is the correct behavior in this case: you cannot access that
> > directory (it is private to another user).
> 
> Then why does (file-accessible-directory-p
> "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001")
> return t?

I explained that later in my response.

> The doc for that function says that it returns t if I can open the
> directory.

On Windows, that test is not reliable enough.

> > What does the following return?
> > 
> >   (file-extended-attributes "D:/$RECYCLE.BIN/S-1-5-21-3120201979-
> > 235963886-2582836866-1001")
> 
> ((acl . "O:S-1-5-21-3138815620-4253048750-3916773603-37786G:S-1-5-21-3120201979-235963886-2582836866-513D:P(A;OICI;FA;;;BA)(A;OICI;FA;;;SY)(A;OICI;FA;;;S-1-5-21-3120201979-235963886-2582836866-1001)")
>  (selinux-context nil nil nil nil))
> 
> What does that tell you?

That the file is indeed a private one.

> Does it say who the mysterious other user is, to whom the directory
> is private?

That's not what I wanted to see there.  If you want to know who is its
owner, try this:

  (file-attributes "D:/$RECYCLE.BIN/S-1-5-21-3120201979-235963886-2582836866-1001" 'string)

The name of the owner will show up as the 3rd element of the list it
returns, and the name of the group (I'm guessing "None") as the 4th
element.  (Unless file-attributes also signals an error.)

> > Anyway, Windows has an elaborate file access permissions mechanism
> > that is not reflected in the Posix-style APIs used by
> > file-accessible-directory-p and file-attributes.  So you have this
> > inconsistency.  That is unfortunate, but since Windows doesn't have
> > any reliable API that would allow us to perform the accessibility
> > test, we cannot really improve that situation.
> 
> I see.  What would you suggest, in that case, for filtering out
> such directories, so the error is not raised?
> 
> I can wrap the call in `ignore-errors' or equivalent, I suppose.
> But at that point it might be too late (depending on the context).

Catching and ignoring errors is the only way, I think.

> And given this "inconsistency", don't you think this gotcha should be
> mentioned in the doc - of `file-accessible-directory-p', for example?

Maybe.  I'll have to think of a useful way of describing this.

> That predicate would seem to be unusable as a general test for access
> to a directory.  IOW, what it claims it does is hardly what it does,
> apparently.

It's not unusable.  It checks the read-only bit (and the executable
bit for files).

> And perhaps we need another predicate that actually does what this
> one says it does, by trying to access the directory?

It would be too expensive to be useful, I think.  Just trying to read
it, like you did, is good enough.

> > IOW, on Windows you can never be sure you can access a file or
> > directory until you actually try.  Fortunately, there are only a few
> > directories on a typical Windows system where you really cannot
> > access files.  So this problem is rarely seen in practice.
> 
> Can you characterize those "few directories"?

Mostly, those under C:/Users/, and also in the recycle bin.  Also, a
small number of system directories on C:.

> BTW, the doc string of `file-accessible-p' says nothing about what
> it returns if file FILENAME does NOT name a directory you can open.

Anything but t means it's not accessible.  I think this much is clear
from the doc string.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Drew Adams]

> > The doc for that function says that it returns t if I can open the
> > directory.
> 
> On Windows, that test is not reliable enough.

The doc should say that, IMO.

> That's not what I wanted to see there.  If you want to know who is
> its owner, try this:
>   (file-attributes "D:/$RECYCLE.BIN/S-1-5-21-..." 'string)

(t 1 "ME" "Domain Users"
   (20885 54199 0 0)
   (20885 54199 0 0)
   (20885 54199 0 0)
   0 "drwxrwxrwx" t 0 506428215)

Where ME is my user name.  ME is also the value of `user-real-login-name'.
In what way does this return value indicate that I do not (user ME does
not) have access to the file?

The doc of `file-attributes' does not hint (to me) anything about
such a value indicating that the file is private.

> The name of the owner will show up as the 3rd element of the list it
> returns, and the name of the group (I'm guessing "None") as the 4th
> element.  (Unless file-attributes also signals an error.)

So I am apparently the owner, and the group is Domain Users, to which
I belong.  Where do we see indicated the fact that I do not have
real access to the file?

> > I see.  What would you suggest, in that case, for filtering out
> > such directories, so the error is not raised?
> >
> > I can wrap the call in `ignore-errors' or equivalent, I suppose.
> > But at that point it might be too late (depending on the context).
> 
> Catching and ignoring errors is the only way, I think.
> 
> > And given this "inconsistency", don't you think this gotcha should
> > be mentioned in the doc - of `file-accessible-directory-p', for
> > example?
> 
> Maybe.  I'll have to think of a useful way of describing this.

Perhaps what you said above, if nothing more informative can be found:

  On Windows, that test is not reliable enough.

> > That predicate would seem to be unusable as a general test for
> > access to a directory.  IOW, what it claims it does is hardly
> > what it does, apparently.
> 
> It's not unusable.  It checks the read-only bit (and the executable
> bit for files).

Unusable "as a general test for access".  And what you say is not
what the doc claims.  Perhaps the doc should say what you just said,
and mention that that is all it tests, that it is not a general test
for accessibility.

Perhaps the doc should say that the predicate can give a false
positive (t, when the dir is inaccessible).  Can it also give a
false negative (nil when the dir is accessible)?

> > And perhaps we need another predicate that actually does what this
> > one says it does, by trying to access the directory?
> 
> It would be too expensive to be useful, I think.  Just trying to
> read it, like you did, is good enough.
> 
> > BTW, the doc string of `file-accessible-p' says nothing about what
> > it returns if file FILENAME does NOT name a directory you can
> > open.
> 
> Anything but t means it's not accessible.  I think this much is
> clear from the doc string.

That is clear to you, but it is not clear from the doc string, IMHO.
It would be clear if it said it. ;-)

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Eli Zaretskii]

> Date: Tue, 25 Aug 2015 13:55:13 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21346@debbugs.gnu.org
> 
> > That's not what I wanted to see there.  If you want to know who is
> > its owner, try this:
> >   (file-attributes "D:/$RECYCLE.BIN/S-1-5-21-..." 'string)
> 
> (t 1 "ME" "Domain Users"
>    (20885 54199 0 0)
>    (20885 54199 0 0)
>    (20885 54199 0 0)
>    0 "drwxrwxrwx" t 0 506428215)
> 
> Where ME is my user name.  ME is also the value of `user-real-login-name'.
> In what way does this return value indicate that I do not (user ME does
> not) have access to the file?

It doesn't.  Which is why I asked for the output of
file-extended-attributes.

> The doc of `file-attributes' does not hint (to me) anything about
> such a value indicating that the file is private.

If the mode bits were "drwx------", then it'd be private.  But this
cannot happen on Windows, due to the way Emacs emulates the Posix mode
bits.

> > The name of the owner will show up as the 3rd element of the list it
> > returns, and the name of the group (I'm guessing "None") as the 4th
> > element.  (Unless file-attributes also signals an error.)
> 
> So I am apparently the owner, and the group is Domain Users, to which
> I belong.

You are the owner, but you are not given access to that directory,
yes.

> Where do we see indicated the fact that I do not have real access to
> the file?

In the output of file-extended-attributes.  But that is hard to
interpret by mere mortals, so type this command from the Windows shell
prompt to see a human-readable description:

  icacls d:\$RECYCLE.BIN\S-1-5-21-3120201979-235963886-2582836866-1001

It's quite possible that the command will say "Access denied".  If it
does, you need to run that command from a cmd shell session started
"as administrator" (e.g., by right-clicking on "command prompt"
desktop shortcut or on its shortcut in the Start dialog, selecting
"Run as administrator", and clicking YES on the privilege elevation
dialog).

Btw, if you run Emacs from cmd that was started "as administrator",
the call to directory-files for that directory will most probably
succeed, and show you the files in that user's recycle bin.

> > > And given this "inconsistency", don't you think this gotcha should
> > > be mentioned in the doc - of `file-accessible-directory-p', for
> > > example?
> > 
> > Maybe.  I'll have to think of a useful way of describing this.
> 
> Perhaps what you said above, if nothing more informative can be found:
> 
>   On Windows, that test is not reliable enough.

Done.

> > > That predicate would seem to be unusable as a general test for
> > > access to a directory.  IOW, what it claims it does is hardly
> > > what it does, apparently.
> > 
> > It's not unusable.  It checks the read-only bit (and the executable
> > bit for files).
> 
> Unusable "as a general test for access".

I disagree.  It does test that the file exists and is a directory, so
it _will_ tell you when either of these 2 conditions is false.  It
just can yield a false positive regarding the success of opening and
reading the directory.  The alternative is to complicate the heck out
of the implementation, make it much slower, and get false negatives
instead.  I think the alternative is worse.

> Can it also give a false negative (nil when the dir is accessible)?

Not on Windows, AFAIK.

> > > BTW, the doc string of `file-accessible-p' says nothing about what
> > > it returns if file FILENAME does NOT name a directory you can
> > > open.
> > 
> > Anything but t means it's not accessible.  I think this much is
> > clear from the doc string.
> 
> That is clear to you, but it is not clear from the doc string, IMHO.

The doc string says "Return t if file FILENAME names a directory you
can open.".  Ergo, if it returns something else, FILENAME is either
NOT a directory or you can NOT open it.  I fail to see how this could
be unclear, it's elementary logic for predicates.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Eli Zaretskii]

> Date: Wed, 26 Aug 2015 18:46:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 21346@debbugs.gnu.org
> 
> > > > That predicate would seem to be unusable as a general test for
> > > > access to a directory.  IOW, what it claims it does is hardly
> > > > what it does, apparently.
> > > 
> > > It's not unusable.  It checks the read-only bit (and the executable
> > > bit for files).
> > 
> > Unusable "as a general test for access".
> 
> I disagree.  It does test that the file exists and is a directory, so
> it _will_ tell you when either of these 2 conditions is false.  It
> just can yield a false positive regarding the success of opening and
> reading the directory.  The alternative is to complicate the heck out
> of the implementation, make it much slower, and get false negatives
> instead.  I think the alternative is worse.

On further thought, I found a simple and surprisingly efficient method
of testing directory accessibility without any false positives, and
made the corresponding changes.

> > > > And given this "inconsistency", don't you think this gotcha should
> > > > be mentioned in the doc - of `file-accessible-directory-p', for
> > > > example?
> > > 
> > > Maybe.  I'll have to think of a useful way of describing this.
> > 
> > Perhaps what you said above, if nothing more informative can be found:
> > 
> >   On Windows, that test is not reliable enough.
> 
> Done.

I've now removed that caveat from the doc string.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Drew Adams]

> In the output of file-extended-attributes.  But that is hard to
> interpret by mere mortals, so type this command from the Windows
> shell prompt to see a human-readable description:
> 
>   icacls d:\$RECYCLE.BIN\S-1-5-21-3120201979-235963886-2582836866-
>   1001

Thanks for the info.

> > > > And given this "inconsistency", don't you think this gotcha
> > > > should be mentioned in the doc - of `file-accessible-directory-p',
> > > > for example?
> >
> > Perhaps what you said above, if nothing more informative can be
> > found: On Windows, that test is not reliable enough.
> 
> Done.

Thx.

bug#21346: 25.0.50; REGRESSION: `directory-files' raises error for DIR that is `file-accessible-directory-p'

[Drew Adams]

> On further thought, I found a simple and surprisingly efficient method
> of testing directory accessibility without any false positives, and
> made the corresponding changes.

Bravo!  Thx.