iCloud directory access issue on macOS Catalina

iCloud directory access issue on macOS Catalina

[Pankaj Jangid]

Hi,

macOS is increasingly making it difficult to access directories. Even
its own terminal required permission for CDing into icloud directories.

I am facing the same in dired. Just wanted to know if anyone has got this
fixed. I tried to gave "Full Disk Access" to Emacs but still getting
this error.

```
Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
  signal(error ("Listing directory failed but ‘access-file’ worked"))
  error("Listing directory failed but `access-file' worked")
  insert-directory("/Users/pankaj/Documents/" "-al" nil t)
  dired-insert-directory("/Users/pankaj/Documents/" "-al" nil nil t)
  dired-readin-insert()
  dired-readin()
  dired-internal-noselect("~/Documents/" nil)
  dired-noselect("/Users/pankaj/Documents/" nil)
  dired-other-window("/Users/pankaj/Documents/" nil)
  funcall-interactively(dired-other-window "/Users/pankaj/Documents/" nil)
  call-interactively(dired-other-window nil nil)
  command-execute(dired-other-window)
```

Regards,
-- 
Pankaj Jangid

Re: iCloud directory access issue on macOS Catalina

[Perry Smith]

> On Oct 10, 2019, at 2:06 AM, Pankaj Jangid <pankaj.jangid@gmail.com> wrote:
> 
> Hi,
> 
> macOS is increasingly making it difficult to access directories. Even
> its own terminal required permission for CDing into icloud directories.
> 
> I am facing the same in dired. Just wanted to know if anyone has got this
> fixed. I tried to gave "Full Disk Access" to Emacs but still getting
> this error.
> 
> ```
> Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
>  signal(error ("Listing directory failed but ‘access-file’ worked"))
>  error("Listing directory failed but `access-file' worked")
>  insert-directory("/Users/pankaj/Documents/" "-al" nil t)
>  dired-insert-directory("/Users/pankaj/Documents/" "-al" nil nil t)
>  dired-readin-insert()
>  dired-readin()
>  dired-internal-noselect("~/Documents/" nil)
>  dired-noselect("/Users/pankaj/Documents/" nil)
>  dired-other-window("/Users/pankaj/Documents/" nil)
>  funcall-interactively(dired-other-window "/Users/pankaj/Documents/" nil)
>  call-interactively(dired-other-window nil nil)
>  command-execute(dired-other-window)
> ```

I can confirm this.  I’m using EmacsForMacOS latest so emacs version reports:

GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1894.00 Version 10.15 (Build 19A583)) of 2019-10-10

I have my Documents and Desktop in iCloud.  I’ve never tried this before.  The issue seems specific to these two directories.

I added Emacs to the list of “Full Disk Access” which causes it to show up in “Files and Folders” and I also added it to “Developer Tools”.

The problem can be seen (I think) easier by starting M-x shell:

> pedz@MysticSlate / % cd
> cd
> pedz@MysticSlate ~ % ls
> ls
> Desktop		Downloads	Movies		Pictures	Source		d		word-search.rb
> Documents	Library		Music		Public		bin		new-gnupg
> pedz@MysticSlate ~ % ls ~/Documents
> ls ~/Documents
> ls: Documents: Operation not permitted
> pedz@MysticSlate ~ % cd Documents/
> cd Documents/
> pedz@MysticSlate Documents % ls
> ls
> ls: .: Operation not permitted
> pedz@MysticSlate Documents % pwd
> pwd
> /Users/pedz/Documents
> pedz@MysticSlate Documents % 

I piddled around a bit but never succeeded.  I discovered that all the files and directories under /Applications/Emacs.app have the com.apple.quarantine xattr so I removed that 

sudo xattr -dr com.apple.quarantine /Applications/Emacs.app  

… no effect

I thought perhaps it was a lineage type deal.  This version of emacs actually starts up a small ruby shell and then does an exec so the final executable running is not the original Emacs.app that the finder spawned.  You can see this below:

This is the process tree before any changes (this is done at the shell prompt inside a “*shell*” buffer inside Emacs)

> pstree -p $$
> -+= 00001 root /sbin/launchd
>  \-+= 07613 pedz /Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14
>    \-+= 07620 pedz /bin/zsh -i
>      \-+= 07624 pedz pstree -p 7620
>        \--- 07625 root ps -axwwo user,pid,ppid,pgid,command

I modified the script to not do an exec but just do a “system” and so now the process stack looks like this:

> pstree -p $$
> -+= 00001 root /sbin/launchd
>  \-+= 07655 pedz /usr/bin/ruby /Applications/Emacs.app/Contents/MacOS/Emacs <<<<<<<
>    \-+- 07661 pedz /Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14
>      \-+= 07663 pedz /bin/zsh -i
>        \-+= 07665 pedz pstree -p 7663
>          \--- 07666 root ps -axwwo user,pid,ppid,pgid,command

but it still does not work.

This clearly isn’t an emacs problem.  I’m going to go beat the bushes in a few other places and see if I can find a solution.

I also opened an issue on the EmacsForMacOS GitHub repository <https://github.com/caldwell/build-emacs/issues/84>

pedz

Re: iCloud directory access issue on macOS Catalina

[Perry Smith]

Someone at the EmacsForMacOS GitHub discovered a few things about
this.  (Not exactly solved but a relatively simple work around)

Add Emacs.app to "Full Disk Access", then inside Emacs, use
ns-open-file-using-panel (bound to Cmd-o by default) to open an
arbitrary **file** directly inside ~/Documents.  The GUI won't let you
open a directory.  I bet that could be enhanced a little.  After that
emacs can dired in ~/Documents.

There seems to be three of these: ~/Documents, ~/Desktop, and
~/Downloads. I'm surprised ~/Downloads is in there because its not
copied to iCloud.

The ns-open-file-using-panel must be done once for each of these
three. After that, it appears to work forever and ever. I even
rebooted and it still worked.


> On Oct 10, 2019, at 11:23 AM, Perry Smith <pedz@easesoftware.com> wrote:
> 
> 
> 
>> On Oct 10, 2019, at 2:06 AM, Pankaj Jangid <pankaj.jangid@gmail.com> wrote:
>> 
>> Hi,
>> 
>> macOS is increasingly making it difficult to access directories. Even
>> its own terminal required permission for CDing into icloud directories.
>> 
>> I am facing the same in dired. Just wanted to know if anyone has got this
>> fixed. I tried to gave "Full Disk Access" to Emacs but still getting
>> this error.
>> 
>> ```
>> Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
>> signal(error ("Listing directory failed but ‘access-file’ worked"))
>> error("Listing directory failed but `access-file' worked")
>> insert-directory("/Users/pankaj/Documents/" "-al" nil t)
>> dired-insert-directory("/Users/pankaj/Documents/" "-al" nil nil t)
>> dired-readin-insert()
>> dired-readin()
>> dired-internal-noselect("~/Documents/" nil)
>> dired-noselect("/Users/pankaj/Documents/" nil)
>> dired-other-window("/Users/pankaj/Documents/" nil)
>> funcall-interactively(dired-other-window "/Users/pankaj/Documents/" nil)
>> call-interactively(dired-other-window nil nil)
>> command-execute(dired-other-window)
>> ```
> 
> I can confirm this.  I’m using EmacsForMacOS latest so emacs version reports:
> 
> GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1894.00 Version 10.15 (Build 19A583)) of 2019-10-10
> 
> I have my Documents and Desktop in iCloud.  I’ve never tried this before.  The issue seems specific to these two directories.
> 
> I added Emacs to the list of “Full Disk Access” which causes it to show up in “Files and Folders” and I also added it to “Developer Tools”.
> 
> The problem can be seen (I think) easier by starting M-x shell:
> 
>> pedz@MysticSlate / % cd
>> cd
>> pedz@MysticSlate ~ % ls
>> ls
>> Desktop		Downloads	Movies		Pictures	Source		d		word-search.rb
>> Documents	Library		Music		Public		bin		new-gnupg
>> pedz@MysticSlate ~ % ls ~/Documents
>> ls ~/Documents
>> ls: Documents: Operation not permitted
>> pedz@MysticSlate ~ % cd Documents/
>> cd Documents/
>> pedz@MysticSlate Documents % ls
>> ls
>> ls: .: Operation not permitted
>> pedz@MysticSlate Documents % pwd
>> pwd
>> /Users/pedz/Documents
>> pedz@MysticSlate Documents % 
> 
> I piddled around a bit but never succeeded.  I discovered that all the files and directories under /Applications/Emacs.app have the com.apple.quarantine xattr so I removed that 
> 
> sudo xattr -dr com.apple.quarantine /Applications/Emacs.app  
> 
> … no effect
> 
> I thought perhaps it was a lineage type deal.  This version of emacs actually starts up a small ruby shell and then does an exec so the final executable running is not the original Emacs.app that the finder spawned.  You can see this below:
> 
> This is the process tree before any changes (this is done at the shell prompt inside a “*shell*” buffer inside Emacs)
> 
>> pstree -p $$
>> -+= 00001 root /sbin/launchd
>> \-+= 07613 pedz /Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14
>>   \-+= 07620 pedz /bin/zsh -i
>>     \-+= 07624 pedz pstree -p 7620
>>       \--- 07625 root ps -axwwo user,pid,ppid,pgid,command
> 
> I modified the script to not do an exec but just do a “system” and so now the process stack looks like this:
> 
>> pstree -p $$
>> -+= 00001 root /sbin/launchd
>> \-+= 07655 pedz /usr/bin/ruby /Applications/Emacs.app/Contents/MacOS/Emacs <<<<<<<
>>   \-+- 07661 pedz /Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14
>>     \-+= 07663 pedz /bin/zsh -i
>>       \-+= 07665 pedz pstree -p 7663
>>         \--- 07666 root ps -axwwo user,pid,ppid,pgid,command
> 
> but it still does not work.
> 
> This clearly isn’t an emacs problem.  I’m going to go beat the bushes in a few other places and see if I can find a solution.
> 
> I also opened an issue on the EmacsForMacOS GitHub repository <https://github.com/caldwell/build-emacs/issues/84>
> 
> pedz
> 

Re: iCloud directory access issue on macOS Catalina

[Pankaj Jangid]

Perry Smith <pedz@easesoftware.com> writes:

> Someone at the EmacsForMacOS GitHub discovered a few things about
> this.  (Not exactly solved but a relatively simple work around)
>
> Add Emacs.app to "Full Disk Access", then inside Emacs, use
> ns-open-file-using-panel (bound to Cmd-o by default) to open an
> arbitrary **file** directly inside ~/Documents.  The GUI won't let you
> open a directory.  I bet that could be enhanced a little.  After that
> emacs can dired in ~/Documents.
>

I confirm that this approach worked on my Mac.

> There seems to be three of these: ~/Documents, ~/Desktop, and
> ~/Downloads. I'm surprised ~/Downloads is in there because its not
> copied to iCloud.
>

Yes. ~/Downloads is surprising. This implies that the problem isn't
related to iCloud but it is some other permissions stuff, which is
specific to macOS 10.15 (Catalina).

> The ns-open-file-using-panel must be done once for each of these
> three. After that, it appears to work forever and ever. I even
> rebooted and it still worked.
>
>

--
Regards,
Pankaj Jangid

Re: iCloud directory access issue on macOS Catalina

[Pankaj Jangid]

Pankaj Jangid <pankaj.jangid@gmail.com> writes:

> macOS is increasingly making it difficult to access directories. Even
> its own terminal required permission for CDing into icloud directories.
>
> I am facing the same in dired. Just wanted to know if anyone has got this
> fixed. I tried to gave "Full Disk Access" to Emacs but still getting
> this error.
>
> ```
> Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
>   signal(error ("Listing directory failed but ‘access-file’ worked"))
>   error("Listing directory failed but `access-file' worked")
>   insert-directory("/Users/pankaj/Documents/" "-al" nil t)
>   dired-insert-directory("/Users/pankaj/Documents/" "-al" nil nil t)
>   dired-readin-insert()
>   dired-readin()
>   dired-internal-noselect("~/Documents/" nil)
>   dired-noselect("/Users/pankaj/Documents/" nil)
>   dired-other-window("/Users/pankaj/Documents/" nil)
>   funcall-interactively(dired-other-window "/Users/pankaj/Documents/" nil)
>   call-interactively(dired-other-window nil nil)
>   command-execute(dired-other-window)
> ```

This issue has re-appeared in macOS 10.15.1 update. And the workaround
using ns-open-file-using-panel is also not working this time.

Regards,
-- 
Pankaj Jangid

Re: iCloud directory access issue on macOS Catalina

[Perry Smith]

There is a useful discussion on GitHub:
https://github.com/caldwell/build-emacs/issues/84 <https://github.com/caldwell/build-emacs/issues/84>

Parts of the discussion are specific to the “EmacsOnMacOS” method of
starting emacs but other bits are more general.

Once I’ve used the ns-open-file-using-panel method, I’ve not had
any other issues but I have not moved up to 10.15.1 yet.  I’m still
on 10.15.  I’m currently fighting with other updates and I have
a tiny internet down link but I will try and update to 10.15.1 later
today and report back.


> On Oct 31, 2019, at 8:39 AM, Pankaj Jangid <pankaj.jangid@gmail.com> wrote:
> 
> Pankaj Jangid <pankaj.jangid@gmail.com> writes:
> 
>> macOS is increasingly making it difficult to access directories. Even
>> its own terminal required permission for CDing into icloud directories.
>> 
>> I am facing the same in dired. Just wanted to know if anyone has got this
>> fixed. I tried to gave "Full Disk Access" to Emacs but still getting
>> this error.
>> 
>> ```
>> Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
>>  signal(error ("Listing directory failed but ‘access-file’ worked"))
>>  error("Listing directory failed but `access-file' worked")
>>  insert-directory("/Users/pankaj/Documents/" "-al" nil t)
>>  dired-insert-directory("/Users/pankaj/Documents/" "-al" nil nil t)
>>  dired-readin-insert()
>>  dired-readin()
>>  dired-internal-noselect("~/Documents/" nil)
>>  dired-noselect("/Users/pankaj/Documents/" nil)
>>  dired-other-window("/Users/pankaj/Documents/" nil)
>>  funcall-interactively(dired-other-window "/Users/pankaj/Documents/" nil)
>>  call-interactively(dired-other-window nil nil)
>>  command-execute(dired-other-window)
>> ```
> 
> This issue has re-appeared in macOS 10.15.1 update. And the workaround
> using ns-open-file-using-panel is also not working this time.
> 
> Regards,
> -- 
> Pankaj Jangid
> 

Re: iCloud directory access issue on macOS Catalina

[Pankaj Jangid]

Perry Smith <pedz@easesoftware.com> writes:

> There is a useful discussion on GitHub:
> https://github.com/caldwell/build-emacs/issues/84 <https://github.com/caldwell/build-emacs/issues/84>
>
> Parts of the discussion are specific to the “EmacsOnMacOS” method of
> starting emacs but other bits are more general.
>
> Once I’ve used the ns-open-file-using-panel method, I’ve not had
> any other issues but I have not moved up to 10.15.1 yet.  I’m still
> on 10.15.  I’m currently fighting with other updates and I have
> a tiny internet down link but I will try and update to 10.15.1 later
> today and report back.
>
Thanks for the link. I'll check that out. Till then, do try and verify
it on 10.15.1.

Regards,
-- 
Pankaj Jangid

Re: iCloud directory access issue on macOS Catalina

[Pankaj Jangid]

>> There is a useful discussion on GitHub:
>> https://github.com/caldwell/build-emacs/issues/84 <https://github.com/caldwell/build-emacs/issues/84>
>>
>> Parts of the discussion are specific to the “EmacsOnMacOS” method of
>> starting emacs but other bits are more general.

I tried to build a fresh. Looks like that the build process is now
changed in the above repository. New output generated doesn't have the
ruby script launcher. This new build is working fine on 10.15.1
(Catalina). It asks for permission and there after it works fine.

Regards,
-- 
Pankaj Jangid

Re: iCloud directory access issue on macOS Catalina

[Perry Smith]

> On Oct 31, 2019, at 11:53 AM, Pankaj Jangid <pankaj.jangid@gmail.com> wrote:
> 
> 
>>> There is a useful discussion on GitHub:
>>> https://github.com/caldwell/build-emacs/issues/84 <https://github.com/caldwell/build-emacs/issues/84>
>>> 
>>> Parts of the discussion are specific to the “EmacsOnMacOS” method of
>>> starting emacs but other bits are more general.
> 
> I tried to build a fresh. Looks like that the build process is now
> changed in the above repository. New output generated doesn't have the
> ruby script launcher. This new build is working fine on 10.15.1
> (Catalina). It asks for permission and there after it works fine.

I updated to 10.15.1 and I’m not seeing any issues coming back.

When this was first discovered, I poked and playsd with various things
to no success quite a bit and perhaps I’ve got something in a different
state.

But eventually I took the advice of using ns-open-file-using-panel to do
directories on ~/Desktop, Downloads, and Documents (once each) and
I’ve not had any issues since.

I’m using one of the nightly builds of EmacsForMacOS:

GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1894.10 Version 10.15.1 (Build 19B88)) of 2019-10-31

The timestamps indicate Oct. 9th.

pedz