bug#55743: 28.1.50; No directory error in comp-run-async-workers

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

0. emacs -Q
1. C-x C-f emacs/lisp/s TAB M-v

Then after switching to the Completions window
after a short delay this error is raised:

Debugger entered--Lisp error: (file-missing "Setting current directory" "No such file or directory" "emacs/lisp/s/")
  make-process(:name "Compiling: emacs/lisp/net/goto-addr.el" :buffer #<buffer *Async-native-compile-log*> :command ("emacs/src/emacs" "--batch" "-l" "/tmp/emacs-async-comp-goto-addr-cVqUYL.el") :sentinel #f(compiled-function (process event) #<bytecode -0x812502d31259a48>) :noquery t)
  comp-run-async-workers()
  #f(compiled-function (process event) #<bytecode -0xb3837fe88441506>)(#<process Compiling: emacs/lisp/emacs-lisp/bytecomp.el> "finished\n")
  completing-read-default("Find file: " read-file-name-internal file-exists-p confirm-after-completion "emacs/lisp/" file-name-history "emacs/lisp/" nil)
  read-file-name-default("Find file: " nil "emacs/lisp/" confirm-after-completion nil nil)
  read-file-name("Find file: " nil "emacs/lisp/" confirm-after-completion)
  find-file-read-args("Find file: " confirm-after-completion)
  command-execute(find-file)

This is because default-directory is set in the Completions buffer
to a non-existent "emacs/lisp/s/" from incomplete minibuffer input.
And comp-run-async-workers fails to make a process in that non-existent dir.

So here are two problems: why completion sets default-directory to non-existent dir,
and why comp-run-async-workers tries to run a process in such arbitrary invalid dirs.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Tue, 31 May 2022 20:31:36 +0300
> 
> So here are two problems: why completion sets default-directory to non-existent dir,

That's the real problem.

> and why comp-run-async-workers tries to run a process in such arbitrary invalid dirs.

It is not comp-run-async-workers that does it, it's the low-level
infrastructure in subroutines of make-process: it makes sure the
directory in which the process will run is valid, and if it isn't
signals an error.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>> So here are two problems: why completion sets default-directory to non-existent dir,
>
> That's the real problem.

Oh, I see, it's a known problem.  In completion-setup-function:

         (base-dir
          ;; FIXME: This is a bad hack.  We try to set the default-directory
          ;; in the *Completions* buffer so that the relative file names
          ;; displayed there can be treated as valid file names, independently
          ;; from the completion context.  But this suffers from many problems:
          ;; - It's not clear when the completions are file names.  With some
          ;;   completion tables (e.g. bzr revision specs), the listed
          ;;   completions can mix file names and other things.
          ;; - It doesn't pay attention to possible quoting.
          ;; - With fancy completion styles, the code below will not always
          ;;   find the right base directory.
          (if minibuffer-completing-file-name
              (file-name-as-directory
               (expand-file-name
                (buffer-substring (minibuffer-prompt-end) (point)))))))
      ...
      (if base-dir (setq default-directory base-dir))

>> and why comp-run-async-workers tries to run a process in such arbitrary invalid dirs.
>
> It is not comp-run-async-workers that does it, it's the low-level
> infrastructure in subroutines of make-process: it makes sure the
> directory in which the process will run is valid, and if it isn't
> signals an error.

Shouldn't native compilation run in the directory where the emacs
source files are located instead of running in the default directory
of an arbitrary buffer that might be invalid?

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 55743@debbugs.gnu.org
> Date: Tue, 31 May 2022 22:49:37 +0300
> 
> >> and why comp-run-async-workers tries to run a process in such arbitrary invalid dirs.
> >
> > It is not comp-run-async-workers that does it, it's the low-level
> > infrastructure in subroutines of make-process: it makes sure the
> > directory in which the process will run is valid, and if it isn't
> > signals an error.
> 
> Shouldn't native compilation run in the directory where the emacs
> source files are located instead of running in the default directory
> of an arbitrary buffer that might be invalid?

No, because native compilation could be invoked to compile a file that
has nothing to do with the Emacs source tree, e.g., some file of the
user or some 3rd-party package installed in some arbitrary place.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>> >> and why comp-run-async-workers tries to run a process in such arbitrary invalid dirs.
>> >
>> > It is not comp-run-async-workers that does it, it's the low-level
>> > infrastructure in subroutines of make-process: it makes sure the
>> > directory in which the process will run is valid, and if it isn't
>> > signals an error.
>>
>> Shouldn't native compilation run in the directory where the emacs
>> source files are located instead of running in the default directory
>> of an arbitrary buffer that might be invalid?
>
> No, because native compilation could be invoked to compile a file that
> has nothing to do with the Emacs source tree, e.g., some file of the
> user or some 3rd-party package installed in some arbitrary place.

Still, shouldn't it run compilation in the same directory
where that file is located?  Isn't this better than running
compilation in a random dir from a random buffer?

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 55743@debbugs.gnu.org
> Date: Wed, 01 Jun 2022 09:13:14 +0300
> 
> >> Shouldn't native compilation run in the directory where the emacs
> >> source files are located instead of running in the default directory
> >> of an arbitrary buffer that might be invalid?
> >
> > No, because native compilation could be invoked to compile a file that
> > has nothing to do with the Emacs source tree, e.g., some file of the
> > user or some 3rd-party package installed in some arbitrary place.
> 
> Still, shouldn't it run compilation in the same directory
> where that file is located?  Isn't this better than running
> compilation in a random dir from a random buffer?

No, it isn't better, because the directory should be set by the
caller.  The compilation primitive cannot second-guess what the caller
meant.  For example, the file to be compiled could load other files,
and load-path could mention the current directory in some relative
form; or the directory of the file could not be chdir'ed into, but
files in it could be accessed; or some VCS could be involved (so you
need to be in the VCS repository), or something else.  make-process
uses the default-directory of the current buffer because making sure
that directory is the correct one is left to the caller, and it makes
it easy for the caller to ensure the compilation runs in the correct
directory.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>> >> Shouldn't native compilation run in the directory where the emacs
>> >> source files are located instead of running in the default directory
>> >> of an arbitrary buffer that might be invalid?
>> >
>> > No, because native compilation could be invoked to compile a file that
>> > has nothing to do with the Emacs source tree, e.g., some file of the
>> > user or some 3rd-party package installed in some arbitrary place.
>> 
>> Still, shouldn't it run compilation in the same directory
>> where that file is located?  Isn't this better than running
>> compilation in a random dir from a random buffer?
>
> No, it isn't better, because the directory should be set by the
> caller.  The compilation primitive cannot second-guess what the caller
> meant.  For example, the file to be compiled could load other files,
> and load-path could mention the current directory in some relative
> form; or the directory of the file could not be chdir'ed into, but
> files in it could be accessed; or some VCS could be involved (so you
> need to be in the VCS repository), or something else.  make-process
> uses the default-directory of the current buffer because making sure
> that directory is the correct one is left to the caller, and it makes
> it easy for the caller to ensure the compilation runs in the correct
> directory.

I don't understand who is the caller?  The user?  The user visits
an non-existent directory, by e.g. `C-x C-f /bla/bla/bla', then
emacs compilation kicks in using that directory, and signals the error.
If this is the intended behavior, then I guess this bug report can be closed,
since I fixed the non-existent directory case in *Completions*.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 55743@debbugs.gnu.org
> Date: Wed, 01 Jun 2022 22:13:37 +0300
> 
> >> Still, shouldn't it run compilation in the same directory
> >> where that file is located?  Isn't this better than running
> >> compilation in a random dir from a random buffer?
> >
> > No, it isn't better, because the directory should be set by the
> > caller.  The compilation primitive cannot second-guess what the caller
> > meant.  For example, the file to be compiled could load other files,
> > and load-path could mention the current directory in some relative
> > form; or the directory of the file could not be chdir'ed into, but
> > files in it could be accessed; or some VCS could be involved (so you
> > need to be in the VCS repository), or something else.  make-process
> > uses the default-directory of the current buffer because making sure
> > that directory is the correct one is left to the caller, and it makes
> > it easy for the caller to ensure the compilation runs in the correct
> > directory.
> 
> I don't understand who is the caller?  The user?  The user visits
> an non-existent directory, by e.g. `C-x C-f /bla/bla/bla', then
> emacs compilation kicks in using that directory, and signals the error.

Please tell how this case is different from the one below:

  emacs -Q
  M-: (setq default-directory "/non-existent/foo/bar") RET
  M-! ls RET

You get the same error about "setting current directory".  Do yopu
think it's a bug in M-! ?

> If this is the intended behavior, then I guess this bug report can be closed,
> since I fixed the non-existent directory case in *Completions*.

Thanks, I think this can indeed be closed.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

> Please tell how this case is different from the one below:
>
>   emacs -Q
>   M-: (setq default-directory "/non-existent/foo/bar") RET
>   M-! ls RET
>
> You get the same error about "setting current directory".  Do yopu
> think it's a bug in M-! ?

M-! is a command explicitly initiated by the user in the current buffer.
OTOH, when native compilation signals an error, this is totally unexpected.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 55743@debbugs.gnu.org
> Date: Thu, 02 Jun 2022 10:40:12 +0300
> 
> > Please tell how this case is different from the one below:
> >
> >   emacs -Q
> >   M-: (setq default-directory "/non-existent/foo/bar") RET
> >   M-! ls RET
> >
> > You get the same error about "setting current directory".  Do yopu
> > think it's a bug in M-! ?
> 
> M-! is a command explicitly initiated by the user in the current buffer.

Is it really relevant? the same will happen if shell-command is called
non-interactively by some Lisp program.

We require that default-directory be valid whenever Emacs needs to run
a subprocess.  This is a basic requirement for all Lisp programs using
this facility.  Native compilation is not different from any other
Lisp program that invokes a subprocess.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We require that default-directory be valid whenever Emacs needs to run
> a subprocess.  This is a basic requirement for all Lisp programs using
> this facility.  Native compilation is not different from any other
> Lisp program that invokes a subprocess.

Any asynchronous background process has to ensure that it starts from a
sensible directory -- not whatever default-directory that happens to be
in place in the current buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Juri Linkov <juri@linkov.net>,  55743@debbugs.gnu.org
> Date: Thu, 02 Jun 2022 10:17:10 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We require that default-directory be valid whenever Emacs needs to run
> > a subprocess.  This is a basic requirement for all Lisp programs using
> > this facility.  Native compilation is not different from any other
> > Lisp program that invokes a subprocess.
> 
> Any asynchronous background process has to ensure that it starts from a
> sensible directory -- not whatever default-directory that happens to be
> in place in the current buffer.

But what is the "sensible" directory in this case?  I don't think we
can know it, because what it is depends on many factors we cannot
possibly take into consideration.

IMO, it's the other way around: any Lisp program that could
potentially trigger a subprocess, including compilation, should make
sure the default-directory is always valid.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> IMO, it's the other way around: any Lisp program that could
> potentially trigger a subprocess, including compilation, should make
> sure the default-directory is always valid.

No, the thing that starts the external process should ensure that it can
do so.  See `with-existing-directory'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 55743@debbugs.gnu.org,  juri@linkov.net
> Date: Thu, 02 Jun 2022 10:51:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > IMO, it's the other way around: any Lisp program that could
> > potentially trigger a subprocess, including compilation, should make
> > sure the default-directory is always valid.
> 
> No, the thing that starts the external process should ensure that it can
> do so.  See `with-existing-directory'.

Once again, how would you know what is the correct directory that will
allow the compilation (in this case) or any external process (in other
cases) DTRT?  The place where we invoke comp-run-async-workers cannot
possibly know enough about the compilation to make such decisions.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Once again, how would you know what is the correct directory that will
> allow the compilation (in this case) or any external process (in other
> cases) DTRT?  The place where we invoke comp-run-async-workers cannot
> possibly know enough about the compilation to make such decisions.

comp-run-async-workers knows that it needs an existing directory to
work, so it should arrange to have an existing directory be the current
one.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 55743@debbugs.gnu.org,  juri@linkov.net
> Date: Fri, 03 Jun 2022 05:10:36 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Once again, how would you know what is the correct directory that will
> > allow the compilation (in this case) or any external process (in other
> > cases) DTRT?  The place where we invoke comp-run-async-workers cannot
> > possibly know enough about the compilation to make such decisions.
> 
> comp-run-async-workers knows that it needs an existing directory to
> work, so it should arrange to have an existing directory be the current
> one.

Yes, we can arrange for an existing directory, but:

  . it could be wrong, i.e. it could yield wrong compilation results
  . it will silently sweep under the carpet cases like this one, where
    a Lisp program makes default-directory invalid -- don't we want to
    uncover such cases?

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Andreas Schwab]

On Jun 03 2022, Eli Zaretskii wrote:

> Yes, we can arrange for an existing directory, but:
>
>   . it could be wrong, i.e. it could yield wrong compilation results

A remote directory would also be wrong.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  55743@debbugs.gnu.org,
>   juri@linkov.net
> Date: Fri, 03 Jun 2022 10:22:03 +0200
> 
> On Jun 03 2022, Eli Zaretskii wrote:
> 
> > Yes, we can arrange for an existing directory, but:
> >
> >   . it could be wrong, i.e. it could yield wrong compilation results
> 
> A remote directory would also be wrong.

Indeed, thanks.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Andreas Schwab]

On Jun 02 2022, Eli Zaretskii wrote:

> Native compilation is not different from any other Lisp program that
> invokes a subprocess.

But native compilation is more like infrastructure, and not a Lisp
program.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Juri Linkov <juri@linkov.net>,  55743@debbugs.gnu.org
> Date: Thu, 02 Jun 2022 10:17:42 +0200
> 
> On Jun 02 2022, Eli Zaretskii wrote:
> 
> > Native compilation is not different from any other Lisp program that
> > invokes a subprocess.
> 
> But native compilation is more like infrastructure, and not a Lisp
> program.

Yes.  But that is not relevant for the issue at hand.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>> But native compilation is more like infrastructure, and not a Lisp
>> program.
>
> Yes.  But that is not relevant for the issue at hand.

So you think that native compilation failure is a user error?

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Andreas Schwab <schwab@suse.de>,  55743@debbugs.gnu.org
> Date: Thu, 02 Jun 2022 20:30:29 +0300
> 
> >> But native compilation is more like infrastructure, and not a Lisp
> >> program.
> >
> > Yes.  But that is not relevant for the issue at hand.
> 
> So you think that native compilation failure is a user error?

No, I think it's an error in the command that caused default-directory
be set to a non-existent directory -- the error which you already
fixed.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>> >> But native compilation is more like infrastructure, and not a Lisp
>> >> program.
>> >
>> > Yes.  But that is not relevant for the issue at hand.
>>
>> So you think that native compilation failure is a user error?
>
> No, I think it's an error in the command that caused default-directory
> be set to a non-existent directory -- the error which you already
> fixed.

I meant other cases such as M-: (setq default-directory "/non-existent/foo/bar") RET

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: schwab@suse.de,  55743@debbugs.gnu.org
> Date: Fri, 03 Jun 2022 10:49:13 +0300
> 
> >> >> But native compilation is more like infrastructure, and not a Lisp
> >> >> program.
> >> >
> >> > Yes.  But that is not relevant for the issue at hand.
> >>
> >> So you think that native compilation failure is a user error?
> >
> > No, I think it's an error in the command that caused default-directory
> > be set to a non-existent directory -- the error which you already
> > fixed.
> 
> I meant other cases such as M-: (setq default-directory "/non-existent/foo/bar") RET

Yes, I think it's a user error, similar to deleting a directory that
is the CWD of some shell.

bug#55743: 28.1.50; No directory error in comp-run-async-workers

[Juri Linkov]

>>> So here are two problems: why completion sets default-directory to non-existent dir,
>>
>> That's the real problem.
>
> Oh, I see, it's a known problem.  In completion-setup-function:
>
>          (base-dir
>           ;; FIXME: This is a bad hack.  We try to set the default-directory
>           ;; in the *Completions* buffer so that the relative file names
>           ;; displayed there can be treated as valid file names, independently
>           ;; from the completion context.  But this suffers from many problems:
>           ;; - It's not clear when the completions are file names.  With some
>           ;;   completion tables (e.g. bzr revision specs), the listed
>           ;;   completions can mix file names and other things.
>           ;; - It doesn't pay attention to possible quoting.
>           ;; - With fancy completion styles, the code below will not always
>           ;;   find the right base directory.
>           (if minibuffer-completing-file-name
>               (file-name-as-directory
>                (expand-file-name
>                 (buffer-substring (minibuffer-prompt-end) (point)))))))
>       ...
>       (if base-dir (setq default-directory base-dir))

The problem can be solved by this patch that handles both the most
frequent cases: when the completion string is the default directory,
and when a file prefix is added to it.

diff --git a/lisp/simple.el b/lisp/simple.el
index 103e7f33dd..81e04f28e9 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -9803,7 +9802,7 @@ completion-setup-function
           ;; - With fancy completion styles, the code below will not always
           ;;   find the right base directory.
           (if minibuffer-completing-file-name
-              (file-name-as-directory
+              (file-name-directory
                (expand-file-name
                 (buffer-substring (minibuffer-prompt-end) (point)))))))
     (with-current-buffer standard-output