bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

Hello maintainers,

I've asked this in Emacs devel already:
https://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00037.html. Someone
suggested I report it as a bug.  It happens Windows (machine of bug
report), but also on my Ubuntu virtual server.

I would expect while-no-input to break out of accept-process-output very
quickly after user keyboard input.  These expectations are met except
for some situations.  Here's some test code:

   (defmacro joaot/time (&rest body)
     `(let ((start (current-time)))
        (prog1
            (progn ,@body)
          (let ((msg (format "Took %s seconds and returned "
                             (format-time-string
                              "%S.%3N"
                              (time-subtract (current-time) start)))))
            (if current-prefix-arg
                (insert "; " msg)
              (message msg))))))

Now, after each of these, I'm pressing C-u C-x C-e SPC as fast as I can.
The second result is unexpected, all the others are fine. Moreover, it
varies from 0.7s to as much as 5s.  

    (joaot/time
     (while-no-input
       (while t (accept-process-output nil 0.05)))); Expected, took 00.201 seconds and returned t 
     
    (joaot/time
     (while-no-input
       (while t (accept-process-output nil 30)))); Took 02.694 seconds and returned t 
     
    (joaot/time
     (while (sit-for 30))); Took 00.126 seconds and returned nil 
     
    (joaot/time
     (while (sit-for 0.1))) ; Took 00.093 seconds and returned nil

I tried quickly pluggin GDB in at the right time, but I don't know if
I'm using the right GDB (using msys's which is pretty slow) and I
probably should be using an unoptimized Emacs.  Anyway, as a start this
is what "bt full" look like:

   (gdb) bt full
   #0  0x00007ffc97b0d8c1 in ntdll!DbgBreakPoint () from /c/WINDOWS/SYSTEM32/ntdll.dll
   No symbol table info available.
   #1  0x00007ffc97b39a0b in ntdll!DbgUiRemoteBreakin () from /c/WINDOWS/SYSTEM32/ntdll.dll
   No symbol table info available.
   #2  0x00007ffc952b3034 in KERNEL32!BaseThreadInitThunk ()
      from /c/WINDOWS/System32/KERNEL32.DLL
   No symbol table info available.
   #3  0x00007ffc97ae1461 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
   No symbol table info available.
   #4  0x0000000000000000 in ?? ()
   No symbol table info available.
   Backtrace stopped: previous frame inner to this frame (corrupt stack?)
   (gdb) xbacktrace
   Undefined command: "xbacktrace".  Try "help".

xbacktrace probably failed because of some Python error loading .gdbinit

In GNU Emacs 27.0.50 (build 1, x86_64-w64-mingw32)
 of 2018-10-02 built on GONDOMAR
Repository revision: dfbb207ff946792efebb31c0c59b8245c304544a
Windowing system distributor 'Microsoft Corp.', version 10.0.17134
System Description: Microsoft Windows 10 Pro (v10.0.1803.17134.286)

Recent messages:
joaot/time
Quit
Mark set
Quit
Mark set
Is this a siscog mail? (y or n) n
Quit [4 times]
Mark set [2 times]
Auto-saving...done
Type C-x 1 to delete the help window.
Quit [3 times]
Configured using:
 'configure --prefix=/c/emacs/emacs-26 --without-imagemagick
 --without-dbus'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS THREADS JSON LCMS2 GMP

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: PTG
  locale-coding-system: cp1252

Major mode: Message

Minor modes in effect:
  gnus-message-citation-mode: t
  diff-auto-refine-mode: t
  savehist-mode: t
  winner-mode: t
  ido-everywhere: t
  electric-pair-mode: t
  delete-selection-mode: t
  global-auto-revert-mode: t
  show-paren-mode: t
  mml-mode: t
  company-quickhelp-mode: t
  company-quickhelp-local-mode: t
  global-aggressive-indent-mode: t
  shell-dirtrack-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: message-do-auto-fill
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
Error during checking
Features:
(shadow emacsbug darkroom face-remap gnus-dup flymake-cc
display-line-numbers autoload trace imenu ...)

Memory information:
((conses 16 944904 133288)
 (symbols 56 54804 49)
 (strings 32 196879 15626)
 (string-bytes 1 4881114)
 (vectors 16 98591)
 (vector-slots 8 2249333 86252)
 (floats 8 863 1865)
 (intervals 56 47943 2025)
 (buffers 992 132))

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 08 Oct 2018 11:48:01 +0100
> 
> I would expect while-no-input to break out of accept-process-output very
> quickly after user keyboard input.  These expectations are met except
> for some situations.

I think your expectations are incorrect.  My expectations are that if
you call accept-process-output with a timeout of 30 sec, then
while-no-input will return after 30 sec plus a small delay.  It's just
that in order to see this in action, your experiment must be done in a
"clean room", and that is not easy.

First, one basic fact: accept-process-output calls
wait_reading_process_output in a way that instructs it not to wait for
keyboard input, you can see that clearly in the code (read_kbd is
passed as zero).  This means that wait_reading_process_output will
call pselect with the timeout of 30 sec (in your case), and will wait
that long before it returns (unless there's a subprocess that gives us
some stuff).  So why would you expect while-no-input that calls
accept-process-output with that timeout to return earlier?

Maybe you expect while-no-input to interrupt the pselect call when you
press SPC?  But that's not how keyboard input works in Emacs.  In some
configurations (e.g., GUI frame on X), keyboard input indeed delivers
a signal to Emacs, but the signal handler just sets a flag and
returns, it doesn't jump out of the pselect call.  It is then the job
of the running Lisp code to check whether keyboard input is available,
and act accordingly: set the quit-flag, and then test that flag soon
enough to return from while-no-input.  But since the "running Lisp
code" is in this case stuck in the pselect call, it cannot do anything
before pselect returns, right?

Now to the "clean room" part: the reason why you don't always see this
30-sec delay is because there are several factors that "contaminate"
the picture:

  . timers -- these cause us to reduce the timeout until the
    expiration time of the next timer, so pselect returns earlier than
    after 30 sec
  . the first time wait_reading_process_output is called, it almost
    immediately checks for available keyboard input, before it calls
    pselect

Therefore, to see the 30-sec delay, you need:

  . stop all timers, which in "emacs -Q" means:
    . disable blink-cursor-mode
    . disable global-eldoc-mode
    . disable font-lock-mode
    . cancel the undo--auto-bindary-timer (I did that via list-timers)
  . type "C-u C-x C-e", wait for a few seconds, and only then type SPC

If I do all of the above, I see 30 sec plus a small delay every time I
run your 2nd test.

> I tried quickly pluggin GDB in at the right time, but I don't know if
> I'm using the right GDB (using msys's which is pretty slow) and I
> probably should be using an unoptimized Emacs.  Anyway, as a start this
> is what "bt full" look like:
> 
>    (gdb) bt full
>    #0  0x00007ffc97b0d8c1 in ntdll!DbgBreakPoint () from /c/WINDOWS/SYSTEM32/ntdll.dll
>    No symbol table info available.
>    #1  0x00007ffc97b39a0b in ntdll!DbgUiRemoteBreakin () from /c/WINDOWS/SYSTEM32/ntdll.dll
>    No symbol table info available.
>    #2  0x00007ffc952b3034 in KERNEL32!BaseThreadInitThunk ()
>       from /c/WINDOWS/System32/KERNEL32.DLL
>    No symbol table info available.
>    #3  0x00007ffc97ae1461 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
>    No symbol table info available.
>    #4  0x0000000000000000 in ?? ()

On Windows, when you attach a debugger to a program, the OS creates a
special thread in the debuggee, and that is the thread whose backtrace
you see here.  That thread is never an interesting one, so the first
thing you need to do after attaching is switch to the Lisp thread, by
typing "thread 1" at the GDB prompt.  Then the backtrace will make
much more sense.

>    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>    (gdb) xbacktrace
>    Undefined command: "xbacktrace".  Try "help".
> 
> xbacktrace probably failed because of some Python error loading .gdbinit

No, it says "Undefined command", which means it doesn't know what
xbacktrace is.  You probably didn't start GDB from the Emacs's src
directory, so it didn't read the .gdbinit file which defines that
command.  You can do that manually by typing the command

  (gdb) source /path/to/emacs/src/.gdbinit

(This last part is not specific to Widnows.)

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 5251 bytes --]

Hello Eli,

I will fully read and process your thorough reply later tonight or
tomorrow, but in the meantime let me just restate, or clarify in case it
wasn't clear, that I expect the 30s, 15s and 0.1s case to break equally as
quickly, namely as quick as I can type the first input.

And indeed that's what happens on Linux and Mac OSX, but not on Windows.
If your reply already addresses this apparent discrepancy, then I apologize
for my premature clarification in advance.

Thanks,
João



On Mon, Oct 8, 2018, 16:06 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Mon, 08 Oct 2018 11:48:01 +0100
> >
> > I would expect while-no-input to break out of accept-process-output very
> > quickly after user keyboard input.  These expectations are met except
> > for some situations.
>
> I think your expectations are incorrect.  My expectations are that if
> you call accept-process-output with a timeout of 30 sec, then
> while-no-input will return after 30 sec plus a small delay.  It's just
> that in order to see this in action, your experiment must be done in a
> "clean room", and that is not easy.
>
> First, one basic fact: accept-process-output calls
> wait_reading_process_output in a way that instructs it not to wait for
> keyboard input, you can see that clearly in the code (read_kbd is
> passed as zero).  This means that wait_reading_process_output will
> call pselect with the timeout of 30 sec (in your case), and will wait
> that long before it returns (unless there's a subprocess that gives us
> some stuff).  So why would you expect while-no-input that calls
> accept-process-output with that timeout to return earlier?
>
> Maybe you expect while-no-input to interrupt the pselect call when you
> press SPC?  But that's not how keyboard input works in Emacs.  In some
> configurations (e.g., GUI frame on X), keyboard input indeed delivers
> a signal to Emacs, but the signal handler just sets a flag and
> returns, it doesn't jump out of the pselect call.  It is then the job
> of the running Lisp code to check whether keyboard input is available,
> and act accordingly: set the quit-flag, and then test that flag soon
> enough to return from while-no-input.  But since the "running Lisp
> code" is in this case stuck in the pselect call, it cannot do anything
> before pselect returns, right?
>
> Now to the "clean room" part: the reason why you don't always see this
> 30-sec delay is because there are several factors that "contaminate"
> the picture:
>
>   . timers -- these cause us to reduce the timeout until the
>     expiration time of the next timer, so pselect returns earlier than
>     after 30 sec
>   . the first time wait_reading_process_output is called, it almost
>     immediately checks for available keyboard input, before it calls
>     pselect
>
> Therefore, to see the 30-sec delay, you need:
>
>   . stop all timers, which in "emacs -Q" means:
>     . disable blink-cursor-mode
>     . disable global-eldoc-mode
>     . disable font-lock-mode
>     . cancel the undo--auto-bindary-timer (I did that via list-timers)
>   . type "C-u C-x C-e", wait for a few seconds, and only then type SPC
>
> If I do all of the above, I see 30 sec plus a small delay every time I
> run your 2nd test.
>
> > I tried quickly pluggin GDB in at the right time, but I don't know if
> > I'm using the right GDB (using msys's which is pretty slow) and I
> > probably should be using an unoptimized Emacs.  Anyway, as a start this
> > is what "bt full" look like:
> >
> >    (gdb) bt full
> >    #0  0x00007ffc97b0d8c1 in ntdll!DbgBreakPoint () from
> /c/WINDOWS/SYSTEM32/ntdll.dll
> >    No symbol table info available.
> >    #1  0x00007ffc97b39a0b in ntdll!DbgUiRemoteBreakin () from
> /c/WINDOWS/SYSTEM32/ntdll.dll
> >    No symbol table info available.
> >    #2  0x00007ffc952b3034 in KERNEL32!BaseThreadInitThunk ()
> >       from /c/WINDOWS/System32/KERNEL32.DLL
> >    No symbol table info available.
> >    #3  0x00007ffc97ae1461 in ntdll!RtlUserThreadStart () from
> /c/WINDOWS/SYSTEM32/ntdll.dll
> >    No symbol table info available.
> >    #4  0x0000000000000000 in ?? ()
>
> On Windows, when you attach a debugger to a program, the OS creates a
> special thread in the debuggee, and that is the thread whose backtrace
> you see here.  That thread is never an interesting one, so the first
> thing you need to do after attaching is switch to the Lisp thread, by
> typing "thread 1" at the GDB prompt.  Then the backtrace will make
> much more sense.
>
> >    Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> >    (gdb) xbacktrace
> >    Undefined command: "xbacktrace".  Try "help".
> >
> > xbacktrace probably failed because of some Python error loading .gdbinit
>
> No, it says "Undefined command", which means it doesn't know what
> xbacktrace is.  You probably didn't start GDB from the Emacs's src
> directory, so it didn't read the .gdbinit file which defines that
> command.  You can do that manually by typing the command
>
>   (gdb) source /path/to/emacs/src/.gdbinit
>
> (This last part is not specific to Widnows.)
>

[-- Attachment #2: Type: text/html, Size: 6271 bytes --]

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 8 Oct 2018 21:06:12 +0100
> Cc: 32986@debbugs.gnu.org
> 
> I will fully read and process your thorough reply later tonight or tomorrow, but in the meantime let me just
> restate, or clarify in case it wasn't clear, that I expect the 30s, 15s and 0.1s case to break equally as quickly,
> namely as quick as I can type the first input.
> 
> And indeed that's what happens on Linux and Mac OSX, but not on Windows.  If your reply already addresses
> this apparent discrepancy, then I apologize for my premature clarification in advance.

I didn't investigate the difference in behavior between Windows and
GNU/Linux, because I see similar behavior on both systems if I
neutralize all the "contaminating" factors which I described.  It is
possible that it's easier to get the 30-sec delay on Windows because
keyboard input works without signals there, and uses messaging between
2 threads running within the Emacs process.

In any case, my point is that your expectation for immediate return is
incorrect, and I tried to explain why.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1743 bytes --]

Thank you for the clarification. I have now read the original explanation,
and it makes sense.  Ultimately, I think the sit-for is the right approach
for my wait-for-any-process-or-input problem.  Am I right to assume  it's
not affected by your explanation, and that I can expect immediate return
there?

If so, there's some unfortunate combination of factors that cause a
hard-to-reproduce hang in Emacs, at least in the packages where I use it.
But that's a matter for a future issue...

Thanks,
João

On Mon, Oct 8, 2018, 21:25 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Mon, 8 Oct 2018 21:06:12 +0100
> > Cc: 32986@debbugs.gnu.org
> >
> > I will fully read and process your thorough reply later tonight or
> tomorrow, but in the meantime let me just
> > restate, or clarify in case it wasn't clear, that I expect the 30s, 15s
> and 0.1s case to break equally as quickly,
> > namely as quick as I can type the first input.
> >
> > And indeed that's what happens on Linux and Mac OSX, but not on
> Windows.  If your reply already addresses
> > this apparent discrepancy, then I apologize for my premature
> clarification in advance.
>
> I didn't investigate the difference in behavior between Windows and
> GNU/Linux, because I see similar behavior on both systems if I
> neutralize all the "contaminating" factors which I described.  It is
> possible that it's easier to get the 30-sec delay on Windows because
> keyboard input works without signals there, and uses messaging between
> 2 threads running within the Emacs process.
>
> In any case, my point is that your expectation for immediate return is
> incorrect, and I tried to explain why.
>

[-- Attachment #2: Type: text/html, Size: 2337 bytes --]

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 8 Oct 2018 21:39:53 +0100
> Cc: 32986@debbugs.gnu.org
> 
> Thank you for the clarification. I have now read the original explanation, and it makes sense.  Ultimately, I think
> the sit-for is the right approach for my wait-for-any-process-or-input problem.  Am I right to assume  it's not
> affected by your explanation, and that I can expect immediate return there?

sit-for waits for keyboard input, so yes, it should return once the
user presses some key.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

On Tue, Oct 9, 2018, 03:32 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Mon, 8 Oct 2018 21:39:53 +0100
> > Cc: 32986@debbugs.gnu.org
> >
> > Thank you for the clarification. I have now read the original
> explanation, and it makes sense.  Ultimately, I think
> > the sit-for is the right approach for my wait-for-any-process-or-input
> problem.  Am I right to assume  it's not
> > affected by your explanation, and that I can expect immediate return
> there?
>
> sit-for waits for keyboard input, so yes, it should return once the
> user presses some key.
>

Right. A final question: process input is also considered during the
sit-for, meaning a filter can throw to an enclosing tag and end it
prematurely and immediately, right?

>

[-- Attachment #2: Type: text/html, Size: 1455 bytes --]

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 9 Oct 2018 09:47:58 +0100
> Cc: 32986@debbugs.gnu.org
> 
> A final question: process input is also considered during the sit-for, meaning a filter can throw to an
> enclosing tag and end it prematurely and immediately, right?

You mean, if the only input is from a subprocess?  No, I don't think
so.  Do you have evidence to the contrary?

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 9 Oct 2018 09:47:58 +0100
>> Cc: 32986@debbugs.gnu.org
>> 
>> A final question: process input is also considered during the
>> sit-for, meaning a filter can throw to an
>> enclosing tag and end it prematurely and immediately, right?
>
> You mean, if the only input is from a subprocess?  No, I don't think
> so.  Do you have evidence to the contrary?

Hi Eli,

Sorry for the delay in my reply.

I meant the following, which appears to do what I want in my testing:

   (joaot/time
    (catch 'done
      (progn (make-process :name "test"
                           :filter (lambda (proc string)
                                     (message "Hey %s just got %s" proc string)
                                     (throw 'done nil))
                           :command '("sh" "-c" "sleep 2 && echo bla"))
             (while (sit-for 30))))) ; Took 02.011 seconds and returned nil

The above result is with C-u C-x C-e and no further keyboard input. If
however I press C-u C-x C-e SPC as in my previous experiments, I get the
expected very small delay.

In other words, subprocess input is considered during a 'sit-for'. It
doesn't end it, but that's OK if catch/throw is used.

I'd just like to confirm that I'm not dealing with an "dirty room"
situation here, to use your metaphor.

João

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: 32986@debbugs.gnu.org
> Date: Sun, 14 Oct 2018 23:08:30 +0100
> 
> In other words, subprocess input is considered during a 'sit-for'. It
> doesn't end it, but that's OK if catch/throw is used.
> 
> I'd just like to confirm that I'm not dealing with an "dirty room"
> situation here, to use your metaphor.

I think you are right.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Yes, I agree (with myself :-).   I've just "cleaned the room" according to
your instructions and I still see the desired behavior.

So I guess this wraps up the issue. Thanks for all the help.

João

On Mon, Oct 15, 2018 at 4:03 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Cc: 32986@debbugs.gnu.org
> > Date: Sun, 14 Oct 2018 23:08:30 +0100
> >
> > In other words, subprocess input is considered during a 'sit-for'. It
> > doesn't end it, but that's OK if catch/throw is used.
> >
> > I'd just like to confirm that I'm not dealing with an "dirty room"
> > situation here, to use your metaphor.
>
> I think you are right.
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 1316 bytes --]

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Dmitry Gutov]

On 15.10.2018 1:08, João Távora wrote:

>     (joaot/time
>      (catch 'done
>        (progn (make-process :name "test"
>                             :filter (lambda (proc string)
>                                       (message "Hey %s just got %s" proc string)
>                                       (throw 'done nil))
>                             :command '("sh" "-c" "sleep 2 && echo bla"))
>               (while (sit-for 30))))) ; Took 02.011 seconds and returned nil

It's an interesting alternative to Company's async backend interface, 
where we have a similar piece of code waiting until either completions 
are returned or the user types something (in company--fetch-candidates).

Which of course makes sense, given that CAPF has no async calling 
convention.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

Indeed, I developed this for `company-capf`-based backends.

I still haven't ported it to jsonrpc.el tho (I think, will check...)

I've been meaning to ask you: in the long run, do you consider
separating company's frontend logic completely so that a simpler
mode non-company completion frontend can be integrated with
emacs (one that would use company-capf exclusively)?

João

On Tue, Dec 11, 2018 at 6:09 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 15.10.2018 1:08, João Távora wrote:
>
> >     (joaot/time
> >      (catch 'done
> >        (progn (make-process :name "test"
> >                             :filter (lambda (proc string)
> >                                       (message "Hey %s just got %s" proc
> string)
> >                                       (throw 'done nil))
> >                             :command '("sh" "-c" "sleep 2 && echo bla"))
> >               (while (sit-for 30))))) ; Took 02.011 seconds and returned
> nil
>
> It's an interesting alternative to Company's async backend interface,
> where we have a similar piece of code waiting until either completions
> are returned or the user types something (in company--fetch-candidates).
>
> Which of course makes sense, given that CAPF has no async calling
> convention.
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 2033 bytes --]

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Dmitry Gutov]

On 12.12.2018 23:42, João Távora wrote:

> I've been meaning to ask you: in the long run, do you consider
> separating company's frontend logic completely so that a simpler
> mode non-company completion frontend can be integrated with
> emacs (one that would use company-capf exclusively)?

Do you mean visualizations only? That would need some more code for 
company-capf to work. I guess if somebody used the pseudo-tooltip code 
(and something else) to visualize completion-at-point completions, that 
would be fine. I have no actual plans to implement that myself, though.

Otherwise, I'm not sure what you're proposing exactly, and what is the 
benefit vs just bundling Company with Emacs.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Dmitry Gutov]

On 12.12.2018 23:42, João Távora wrote:
> Indeed, I developed this for `company-capf`-based backends.

I imagine the chief drawback of this approach is if the user would try 
to "group" several backends like this, Company style.

CAPF has no handy way to do this, but we could add a macro in the 
future, and anyway, it's already possible to do this by means of 
function composition. It would be handy if, when we get to this point, 
several completion tables like this could fetch their completions 
simultaneously.

Just something to think about, for the future. In the meantime, LSP 
style with a smart server and one backend working with it is probably 
good enough for >90% users anyway.

bug#32986: 27.0.50; unexpected delay in while-no-input + accept-process-output

[Stefan Monnier]

>> I would expect while-no-input to break out of accept-process-output very
>> quickly after user keyboard input.  These expectations are met except
>> for some situations.
> I think your expectations are incorrect.

I beg to differ.

> My expectations are that if you call accept-process-output with
> a timeout of 30 sec, then while-no-input will return after 30 sec plus
> a small delay.

I'm OK if you don't consider it a "bug", but it does not satisfy the
needs behind `while-no-input`, so I think it's definitely something that
we should aim to improve.

The needs behind `while-no-input` often fail to be satisfied by our
C code, so this is not the first nor the last problem with it.
But if we can fix this case, Emacs will be better for it.

> First, one basic fact: accept-process-output calls
> wait_reading_process_output in a way that instructs it not to wait for
> keyboard input, you can see that clearly in the code (read_kbd is
> passed as zero).

What happens if we change this?

> This means that wait_reading_process_output will
> call pselect with the timeout of 30 sec (in your case), and will wait
> that long before it returns (unless there's a subprocess that gives us
> some stuff).  So why would you expect while-no-input that calls
> accept-process-output with that timeout to return earlier?

The question is not "given the current implementation, why should it
behave differently".  I expect it to return immediately because Emacs
is just there sitting idly so there is no good reason why it shouldn't
also check keyboard input at the same time as it's checking
process output.

> Therefore, to see the 30-sec delay, you need:
>
>   . stop all timers, which in "emacs -Q" means:
>     . disable blink-cursor-mode
>     . disable global-eldoc-mode
>     . disable font-lock-mode
>     . cancel the undo--auto-bindary-timer (I did that via list-timers)
>   . type "C-u C-x C-e", wait for a few seconds, and only then type SPC
>
> If I do all of the above, I see 30 sec plus a small delay every time I
> run your 2nd test.

IOW you cannot reliably expect it to wait 30s, so changing it so it
returns more promptly won't break any code.  That's good news.
All we need is for someone to come up with a patch.


        Stefan