Speeding up Flymake in emacs-lisp-mode

Speeding up Flymake in emacs-lisp-mode

[João Távora]

Hi,

If you've tried M-x flymake-mode in emacs-lisp sources, you'll have
noticed there's a byte-compilation backend.  This relies on launching an
external 'emacs -Q' on an idle timer which does little but byte-compile
the a file with the buffer's contents and collect messages, which are
then used to annotate the buffer.

It works very well on Mac and Linux, but on windows is unusably slow
(presumably) because of a ~1 sec process-launching overhead.

It'd probably be better if only one process was launched and asked
(perhaps using server.el) to compile the files.  But this would have a
major disadvantage over the current approach, which is that the elisp
runtime is tainted by previous compilations.  The external process
approach doesn't have this disadvantage, since every emacs is clean when
launched.

Which brings me to my question: is there a way to re-flash the current
emacs's image with a brand new one, or to erase any side-effects of
previous byte-compilations? The only pitfall would be that I would need
server.el's bits to escape this reflashing otherwise I would be locked
out.

Thanks,
João

Re: Speeding up Flymake in emacs-lisp-mode

[Stefan Monnier]

> Which brings me to my question: is there a way to re-flash the current
> emacs's image with a brand new one, or to erase any side-effects of
> previous byte-compilations? The only pitfall would be that I would need
> server.el's bits to escape this reflashing otherwise I would be locked
> out.

There is no such thing, but you should be able to get some approximation
of it using the same approach I used on my "dump to .elc" attempts.

I.e. something like:

    (defun emacs-snapshot-get ()
      (let ((syms ()))
        (mapatoms
          (lambda (sym)
            (let ((ns (make-symbol (symbol-name sym))))
              (setf (symbol-function ns) (symbol-function sym))
              (setf (symbol-plist ns) (symbol-plist sym))
              (when (default-boundp sym)
                (setf (default-value ns) (default-value sym))))))
        syms))

    (defun emacs-snapshot-revert (snapshot)
      (dolist (sym snapshot)
        (let ((ns (intern (symbol-name sym))))
          (setf (symbol-function ns) (symbol-function sym))
          (setf (symbol-plist ns) (symbol-plist sym))
          (when (default-boundp sym)
            (setf (default-value ns) (default-value sym))))))

Of course, this may also just eat your lunch, so handle with care.


        Stefan "who was careful not to test it"

Re: Speeding up Flymake in emacs-lisp-mode

[Noam Postavsky]

On Fri, 2 Nov 2018 at 07:55, João Távora <joaotavora@gmail.com> wrote:

> It'd probably be better if only one process was launched and asked
> (perhaps using server.el) to compile the files.

Alternate idea: keep running a single compilation per Emac subprocess,
but tell it what to compile by sending to stdin instead of command
line args. That way, the process startup can happen asynchronously,
and when you want to compile something you can arrange to have a
subprocess ready and waiting for input.

Re: Speeding up Flymake in emacs-lisp-mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

On Fri, Nov 2, 2018 at 12:26 PM Noam Postavsky <npostavs@gmail.com> wrote:

> On Fri, 2 Nov 2018 at 07:55, João Távora <joaotavora@gmail.com> wrote:
>
> > It'd probably be better if only one process was launched and asked
> > (perhaps using server.el) to compile the files.
>
> Alternate idea: keep running a single compilation per Emac subprocess,
> but tell it what to compile by sending to stdin instead of command
> line args. That way, the process startup can happen asynchronously,
> and when you want to compile something you can arrange to have a
> subprocess ready and waiting for input.
>

Thanks.

First, I was under the impression that stdin/stdout for emacs was
not easy. Glad to hear it isn't, but how do I read from stdin?

Are you suggesting I keep a pool of ready to invoke emacsen
and discard them once they become "dirty"?

What do you mean by asynchronous process startup? Isn't that
a "stop-the-world" operation however it is invoked?

João Távora

[-- Attachment #2: Type: text/html, Size: 1551 bytes --]

Re: Speeding up Flymake in emacs-lisp-mode

[Stefan Monnier]

> First, I was under the impression that stdin/stdout for emacs was
> not easy.

Usually it's not, but when running in batch mode it's not so bad.

> Glad to hear it isn't, but how do I read from stdin?

async.el does it with (read t), IIRC.

> Are you suggesting I keep a pool of ready to invoke emacsen
> and discard them once they become "dirty"?

I think so, yes.  I'd limit the pool to a single process.
It should be easy to do directly within async.el.
Basically, change async-start so that instead of

    (let ((proc (async-start-process ...)))
      (async--transmit-sexp proc ...))

it does

    (let ((proc async--eager-process))
      (async--transmit-sexp proc ...)
      (setq async--eager-process (async-start-process ...)))


-- Stefan

Re: Speeding up Flymake in emacs-lisp-mode

[Noam Postavsky]

On Fri, 2 Nov 2018 at 09:12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > First, I was under the impression that stdin/stdout for emacs was
> > not easy.
>
> Usually it's not, but when running in batch mode it's not so bad.

Yup, (elisp) Batch Mode:

    input that would normally come from the minibuffer is read from the
    standard input descriptor.

> > Glad to hear it isn't, but how do I read from stdin?
>
> async.el does it with (read t), IIRC.

read-from-minibuffer should also work if non-sexp input is wanted.

>> What do you mean by asynchronous process startup? Isn't that
>> a "stop-the-world" operation however it is invoked?

The subprocess created by make-process (or start-process) doesn't
block the parent Emacs, it runs in parallel.

(benchmark
 1 '(call-process (concat invocation-directory invocation-name)
                  nil nil nil "-Q" "--batch"))
;=> "Elapsed time: 0.185000s"

(benchmark 1 '(start-process
               "emacs" nil (concat invocation-directory invocation-name)
               "-Q" "--batch"))
;=> "Elapsed time: 0.031000s"

Re: Speeding up Flymake in emacs-lisp-mode

[João Távora]

Noam Postavsky <npostavs@gmail.com> writes:

> On Fri, 2 Nov 2018 at 09:12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>> > First, I was under the impression that stdin/stdout for emacs was
>> > not easy.
>>
>> Usually it's not, but when running in batch mode it's not so bad.
>
> Yup, (elisp) Batch Mode:
>
>     input that would normally come from the minibuffer is read from the
>     standard input descriptor.
>
>> > Glad to hear it isn't, but how do I read from stdin?
>>
>> async.el does it with (read t), IIRC.
>
> read-from-minibuffer should also work if non-sexp input is wanted.
>
>>> What do you mean by asynchronous process startup? Isn't that
>>> a "stop-the-world" operation however it is invoked?
>
> The subprocess created by make-process (or start-process) doesn't
> block the parent Emacs, it runs in parallel.
>
> (benchmark
>  1 '(call-process (concat invocation-directory invocation-name)
>                   nil nil nil "-Q" "--batch"))
> ;=> "Elapsed time: 0.185000s"
>
> (benchmark 1 '(start-process
>                "emacs" nil (concat invocation-directory invocation-name)
>                "-Q" "--batch"))
> ;=> "Elapsed time: 0.031000s"
>

Of course I know it runs in parallel :-), but creating the process is
20x slower on windows.

   (benchmark 1 '(start-process
                  "emacs" nil (concat invocation-directory invocation-name)
                  "-Q" "--batch")) ;=> "Elapsed time: 0.679212s"

But now I see what you mean, I think!  You mean just put another
long-running emacs, reading sexps from stdin, but shielding the main
emacs from doing the slow process creation?

That should indeed improve things a lot and seems like a fairly clean
solution (even if it still means lots of processes): I think flymake
users are more or less OK with a long delay between typing and seeing
annotations, they just don't want this process to slow down actual
typing and editing.

João

Re: Speeding up Flymake in emacs-lisp-mode

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> First, I was under the impression that stdin/stdout for emacs was
>> not easy.
>
> Usually it's not, but when running in batch mode it's not so bad.
>
>> Glad to hear it isn't, but how do I read from stdin?
>
> async.el does it with (read t), IIRC.
>
>> Are you suggesting I keep a pool of ready to invoke emacsen
>> and discard them once they become "dirty"?
>
> I think so, yes.  I'd limit the pool to a single process.

Actually, now I think Noam was suggesting something else, which is
insulating slow process creation from the "main" emacs.  And that indeed
requires just a single extra process.

> It should be easy to do directly within async.el.
> Basically, change async-start so that instead of
>
>     (let ((proc (async-start-process ...)))
>       (async--transmit-sexp proc ...))
>
> it does
>
>     (let ((proc async--eager-process))
>       (async--transmit-sexp proc ...)
>       (setq async--eager-process (async-start-process ...)))

Yeah, but I don't see why I need to restart async--eager-process
(assuming that's what async-start-process does).

Is async.el in ELPA?

João

Re: Speeding up Flymake in emacs-lisp-mode

[Noam Postavsky]

On Fri, 2 Nov 2018 at 10:01, João Távora <joaotavora@gmail.com> wrote:

> But now I see what you mean, I think!  You mean just put another
> long-running emacs, reading sexps from stdin, but shielding the main
> emacs from doing the slow process creation?

I had actually meant something more like what Stefan suggested, but
your idea to add another layer of indirection sounds good. :)

Re: Speeding up Flymake in emacs-lisp-mode

[Stefan Monnier]

>>     (let ((proc async--eager-process))
>>       (async--transmit-sexp proc ...)
>>       (setq async--eager-process (async-start-process ...)))
>
> Yeah, but I don't see why I need to restart async--eager-process
> (assuming that's what async-start-process does).

Because the process that was in async--eager-process has been "consumed"
for the current sexp, so you need to create a new one which will be used
for the next request.

But indeed, if the problem is that async-start-process takes a long
*synchronous* time, then swapping the two won't help.

> Is async.el in ELPA?

Yes, and more specifically in GNU ELPA.


        Stefan