Re: Async IO and queing process sentinels (was: Concurrency via isolated process/thread)

[Ihor Radchenko <yantar92@posteo.net>]

Ihor Radchenko <yantar92@posteo.net> writes:

> I also did testing locally, on my $HOME dir, and got similar results
> with recursive calls taking most of the CPU time:
>
>          463  29%            - directory-files-recursively
>          434  27%             - directory-files-recursively
>          305  19%              - directory-files-recursively
> ...
>
> If you want to improve performance here, you likely need to rewrite
> `directory-files-recursively' without recursion. Doable, and nothing to
> do with IO slowness. 

Not recursion, actually. I did a bit more elaborate profiling with perf
and it looks like most of the time is spent matching regexp against file
names:

The actual tested command was (ignore (directory-files-recursively "/home/yantar92/.data" ".+"))

    33.82%  emacs         emacs                        [.] re_match_2_internal
    14.87%  emacs         emacs                        [.] process_mark_stack
     8.47%  emacs         emacs                        [.] Fnconc
     6.07%  emacs         emacs                        [.] re_search_2
     2.36%  emacs         emacs                        [.] unbind_to
     2.08%  emacs         emacs                        [.] sweep_strings
     1.98%  emacs         emacs                        [.] compile_pattern
     1.64%  emacs         emacs                        [.] execute_charset
     1.64%  emacs         emacs                        [.] assq_no_quit
     1.40%  emacs         emacs                        [.] sweep_conses
     1.01%  emacs         emacs                        [.] plist_get
     0.97%  emacs         emacs                        [.] set_buffer_internal_2
     0.86%  emacs         emacs                        [.] RE_SETUP_SYNTAX_TABLE_FOR_OBJECT
     0.84%  emacs         emacs                        [.] mark_interval_tree_1
     0.75%  emacs         emacs                        [.] internal_equal
     0.73%  emacs         emacs                        [.] Ffind_file_name_handler

There was quite a bit of GC (not included into Elisp profile), that
shows up as *mark* and *sweep* calls.

No IO at all shows up in the backtrace. Even Ffind_file_name_handler is
simply matching filename against `file-name-handler-alist', calling
`insert-directory-program' somewhere in the process (the call does not
show up anywhere high in the profile).

Most of the time is spent doing regexp matching in various places and
building the actual (long) list of files.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>