Re: [Emacs-diffs] master 55ec674: * lisp/multifile.el: New file, extracted from etags.el

[Dmitry Gutov <dgutov@yandex.ru>]

On 03.01.2019 0:53, Juri Linkov wrote:
> On the very old computer from the year 2010, but the most interesting are relative times:
> 
> (benchmark 10 '(project-find-regexp "xyz1"))
> => 7s
> (benchmark 10 '(project-files-pipe-grep "xyz1"))
> => 17s

This is too bad. But did you use project-files-pipe-grep from 
446bcaed37b66ec112aaec7a7960e20b969c8012 or from 
c708231803712bd37154c140afdfd8468cac603e?

It would be helpful to test both implementations.

> (benchmark 10 '(project-files (project-current t)))
> => 11s

This is weird. I can understand that listing all files can be slower on 
an old, HDD-based computer. But both project-find-regexp and 
project-files use 'find ... -path], and the former even adds Grep on top 
of it. Why is the "simpler" operation slower?

Is it about piping the long list of files to Emacs? Why is the 'git 
ls-files' example so fast, then? It returns the same long list.

> (benchmark 10 '(shell-command-to-string "find ... \\( -path ... \\) -prune -o  -type f -print0"))
> => 11s
> 
> (benchmark 10 '(shell-command-to-string "git ls-files"))
> => 0.07s

Could you try making a full project-files implementation on top of it? I 
wonder how much slower it will be.

At least test (split-string (shell-command-to-string "git ls-files -z") 
"\0" t)

> IMHO, everything is clear: “find” with “-path” filters is slow,
> whereas “git ls-files” is fast.

We're all aware that 'git ls-files' is fast.

But not every project backend is going to be using 'git ls-files' (or a 
Git repository). So we should make sure that project-find-regexp does 
not noticeably regress when using the fallback implementation of 
project-files (based on 'find') if we're going to change its 
implementation to be based on project-files. Or regresses as little as 
possible.

And then we can implement a faster project-files for the built-in 
project backend (based on VC), but only, again, when used with a VCS 
that supports fast fetching of the list of files.

And to put it in a different perspective: the difference in speed that 
you see between project-find-regexp and project-files-pipe-grep is from 
some overhead somewhere. And the same overhead is likely to manifest 
even if project-files is based on 'git ls-files'.