Does emacs cache something on windows when invoking call-process?

Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

I have the same search command line which is practically instantaneous in the windows command prompt, but it takes several seconds when executed via call-process.

So I try it in the command prompt, it's instantaneous.

Then I try the same command line from emacs via call-process and it takes several seconds.

Subsequent calls via call-process are instantaneous until I don't use the feature for a while. Then the next invocation from emacs becomes slow again.

So this behavior raises 2 questions:

1. Why is the same command much slower when invoked via call-process?

2. What does that first call-process call does/caches in emacs which makes subsequent calls quick until it forgets something and it's slow again?

 

Re: Does emacs cache something on windows when invoking call-process?

[Eli Zaretskii]

> From: Eighty Megabytes <emacsuser@freemail.hu>
> Date: Tue,  9 Jul 2019 16:03:39 +0200 (CEST)
> 
> I have the same search command line which is practically instantaneous in the windows command prompt, but it takes several seconds when executed via call-process.
> 
> So I try it in the command prompt, it's instantaneous.
> 
> Then I try the same command line from emacs via call-process and it takes several seconds.

This doesn't happen to me: I get instantaneous response from Emacs as
well.  On the first attempt as well on the subsequent ones.

> 1. Why is the same command much slower when invoked via call-process?

Do you have some antivirus installed, per chance?

> 2. What does that first call-process call does/caches in emacs which makes subsequent calls quick until it forgets something and it's slow again?

AFAIK it does nothing inside Emacs or by Emacs.  It might do something
on your system level, though.  But that's out of scope of Emacs.

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> From:     Eli Zaretskii
> 
> This doesn't happen to me: I get instantaneous response from Emacs as
> well.  On the first attempt as well on the subsequent ones.

It does for me too usually with other commands.

It's ripgrep.

If I invoke simply ripgrep --help via emacs call-process then it's instantaneous:

"c:\\bin\rg.exe" --help

But if I do this search with it via call-process, so (call-process "c:\\bin\\rg.exe" nil t nil  args...))

"c:\\bin\rg.exe" --color never --no-heading --line-number -U --pcre2 --type php "function\s*fetch_object_info\s*\(|\*\s@method\s+[^ ]+\s+fetch_object_info\(|(\s|->|\$|::)fetch_object_info\s*=\s*|\*\s@property(-read|-write)?\s+([^ ]+\s+)&?\$fetch_object_info(\s+|$)|trait\s*fetch_object_info\s*\{|interface\s*fetch_object_info\s*\{|class\s*fetch_object_info\s*(extends|implements|\{)" c:/proj/test

then it takes 6-7 seconds from Emacs.

This search returns a single line as a result. It is run on a codebase of several dozen files, so not a huge codebase.   The exact same command always returns the resulting line instantaneously from a command prompt.

And that's what I dont't get that how it is possible that the exact same command is immediately gives the result from the command prompt and takes 6-7 seconds from call-process.

> 
> Do you have some antivirus installed, per chance?

Just the default windows defender. 

> 
> > 2. What does that first call-process call does/caches in emacs which makes 
> > subsequent calls quick until it forgets something and it's slow again?
> 
> AFAIK it does nothing inside Emacs or by Emacs.  It might do something
> on your system level, though.  But that's out of scope of Emacs.

If it happens on the system level then shouldn't it affect the same command when executed via a command prompt?

 

Re: Does emacs cache something on windows when invoking call-process?

[Eli Zaretskii]

> From: Eighty Megabytes <emacsuser@freemail.hu>
> Date: Tue,  9 Jul 2019 17:58:34 +0200 (CEST)
> 
> It does for me too usually with other commands.
> 
> It's ripgrep.

I don't have it installed (and don't intend doing so).  So if you can
reproduce this with other commands, which are available on any Windows
machine, we could compare experiences.

> But if I do this search with it via call-process, so (call-process "c:\\bin\\rg.exe" nil t nil  args...))
> 
> "c:\\bin\rg.exe" --color never --no-heading --line-number -U --pcre2 --type php "function\s*fetch_object_info\s*\(|\*\s@method\s+[^ ]+\s+fetch_object_info\(|(\s|->|\$|::)fetch_object_info\s*=\s*|\*\s@property(-read|-write)?\s+([^ ]+\s+)&?\$fetch_object_info(\s+|$)|trait\s*fetch_object_info\s*\{|interface\s*fetch_object_info\s*\{|class\s*fetch_object_info\s*(extends|implements|\{)" c:/proj/test
> 
> then it takes 6-7 seconds from Emacs.

I have no idea how rpgrep works, but if it accesses some database,
perhaps it takes time when the database is not in the system's disk
cache?  This is pure guessing on my part; please let's use a command
that is likely to be on any Windows machine.

> > AFAIK it does nothing inside Emacs or by Emacs.  It might do something
> > on your system level, though.  But that's out of scope of Emacs.
> 
> If it happens on the system level then shouldn't it affect the same command when executed via a command prompt?

I have no idea.  We are both guessing here.

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> I have no idea how rpgrep works, but if it accesses some database,
> perhaps it takes time when the database is not in the system's disk
> cache?

Then it would be slow from the command prompt too. I tried it first
from the command prompt and it was instantaneous. Then I switched to
emacs and run the exact same command again via call-process and it
took 6-7 seconds. (If I then try immediately from the command prompt 
again then there it's instantaneous again,)

> This is pure guessing on my part; please let's use a command
> that is likely to be on any Windows machine.

I tried it with grep and I don't see the slowdown there, so I'm
out of ideas. (Simply using grep is not the answer, because
command line ripgrep is several times faster than grep, that's
why I wanted to use it from emacs too.)

I'll think of other ways of tracking down the problem. E.g. using
some systrace tool or something too see where emacs spends that
6-7 seconds.
 

Re: Does emacs cache something on windows when invoking call-process?

[Eli Zaretskii]

> From: Eighty Megabytes <emacsuser@freemail.hu>
> Date: Tue,  9 Jul 2019 19:08:53 +0200 (CEST)
> 
> > This is pure guessing on my part; please let's use a command
> > that is likely to be on any Windows machine.
> 
> I tried it with grep and I don't see the slowdown there, so I'm
> out of ideas.

What kind of program is ripgrep?  Is it per chance a Cywgin or MSYS2
executable?  If so, perhaps the time you see is spent loading the
Cygwin/MSYS2 DLL.

> I'll think of other ways of tracking down the problem. E.g. using
> some systrace tool or something too see where emacs spends that
> 6-7 seconds.

Good idea.

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> What kind of program is ripgrep?  Is it per chance a Cywgin or MSYS2
> executable?  If so, perhaps the time you see is spent loading the
> Cygwin/MSYS2 DLL.

I ran the emacs test about 5 seconds after the command prompt test.
so I don't think windows would forget about the already loaded
DLLs in a such a short time.

I tried to trace the process and it seems to me when the search is run
via emacs then windows defender checks every source file read by ripgrep
and I assume this slows down the search.

When tracing an invocation via the command prompt then there are no checking
of source files, so it looks like windows defender thinks emacs is a suspicious
program or maybe it considers the invocation of cmdproxy.exe  suspicious.

With windows defender turned off for the given source directory the delay
practically disappears.

Re: Does emacs cache something on windows when invoking call-process?

[Stefan Monnier]

> With windows defender turned off for the given source directory the delay
> practically disappears.

One more case of virus software being not just useless but even harmful.
Just say no!


        Stefan

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> One more case of virus software being not just useless but even harmful.
> Just say no!

I wonder what the practical infection vectors for windows these days
if one is careful about installing stuff.

I don't install random binaries from the net. I do install precompiled
exes, but those are from known sources (e.g. git, emacs, etc.). 

Firefox is a possible vector, because one visits lots of random sites with
it, so it may be possible that a site exploits some yet unknown security
hole.

So the question is: does the latter warrant running a virus a scanner or
is the danger so low that it's better to keep windows defender off?

Re: Does emacs cache something on windows when invoking call-process?

[ISHIKAWA,chiaki]

On 2019/07/10 14:31, Eighty Megabytes wrote:
>> One more case of virus software being not just useless but even harmful.
>> Just say no!
> I wonder what the practical infection vectors for windows these days
> if one is careful about installing stuff.
>
> I don't install random binaries from the net. I do install precompiled
> exes, but those are from known sources (e.g. git, emacs, etc.).
>
> Firefox is a possible vector, because one visits lots of random sites with
> it, so it may be possible that a site exploits some yet unknown security
> hole.
>
> So the question is: does the latter warrant running a virus a scanner or
> is the danger so low that it's better to keep windows defender off?
>
>

You definitely turn the defender ON always unless you have a special 
anti-virus
program such as Norton Internet, etc. aside from windows defender.

Even the respectable newspaper websites these days contract with ads 
agencies to
fill the borders of their web pages with random ads.
And random ads are just that. We may never know what will be shown there.

There *HAVE* been reports of such random ads that insert malware into 
our PCs.

So unless you have a very strict ads blocker such as NoScript,
ad-blocking proxy such as privoxy,
and DNS-based site blocker, I won't feel comfortable.
And actually, I have Norton antivirus on top of the above three under 
windows10.

I don't have any anti-virus window software inside the linux image that 
runs within VirtualBox under Windows10.
I don't do web browsing there.

BTW, under windows10, I do hit Norton's access denied warning (from its 
safe web browsing) when
I wander into no one's territory after I find an interesting PR 
headline/article in a newspaper article
or sites such as gigazine, and maybe after three hops or so following 
the links.

Internet is a very interesting place and useful, but at the same time, 
it is really
full of pitfalls.

BTW,  I have a home FreeNAS server and it serves owncloud.
For remote maintenance, I have enabled remote ssh. But that is only 
accessible
with my PKI certificate. No username and password logins are possible.
Imagine my surprise when I found out that the box is bombarded with 
login attempts from the first week thee years ago.
Maybe more than 100 attempts weekly using various login names and 
password combinations.
(I have stopped counting the attempts after a few months. 
Weekly/Daily/Monthly summaries are sent by FreeNAS software.)
Sometimes a persistent attempt leaves more than a few dozen attempts 
from a single IP address.

So it is up to you to decide.

Chiaki

Re: Does emacs cache something on windows when invoking call-process?

[Eli Zaretskii]

> From: Eighty Megabytes <emacsuser@freemail.hu>
> Date: Wed, 10 Jul 2019 07:31:32 +0200 (CEST)
> 
> So the question is: does the latter warrant running a virus a scanner or
> is the danger so low that it's better to keep windows defender off?

IME, anti-virus software is generally OK (with minor exceptions --
there are a couple of them that should be avoided at all costs), but
you need to configure it so it gets out of your way.  For example,
binaries that you build yourself should be exempt from having all the
files they open examined, it's enough to have the binary itself
examined (to make sure no malware attaches itself to it).  I suspect
Emacs in your use case is one such program.

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> Even the respectable newspaper websites these days contract with ads agencies to
> 
> fill the borders of their web pages with random ads.
> And random ads are just that. We may never know what will be shown there.
> 
> There *HAVE* been reports of such random ads that insert malware into our PCs.


If the browser is the vector then  the solution could simply be running the browser in a sandbox
isolated from the rest of the system, so you don't need to do an antivirus for everything else.

Sure, a very determined attacker could break out of the browser and the sandbox around it,
but in practice most attackers care about infecting average users, so they don't think
about being in an other sandbox after cracking the browser's defenses.

Re: Does emacs cache something on windows when invoking call-process?

[Eighty Megabytes]

> IME, anti-virus software is generally OK (with minor exceptions --
> there are a couple of them that should be avoided at all costs), but
> you need to configure it so it gets out of your way.  For example,
> binaries that you build yourself should be exempt from having all the
> files they open examined, it's enough to have the binary itself
> examined (to make sure no malware attaches itself to it).  I suspect
> Emacs in your use case is one such program.


Only the default Windows Defender is on my machine, but it cannot
be configured, so a binary is "exempt from having all the files
they open examined". I can exclude processes from the check, or
directories, so in this case excluding source directories seemed
to be the way to go.

This made Dumb Jump usable for me on Windows. Dumb Jump is a code
navigation package which does not require building tags tables at
all (so they can't go out of sync), because ripgrep is so quick
that it simply greps the code to find the relevant location when
you want jump to it and it's instantaneous, especially after the
first run when the OS caches the source files in memory.