bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Eli Zaretskii <eliz@gnu.org>]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
> Date: Sat, 09 Jan 2021 10:55:23 +0000
> 
> > What about memory usage when there's a background compilation of Lisp
> > going on?  GCC is known to be a memory hog in some cases, so I wonder
> > what happens in this case with libgccjit.
> 
> In June we changed the way we store immediate objects in the shared and
> this makes the compilation way lighter on the GCC side (both in time and
> memory).  I've no precise data on this other than the experimental
> observation that compiling all Elisp files in Emacs on 32bit systems is
> not anymore an issue.  This IIUC implies that the memory footprint for
> each compilation is always < 2GB.

You assume that the compilations are all done serially?  AFAIK, most
people build Emacs with "make -jN", so parallel compilation is an
important use case.

I guess we will have to collect the information about that, if you say
we don't have it now.

> As a note: in all cases except bootstrap the final pass (the one driving
> libgccjit) is executed as a sub-process, this to protect us from
> eventual GCC leaks and not to generate unnecessary fragmentation.  In
> async compilation we indeed run all the compilation (also the Lisp
> computation) in the child process, so compiling should not have impact
> on the memory footprint of the main Emacs session.

That's fine, but the memory footprint of such a subprocess is also of
interest, as it could be relevant to the overall memory pressure of
the OS, and thus indirectly on the parent Emacs process as well.

> > (Do we allow multiple async compilations, btw? if so, how many
> > concurrent compilations can be running, and how do we or the user
> > control that?)
> 
> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

Thanks.  This needs further tuning, IMO, both per the FIXME
(i.e. provide a primitive to return the number of execution units),
and wrt the default value being half of the available units.  We
should pay attention to the system's load average as well, I think.

> > Also, what are the numbers for a session that has been running for
> > several days?  I understand that it would be hard for you to collect
> > such numbers about all the configurations, but could you show the
> > growth of the configuration you are routinely using, which I presume
> > is --with-x --with-nativecomp and with your config?  As your numbers
> > above show, it starts at 1.5 GiB, but what is the footprint after a
> > day or a week?
> 
> ATM I can provide this number, this is an Aarch64 daemon compiled with
> '--without-x' with an up-time of 25 days and is showing a footprint of
> 765M.

OK, thanks.

> The hard part is to have a reference to compare against as the memory
> footprint is strictly connected to the usage.  One with very regular
> working habits should work like one week on vanilla and one week on
> native-comp to make a comparison.  I've no regular working habits so I
> fear I'm not the best fit for this comparison.

I agree, these numbers still need to be collected.  Maybe we should
ask on emacs-devel that people who use the branch report their
numbers?