bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Édouard Debry]

Hello,

On windows 10, I noticed that emacs could use a lot of memory, up 
to 6Go and
sometimes more. The memory consumption goes up and down from time 
to time while
I am not running any specific program in emacs, apart `lsp-java`

As my personal computer has 16Go of RAM, I can afford 
it. Unfortunately, my work
computer has much less and the whole compute completely freezes at 
one point
when using emacs.

I did not notice that behavior on linux. I do not know if the 
master branch has
the same problem. What could be the problem ?

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Édouard Debry <edouard.debry@gmail.com> writes:

> Hello,
>
> On windows 10, I noticed that emacs could use a lot of memory, up to
> 6Go and
> sometimes more. The memory consumption goes up and down from time to
> time while
> I am not running any specific program in emacs, apart `lsp-java`
>
> As my personal computer has 16Go of RAM, I can afford
> it. Unfortunately, my work
> computer has much less and the whole compute completely freezes at one
> point
> when using emacs.
>
> I did not notice that behavior on linux. I do not know if the master
> branch has
> the same problem. What could be the problem ?

Hi Édouard,

AFAIK was never proved recently Emacs garbage collector is failing to
recall memory, so I guess this is just some Lisp program that is
allocating a lot of memory keeping then those objects referenced.

Others may have more detaliled info about.

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Édouard Debry <edouard.debry@gmail.com> writes:
>
>> Hello,
>>
>> On windows 10, I noticed that emacs could use a lot of memory, up to
>> 6Go and
>> sometimes more. The memory consumption goes up and down from time to
>> time while
>> I am not running any specific program in emacs, apart `lsp-java`
>>
>> As my personal computer has 16Go of RAM, I can afford
>> it. Unfortunately, my work
>> computer has much less and the whole compute completely freezes at one
>> point
>> when using emacs.
>>
>> I did not notice that behavior on linux. I do not know if the master
>> branch has
>> the same problem. What could be the problem ?
>
> Hi Édouard,
>
> AFAIK was never proved recently Emacs garbage collector is failing to
> recall memory, so I guess this is just some Lisp program that is
> allocating a lot of memory keeping then those objects referenced.
>
> Others may have more detaliled info about.
>
>   Andrea
>

IOW I believe this is a duplicate of bug#43389.  Should we merge these?

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Eli Zaretskii]

> Date: Wed, 06 Jan 2021 20:55:35 +0000
> Cc: 45705@debbugs.gnu.org
> From: Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> > On windows 10, I noticed that emacs could use a lot of memory, up to
> > 6Go and
> > sometimes more. The memory consumption goes up and down from time to
> > time while
> > I am not running any specific program in emacs, apart `lsp-java`
> >
> > As my personal computer has 16Go of RAM, I can afford
> > it. Unfortunately, my work
> > computer has much less and the whole compute completely freezes at one
> > point
> > when using emacs.
> >
> > I did not notice that behavior on linux. I do not know if the master
> > branch has
> > the same problem. What could be the problem ?
> 
> Hi Édouard,
> 
> AFAIK was never proved recently Emacs garbage collector is failing to
> recall memory, so I guess this is just some Lisp program that is
> allocating a lot of memory keeping then those objects referenced.

IME, 6 GiB is too much for any Lisp program to explain away.

Also, the memory allocator we use on Windows is known to return memory
to the OS more than glibc on GNU/Linux.  I have never seen my Emacs
session get anywhere near 1 GiB, let alone more, and my sessions run
for many weeks without restarting.

Can you tell what kind of memory footprints you see on your system
with the native-comp branch, after running the session for several
days or more?

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 06 Jan 2021 20:55:35 +0000
>> Cc: 45705@debbugs.gnu.org
>> From: Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> > On windows 10, I noticed that emacs could use a lot of memory, up to
>> > 6Go and
>> > sometimes more. The memory consumption goes up and down from time to
>> > time while
>> > I am not running any specific program in emacs, apart `lsp-java`
>> >
>> > As my personal computer has 16Go of RAM, I can afford
>> > it. Unfortunately, my work
>> > computer has much less and the whole compute completely freezes at one
>> > point
>> > when using emacs.
>> >
>> > I did not notice that behavior on linux. I do not know if the master
>> > branch has
>> > the same problem. What could be the problem ?
>> 
>> Hi Édouard,
>> 
>> AFAIK was never proved recently Emacs garbage collector is failing to
>> recall memory, so I guess this is just some Lisp program that is
>> allocating a lot of memory keeping then those objects referenced.
>
> IME, 6 GiB is too much for any Lisp program to explain away.
>
> Also, the memory allocator we use on Windows is known to return memory
> to the OS more than glibc on GNU/Linux.  I have never seen my Emacs
> session get anywhere near 1 GiB, let alone more, and my sessions run
> for many weeks without restarting.
>
> Can you tell what kind of memory footprints you see on your system
> with the native-comp branch, after running the session for several
> days or more?

Hi Eli,

consumptions of few GiB is something I've seen more then once for long
standing sessions.  You might be right in this being a memory leak,
indeed I've no prove of that (I think we have none for the other
direction either).

Assuming it's a bug I don't see a priori why this should be a different
one respect the one reported for master.

Regards

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
> Date: Fri, 08 Jan 2021 15:50:28 +0000
> 
> consumptions of few GiB is something I've seen more then once for long
> standing sessions.  You might be right in this being a memory leak,
> indeed I've no prove of that (I think we have none for the other
> direction either).

I'm not yet worried about memory leaks, I'm more worried that there's
no memory leaks, and instead using libgccjit indeed requires such
large memory amounts.  Are you sure this is not the case?

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
>> Date: Fri, 08 Jan 2021 15:50:28 +0000
>>
>> consumptions of few GiB is something I've seen more then once for long
>> standing sessions.  You might be right in this being a memory leak,
>> indeed I've no prove of that (I think we have none for the other
>> direction either).
>
> I'm not yet worried about memory leaks, I'm more worried that there's
> no memory leaks, and instead using libgccjit indeed requires such
> large memory amounts.  Are you sure this is not the case?

I see,

if we are interested in comparing the memory footprint of using shared
libraries for functions vs stock byte-code I think we can "statically"
compare two sessions after the startup.

I've compiled current native-comp with and without --with-nativecomp
repeating the experiment with and without X.  These are the data-points:

  |         | --without-x | --without-x --with-nativecomp |      |
  |---------+-------------+-------------------------------+------|
  | -Q      | 49M         | 92M                           | 1.9x |
  | my-conf | 92M         | 179M                          | 1.9x |
  
  
  |         |      | --with-nativecomp |      |
  |---------+------+-------------------+------|
  | -Q      | 536M | 756M              | 1.4x |
  | my-conf | 585M | 1453M             | 2.4x |

So yes shared are using considerably more memory, I think this is
expected as also the file footprint suggests native code is less dense
that byte-code (actually with a quite similar relative results).

Indeed *with use the delta should decay as data are the same and there's
no difference in its representation*, so this picture should be more on
the worst case side than on the average.

Also if we want to see a positive side, multiple Emacs sessions will
share the majority the pages allocated for the shared libraries.

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
> Date: Fri, 08 Jan 2021 22:02:38 +0000
> 
> I've compiled current native-comp with and without --with-nativecomp
> repeating the experiment with and without X.  These are the data-points:
> 
>   |         | --without-x | --without-x --with-nativecomp |      |
>   |---------+-------------+-------------------------------+------|
>   | -Q      | 49M         | 92M                           | 1.9x |
>   | my-conf | 92M         | 179M                          | 1.9x |
>   
>   
>   |         |      | --with-nativecomp |      |
>   |---------+------+-------------------+------|
>   | -Q      | 536M | 756M              | 1.4x |
>   | my-conf | 585M | 1453M             | 2.4x |
> 
> So yes shared are using considerably more memory, I think this is
> expected as also the file footprint suggests native code is less dense
> that byte-code (actually with a quite similar relative results).

Thanks for the data points.

What about memory usage when there's a background compilation of Lisp
going on?  GCC is known to be a memory hog in some cases, so I wonder
what happens in this case with libgccjit.

(Do we allow multiple async compilations, btw? if so, how many
concurrent compilations can be running, and how do we or the user
control that?)

Also, what are the numbers for a session that has been running for
several days?  I understand that it would be hard for you to collect
such numbers about all the configurations, but could you show the
growth of the configuration you are routinely using, which I presume
is --with-x --with-nativecomp and with your config?  As your numbers
above show, it starts at 1.5 GiB, but what is the footprint after a
day or a week?

> Indeed *with use the delta should decay as data are the same and there's
> no difference in its representation*, so this picture should be more on
> the worst case side than on the average.

That's why I asked to see the memory footprint after the session has
run for some time, yes.

> Also if we want to see a positive side, multiple Emacs sessions will
> share the majority the pages allocated for the shared libraries.

Assuming those sessions run the same binary of Emacs, yes.  Otherwise,
only some of them will be shared, the ones that are in common public
directories, and even that only if the running Emacsen are of the same
version.  Most of the shared code is in the pdumper file, btw, which
is outside of the picture for the purposes of this discussion.

Thanks.

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
>> Date: Fri, 08 Jan 2021 22:02:38 +0000
>> 
>> I've compiled current native-comp with and without --with-nativecomp
>> repeating the experiment with and without X.  These are the data-points:
>> 
>>   |         | --without-x | --without-x --with-nativecomp |      |
>>   |---------+-------------+-------------------------------+------|
>>   | -Q      | 49M         | 92M                           | 1.9x |
>>   | my-conf | 92M         | 179M                          | 1.9x |
>>   
>>   
>>   |         |      | --with-nativecomp |      |
>>   |---------+------+-------------------+------|
>>   | -Q      | 536M | 756M              | 1.4x |
>>   | my-conf | 585M | 1453M             | 2.4x |
>> 
>> So yes shared are using considerably more memory, I think this is
>> expected as also the file footprint suggests native code is less dense
>> that byte-code (actually with a quite similar relative results).
>
> Thanks for the data points.
>
> What about memory usage when there's a background compilation of Lisp
> going on?  GCC is known to be a memory hog in some cases, so I wonder
> what happens in this case with libgccjit.

In June we changed the way we store immediate objects in the shared and
this makes the compilation way lighter on the GCC side (both in time and
memory).  I've no precise data on this other than the experimental
observation that compiling all Elisp files in Emacs on 32bit systems is
not anymore an issue.  This IIUC implies that the memory footprint for
each compilation is always < 2GB.

As a note: in all cases except bootstrap the final pass (the one driving
libgccjit) is executed as a sub-process, this to protect us from
eventual GCC leaks and not to generate unnecessary fragmentation.  In
async compilation we indeed run all the compilation (also the Lisp
computation) in the child process, so compiling should not have impact
on the memory footprint of the main Emacs session.

> (Do we allow multiple async compilations, btw? if so, how many
> concurrent compilations can be running, and how do we or the user
> control that?)

Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

> Also, what are the numbers for a session that has been running for
> several days?  I understand that it would be hard for you to collect
> such numbers about all the configurations, but could you show the
> growth of the configuration you are routinely using, which I presume
> is --with-x --with-nativecomp and with your config?  As your numbers
> above show, it starts at 1.5 GiB, but what is the footprint after a
> day or a week?

ATM I can provide this number, this is an Aarch64 daemon compiled with
'--without-x' with an up-time of 25 days and is showing a footprint of
765M.

>> Indeed *with use the delta should decay as data are the same and there's
>> no difference in its representation*, so this picture should be more on
>> the worst case side than on the average.
>
> That's why I asked to see the memory footprint after the session has
> run for some time, yes.

The hard part is to have a reference to compare against as the memory
footprint is strictly connected to the usage.  One with very regular
working habits should work like one week on vanilla and one week on
native-comp to make a comparison.  I've no regular working habits so I
fear I'm not the best fit for this comparison.

Thanks

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
> Date: Sat, 09 Jan 2021 10:55:23 +0000
> 
> > What about memory usage when there's a background compilation of Lisp
> > going on?  GCC is known to be a memory hog in some cases, so I wonder
> > what happens in this case with libgccjit.
> 
> In June we changed the way we store immediate objects in the shared and
> this makes the compilation way lighter on the GCC side (both in time and
> memory).  I've no precise data on this other than the experimental
> observation that compiling all Elisp files in Emacs on 32bit systems is
> not anymore an issue.  This IIUC implies that the memory footprint for
> each compilation is always < 2GB.

You assume that the compilations are all done serially?  AFAIK, most
people build Emacs with "make -jN", so parallel compilation is an
important use case.

I guess we will have to collect the information about that, if you say
we don't have it now.

> As a note: in all cases except bootstrap the final pass (the one driving
> libgccjit) is executed as a sub-process, this to protect us from
> eventual GCC leaks and not to generate unnecessary fragmentation.  In
> async compilation we indeed run all the compilation (also the Lisp
> computation) in the child process, so compiling should not have impact
> on the memory footprint of the main Emacs session.

That's fine, but the memory footprint of such a subprocess is also of
interest, as it could be relevant to the overall memory pressure of
the OS, and thus indirectly on the parent Emacs process as well.

> > (Do we allow multiple async compilations, btw? if so, how many
> > concurrent compilations can be running, and how do we or the user
> > control that?)
> 
> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>

Thanks.  This needs further tuning, IMO, both per the FIXME
(i.e. provide a primitive to return the number of execution units),
and wrt the default value being half of the available units.  We
should pay attention to the system's load average as well, I think.

> > Also, what are the numbers for a session that has been running for
> > several days?  I understand that it would be hard for you to collect
> > such numbers about all the configurations, but could you show the
> > growth of the configuration you are routinely using, which I presume
> > is --with-x --with-nativecomp and with your config?  As your numbers
> > above show, it starts at 1.5 GiB, but what is the footprint after a
> > day or a week?
> 
> ATM I can provide this number, this is an Aarch64 daemon compiled with
> '--without-x' with an up-time of 25 days and is showing a footprint of
> 765M.

OK, thanks.

> The hard part is to have a reference to compare against as the memory
> footprint is strictly connected to the usage.  One with very regular
> working habits should work like one week on vanilla and one week on
> native-comp to make a comparison.  I've no regular working habits so I
> fear I'm not the best fit for this comparison.

I agree, these numbers still need to be collected.  Maybe we should
ask on emacs-devel that people who use the branch report their
numbers?

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
>> Date: Sat, 09 Jan 2021 10:55:23 +0000
>> 
>> > What about memory usage when there's a background compilation of Lisp
>> > going on?  GCC is known to be a memory hog in some cases, so I wonder
>> > what happens in this case with libgccjit.
>> 
>> In June we changed the way we store immediate objects in the shared and
>> this makes the compilation way lighter on the GCC side (both in time and
>> memory).  I've no precise data on this other than the experimental
>> observation that compiling all Elisp files in Emacs on 32bit systems is
>> not anymore an issue.  This IIUC implies that the memory footprint for
>> each compilation is always < 2GB.
>
> You assume that the compilations are all done serially?  AFAIK, most
> people build Emacs with "make -jN", so parallel compilation is an
> important use case.

Yeah, we can say this loose information is only a per compilation unit
upper-bound.

> I guess we will have to collect the information about that, if you say
> we don't have it now.

I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
produce quite accurate results on memory footprint during the
compilation process.  Perhaps he has time and he's so kind to gather
some data on the current state, that would be extremely helpful.

>> As a note: in all cases except bootstrap the final pass (the one driving
>> libgccjit) is executed as a sub-process, this to protect us from
>> eventual GCC leaks and not to generate unnecessary fragmentation.  In
>> async compilation we indeed run all the compilation (also the Lisp
>> computation) in the child process, so compiling should not have impact
>> on the memory footprint of the main Emacs session.
>
> That's fine, but the memory footprint of such a subprocess is also of
> interest, as it could be relevant to the overall memory pressure of
> the OS, and thus indirectly on the parent Emacs process as well.

Indeed, I thought was relevant to make you aware of this mechanism.

>> > (Do we allow multiple async compilations, btw? if so, how many
>> > concurrent compilations can be running, and how do we or the user
>> > control that?)
>> 
>> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2>
>
> Thanks.  This needs further tuning, IMO, both per the FIXME
> (i.e. provide a primitive to return the number of execution units),
> and wrt the default value being half of the available units.  We
> should pay attention to the system's load average as well, I think.

Agree, I guess we'll be able to tune it better when we'll have more
data.  We could even think of using the memory pressure on the system to
make such a decision.

>> > Also, what are the numbers for a session that has been running for
>> > several days?  I understand that it would be hard for you to collect
>> > such numbers about all the configurations, but could you show the
>> > growth of the configuration you are routinely using, which I presume
>> > is --with-x --with-nativecomp and with your config?  As your numbers
>> > above show, it starts at 1.5 GiB, but what is the footprint after a
>> > day or a week?
>> 
>> ATM I can provide this number, this is an Aarch64 daemon compiled with
>> '--without-x' with an up-time of 25 days and is showing a footprint of
>> 765M.
>
> OK, thanks.
>
>> The hard part is to have a reference to compare against as the memory
>> footprint is strictly connected to the usage.  One with very regular
>> working habits should work like one week on vanilla and one week on
>> native-comp to make a comparison.  I've no regular working habits so I
>> fear I'm not the best fit for this comparison.
>
> I agree, these numbers still need to be collected.  Maybe we should
> ask on emacs-devel that people who use the branch report their
> numbers?

Is a good idea, my fear would be only to have very noisy or hard to
interpret results.  A priori I think would be nice to collect such data
also for master too.

Thanks

  Andrea

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 2626 bytes --]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> 
>>> In June we changed the way we store immediate objects in the shared and
>>> this makes the compilation way lighter on the GCC side (both in time and
>>> memory).  I've no precise data on this other than the experimental
>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>> each compilation is always < 2GB.
>>
>> You assume that the compilations are all done serially?  AFAIK, most
>> people build Emacs with "make -jN", so parallel compilation is an
>> important use case.
>
>> I guess we will have to collect the information about that, if you say
>> we don't have it now.
>
> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
> produce quite accurate results on memory footprint during the
> compilation process.  Perhaps he has time and he's so kind to gather
> some data on the current state, that would be extremely helpful.

See also bug#41194#20 and bug#41194#28 where I outlined how the
improvements reduced compilation time and memory usage.

I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
in need of… something (probably exorcism, given the noise).

Until I figure that out, here are the (very hacky) scripts I used to
measure and plot the RAM usage, in case someone else wants to take some
measurements:

- ./monitor.sh $PID finds the most RAM-consuming process among $PID and
  its children, and logs its memory usage (VSZ and RSS) and its
  command-line.

  (Logs are collected every 10 seconds; this probably needs to be
  reduced for faster machines)

- ./plot.py uses matplotlib to make graphs out of these measurements; it
  attempts to replace the command line with the less-verbose diagnostics
  from "make".

- My workflow was to start an emacs session, run M-x compile RET make,
  then ./monitor.sh $PID_OF_EMACS_SESSION.

  (PARENT_RE in plot.py should match the command-line of this parent
   session; its RAM consumption is then labeled as "noise floor" on the
   graph.

   This serves no real purpose and should be removed; monitor.sh should
   be amended to filter the parent session out of monitored PIDs, with
   some error control to handle the lack of child processes when
   compilation is finished.)

- There are some hardcoded things to tweak at the bottom of plot.py,
  e.g. how long should a child process last for it to have a label on
  the graph.


[-- Attachment #2: monitor.sh --]
[-- Type: application/x-shellscript, Size: 350 bytes --]

[-- Attachment #3: plot.py --]
[-- Type: text/x-python, Size: 5200 bytes --]

#!/usr/bin/env python3

from datetime import datetime, timedelta
from pathlib import Path
import re

import matplotlib
from matplotlib import pyplot
from matplotlib.dates import DateFormatter, HourLocator, MinuteLocator
from matplotlib.ticker import EngFormatter


MONITOR_RE = re.compile('\n'.join((
    '(?P<time>.+)',
    r' *(?P<seconds>\d+) +(?P<vsz>\d+) +(?P<rss>\d+) +(?P<args>.+)',
    ' *(?P<memheader>.+)',
    'Mem: *(?P<memvalues>.+)',
    'Swap: *(?P<swapvalues>.+)',
    ''
)), flags=re.MULTILINE)


def list_snapshots(monitor_log):
    snapshots = []

    for match in MONITOR_RE.finditer(monitor_log):
        md = match.groupdict()

        memkeys = md['memheader'].split()
        memvalues = md['memvalues'].split()
        swapvalues = md['swapvalues'].split()

        snapshot = {
            'time': datetime.strptime(md['time'], '%Y-%m-%d-%H:%M:%S'),
            'uptime': int(md['seconds']),
            'vsz': int(md['vsz'])*1024,
            'rss': int(md['rss'])*1024,
            'process': md['args'],
            'mem': {memkeys[i]: int(val)*1024 for i, val in enumerate(memvalues)},
            'swap': {memkeys[i]: int(val)*1024 for i, val in enumerate(swapvalues)}
        }

        snapshots.append(snapshot)

    return snapshots


LOADDEFS_RE = re.compile(
    r'--eval \(setq generated-autoload-file'
    r' \(expand-file-name \(unmsys--file-name "([^"]+)"\)\)\)'
    r' -f batch-update-autoloads'
)

SEMANTIC_RE = re.compile(
    r'-l semantic/(?:wisent|bovine)/grammar -f (?:wisent|bovine)-batch-make-parser'
    r' -o (.+) .+\.[wb]y'
)

ELCELN_RE = re.compile(
    r'\.\./src/(?:bootstrap-)?emacs -batch --no-site-file --no-site-lisp'
    r' --eval \(setq load-prefer-newer t\) -l comp'
    r'(?: -f byte-compile-refresh-preloaded)?'
    r' -f batch-byte-native-compile-for-bootstrap'
    r' (.+\.el)'
)

SHORTENED_NAMES = {
    LOADDEFS_RE: 'GEN',
    SEMANTIC_RE: 'GEN',
    ELCELN_RE: 'ELC+ELN'
}

QUAIL_TIT_RE = re.compile(
    r'-l titdic-cnv -f batch-titdic-convert'
    r' -dir \./\.\./lisp/leim/quail CXTERM-DIC/(.+)\.tit'
)

QUAIL_MISC_RE = re.compile(
    r'-l titdic-cnv -f batch-miscdic-convert'
    r' -dir \./\.\./lisp/leim/quail MISC-DIC/(.+\.(html|map|cin|cns|b5))'
)

QUAIL_JA_RE = re.compile(
    r'-l ja-dic-cnv -f batch-skkdic-convert'
)

PARENT_RE = re.compile(
    r'$^'                       # Adjust to match parent process.
)

TRANSFORMED_NAMES = {
    QUAIL_TIT_RE: lambda m: f'GEN ../lisp/leim/quail/{m.group(1)}.el',
    QUAIL_MISC_RE: lambda m: f'GEN from {m.group(1)}',
    QUAIL_JA_RE: lambda m: f'GEN ../lisp/leim/ja-dic/ja-dic.el',
    PARENT_RE: lambda _: '(noise floor)'
}

def shorten(process):
    for r, name in SHORTENED_NAMES.items():
        match = r.search(process)
        if match is not None:
            return f'{name} {match.group(1)}'

    for r, transform in TRANSFORMED_NAMES.items():
        match = r.search(process)
        if match is not None:
            return transform(match)

    if len(process) > 40:
        return f'{process[:20]}…{process[-20:]}'
    return process


def list_processes(snapshots):
    t0 = snapshots[0]['time']
    current_process = snapshots[0]['process']
    current_process_start = t0

    processes = []

    for s in snapshots[1:]:
        if s['process'] == current_process:
            continue

        s_start = s['time']
        processes.append((
            current_process, current_process_start, s_start-current_process_start
        ))

        current_process = s['process']
        current_process_start = s_start

    processes.append((
        current_process,
        current_process_start,
        snapshots[-1]['time']-current_process_start
    ))

    return processes


snapshots = list_snapshots(Path('monitor.log').read_text())

xs = tuple(s['time'] for s in snapshots)
vsz = tuple(s['vsz'] for s in snapshots)
rss = tuple(s['rss'] for s in snapshots)
memavail = tuple(s['mem']['available'] for s in snapshots)
swapused = tuple(s['swap']['used'] for s in snapshots)

matplotlib.use('TkAgg')
fig, axes = pyplot.subplots(figsize=(128, 9.6))
axes.plot(xs, vsz, label='VSZ (process)')
axes.plot(xs, rss, label='RSS (process)')
axes.plot(xs, memavail, label='available memory (system)', linewidth=0.5)
axes.plot(xs, swapused, label='used swap (system)')

axes.set_xlim(snapshots[0]['time'], snapshots[-1]['time'])
axes.xaxis.set_major_formatter(DateFormatter('%H:%M'))
axes.xaxis.set_major_locator(HourLocator())
axes.xaxis.set_minor_locator(MinuteLocator(tuple(5*i for i in range(1, 12))))
axes.xaxis.set_label_text('Hours')
axes.set_ylim(0)
axes.yaxis.set_major_formatter(EngFormatter(unit='B'))
axes.legend()

for p, start, duration in list_processes(snapshots):
    if duration < timedelta(minutes=2):
        continue
    pyplot.text(start, 1e9, shorten(p), rotation=45)
    pyplot.plot(
        (start, start+duration), (1e9, 1e9),
        marker='|', linewidth=0.5, linestyle='--',
        color='black', alpha=0.8
    )

pyplot.savefig('monitor.pdf')
pyplot.show()

bug#45705: [feature/native-comp] Excessive memory consumption on windows 10

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <akrl@sdf.org>
>>>> 
>>>> In June we changed the way we store immediate objects in the shared and
>>>> this makes the compilation way lighter on the GCC side (both in time and
>>>> memory).  I've no precise data on this other than the experimental
>>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>>> each compilation is always < 2GB.
>>>
>>> You assume that the compilations are all done serially?  AFAIK, most
>>> people build Emacs with "make -jN", so parallel compilation is an
>>> important use case.
>>
>>> I guess we will have to collect the information about that, if you say
>>> we don't have it now.
>>
>> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
>> produce quite accurate results on memory footprint during the
>> compilation process.  Perhaps he has time and he's so kind to gather
>> some data on the current state, that would be extremely helpful.
>
> See also bug#41194#20 and bug#41194#28 where I outlined how the
> improvements reduced compilation time and memory usage.
>
> I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
> in need of… something (probably exorcism, given the noise).

At the moment I think we are interested more in the magnitude order and
also a measure on 64bit would be very interesting.

But you are right I see in bug#41194#20 bug#41194#28 you've already
measured the footprint with the new immediate handling!

IIRC the worstcase was org.el under 600MB.  I think till we get a more
recent measure we should assume this as number (I don't expect huge
changes tho).

Many thanks

  Andrea