From mboxrd@z Thu Jan  1 00:00:00 1970
Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail
From: =?UTF-8?Q?K=C3=A9vin?= Le Gouguec <kevin.legouguec@gmail.com>
Newsgroups: gmane.emacs.bugs
Subject: bug#45705: [feature/native-comp] Excessive memory consumption on
 windows 10
Date: Sat, 09 Jan 2021 18:26:46 +0100
Message-ID: <87im86c97t.fsf@gmail.com>
References: <87y2h5hjve.fsf@gmail.com> <xjfft3ddbug.fsf@sdf.org>
 <831revjyja.fsf@gnu.org> <xjfh7nrbf7f.fsf@sdf.org>
 <83turrif53.fsf@gnu.org> <xjfa6tjaxz5.fsf@sdf.org>
 <83lfd2ilwf.fsf@gnu.org> <xjf5z46bcro.fsf@sdf.org>
 <83y2h2gw9v.fsf@gnu.org> <xjfy2h29tgd.fsf@sdf.org>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214";
	logging-data="6147"; mail-complaints-to="usenet@ciao.gmane.io"
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org
To: Andrea Corallo <akrl@sdf.org>
Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Jan 09 18:27:25 2021
Return-path: <bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org>
Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org
Original-Received: from lists.gnu.org ([209.51.188.17])
	by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org>)
	id 1kyI1M-0001Pv-An
	for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 09 Jan 2021 18:27:24 +0100
Original-Received: from localhost ([::1]:42100 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org>)
	id 1kyI1L-0007VR-5e
	for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 09 Jan 2021 12:27:23 -0500
Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:41916)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1kyI10-0007U1-8H
 for bug-gnu-emacs@gnu.org; Sat, 09 Jan 2021 12:27:02 -0500
Original-Received: from debbugs.gnu.org ([209.51.188.43]:40993)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1kyI10-0002Ko-1H
 for bug-gnu-emacs@gnu.org; Sat, 09 Jan 2021 12:27:02 -0500
Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1kyI0z-00035E-To
 for bug-gnu-emacs@gnu.org; Sat, 09 Jan 2021 12:27:01 -0500
X-Loop: help-debbugs@gnu.org
Resent-From: =?UTF-8?Q?K=C3=A9vin?= Le Gouguec <kevin.legouguec@gmail.com>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@gnu.org
Resent-Date: Sat, 09 Jan 2021 17:27:01 +0000
Resent-Message-ID: <handler.45705.B45705.161021321911842@debbugs.gnu.org>
Resent-Sender: help-debbugs@gnu.org
X-GNU-PR-Message: followup 45705
X-GNU-PR-Package: emacs
Original-Received: via spool by 45705-submit@debbugs.gnu.org id=B45705.161021321911842
 (code B ref 45705); Sat, 09 Jan 2021 17:27:01 +0000
Original-Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 17:26:59 +0000
Original-Received: from localhost ([127.0.0.1]:52536 helo=debbugs.gnu.org)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
 id 1kyI0x-00034t-8z
 for submit@debbugs.gnu.org; Sat, 09 Jan 2021 12:26:59 -0500
Original-Received: from mail-wm1-f43.google.com ([209.85.128.43]:55201)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <kevin.legouguec@gmail.com>) id 1kyI0s-00034S-Kg
 for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 12:26:58 -0500
Original-Received: by mail-wm1-f43.google.com with SMTP id c133so10288370wme.4
 for <45705@debbugs.gnu.org>; Sat, 09 Jan 2021 09:26:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; 
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=;
 b=LzikEmPfaMA8FZph1WMiCwbnuHdUccG9Osi4SAkQaLRiNPLYZh2cH7xdTZ6SR1gphP
 V/tU6jwvmHeqH6/cX/+5WMiwk9TNTHl63LPnHNuTcvRlnebfj8y0B5Ec/3as6i9td1Jp
 A21HlDgKmJ84whmWyiIzYCbOzsKHyS3tX8n6KS9sQAR9Odq9EPneVk/mqcbDemR9TEpe
 8DtYoVTS38WnmzWG3azYZQAzRRXFzzOR+lq/3M9DSew7ouXcu8uiqVYrnkjKGsAe01iE
 j13I+KJiuec66mrCDq1gZERByun7GbeN6jxbpoHDg6/ywN0aFh2x8ifX7BmU6uj1bhpP
 TuCg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=;
 b=eEwVXQuw5yhEzx0JpSva6VXy8/sooksTwjxHoaJBTdaSNozNlUHbW3q5ax/f9ZzwmN
 z3cOqL+W0ztDd/RFqbYH77qWhctvYqmvUOFHXZKnaMMYDfhkz/kLdXSw3ImTId5wRS9k
 fQyacal8GaaQLsyvcBd6QFcf8hkUxqKlUDB3Ss+fzCmeQgzhoST8Xp8vThwC1+QHh+fe
 EPEg7do45jvoLHvX+4XrWaP0lHbWhrmkEsQqL+gX28xeKPxXh1hO70N998gRN5chd3QM
 OsLsfB6bsoreQuzYiVUNfE/naddefQ3n4xV3nLEpIl43aP3NmAViOJlgHVDDZhBDZ4cY
 1dnQ==
X-Gm-Message-State: AOAM531REnwJZYkaG+h01g6qX1t0T5Ed1cjPlx9jio09aDSCy2r3wrka
 tKf8TpbZsmacO1198EyvTXRCQ0iDkPLFH8in
X-Google-Smtp-Source: ABdhPJz380Pp/cGH6Baq4VxLoEuRkjsiPsSYZEAXsYq2laUiz20L3FAwnY2rPzQ9X/U5s4Pa1m2jdw==
X-Received: by 2002:a1c:c254:: with SMTP id s81mr8048005wmf.132.1610213208414; 
 Sat, 09 Jan 2021 09:26:48 -0800 (PST)
Original-Received: from my-little-tumbleweed ([2a01:e0a:20e:d340:922b:34ff:fe95:9aed])
 by smtp.gmail.com with ESMTPSA id
 m21sm15979328wml.13.2021.01.09.09.26.47
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 09 Jan 2021 09:26:47 -0800 (PST)
In-Reply-To: <xjfy2h29tgd.fsf@sdf.org> (Andrea Corallo's message of "Sat, 09
 Jan 2021 12:37:54 +0000")
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
X-BeenThere: bug-gnu-emacs@gnu.org
List-Id: "Bug reports for GNU Emacs,
 the Swiss army knife of text editors" <bug-gnu-emacs.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-gnu-emacs>,
 <mailto:bug-gnu-emacs-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/bug-gnu-emacs>
List-Post: <mailto:bug-gnu-emacs@gnu.org>
List-Help: <mailto:bug-gnu-emacs-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-gnu-emacs>,
 <mailto:bug-gnu-emacs-request@gnu.org?subject=subscribe>
Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org
Original-Sender: "bug-gnu-emacs"
 <bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org>
Xref: news.gmane.io gmane.emacs.bugs:197560
Archived-At: <http://permalink.gmane.org/gmane.emacs.bugs/197560>

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>>=20
>>> In June we changed the way we store immediate objects in the shared and
>>> this makes the compilation way lighter on the GCC side (both in time and
>>> memory).  I've no precise data on this other than the experimental
>>> observation that compiling all Elisp files in Emacs on 32bit systems is
>>> not anymore an issue.  This IIUC implies that the memory footprint for
>>> each compilation is always < 2GB.
>>
>> You assume that the compilations are all done serially?  AFAIK, most
>> people build Emacs with "make -jN", so parallel compilation is an
>> important use case.
>
>> I guess we will have to collect the information about that, if you say
>> we don't have it now.
>
> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to
> produce quite accurate results on memory footprint during the
> compilation process.  Perhaps he has time and he's so kind to gather
> some data on the current state, that would be extremely helpful.

See also bug#41194#20 and bug#41194#28 where I outlined how the
improvements reduced compilation time and memory usage.

I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's
in need of=E2=80=A6 something (probably exorcism, given the noise).

Until I figure that out, here are the (very hacky) scripts I used to
measure and plot the RAM usage, in case someone else wants to take some
measurements:

- ./monitor.sh $PID finds the most RAM-consuming process among $PID and
  its children, and logs its memory usage (VSZ and RSS) and its
  command-line.

  (Logs are collected every 10 seconds; this probably needs to be
  reduced for faster machines)

- ./plot.py uses matplotlib to make graphs out of these measurements; it
  attempts to replace the command line with the less-verbose diagnostics
  from "make".

- My workflow was to start an emacs session, run M-x compile RET make,
  then ./monitor.sh $PID_OF_EMACS_SESSION.

  (PARENT_RE in plot.py should match the command-line of this parent
   session; its RAM consumption is then labeled as "noise floor" on the
   graph.

   This serves no real purpose and should be removed; monitor.sh should
   be amended to filter the parent session out of monitored PIDs, with
   some error control to handle the lack of child processes when
   compilation is finished.)

- There are some hardcoded things to tweak at the bottom of plot.py,
  e.g. how long should a child process last for it to have a label on
  the graph.


--=-=-=
Content-Type: application/x-shellscript
Content-Disposition: attachment; filename=monitor.sh
Content-Transfer-Encoding: base64

IyEvYmluL2Jhc2gKCnNldCAtZXUKCm91dD0kKGRpcm5hbWUgJDApL21vbml0b3IubG9nCnBhcmVu
dD0kMQoKcmVjdXJzZSAoKQp7CiAgICBlY2hvICQxCiAgICBmb3IgY2hpbGQgaW4gJChwcyAtbyBw
aWQ9IC0tcHBpZCAkMSkKICAgIGRvCiAgICAgICAgcmVjdXJzZSAkY2hpbGQKICAgIGRvbmUKfQoK
d2hpbGUgc2xlZXAgMTAKZG8KICAgIGRhdGUgKyIlRi0lVCIgPj4gJHtvdXR9CiAgICBwcyBvIGNw
dXRpbWVzPSx2c3o9LHJzcz0sYXJncz0gLS1zb3J0PS12c3ogcCAkKHJlY3Vyc2UgJHtwYXJlbnR9
KSB8IGhlYWQgLTEgPj4gJHtvdXR9CiAgICBmcmVlID4+ICR7b3V0fQogICAgZWNobyA+PiAke291
dH0KZG9uZQo=
--=-=-=
Content-Type: text/x-python; charset=utf-8
Content-Disposition: attachment; filename=plot.py
Content-Transfer-Encoding: quoted-printable

#!/usr/bin/env python3

from datetime import datetime, timedelta
from pathlib import Path
import re

import matplotlib
from matplotlib import pyplot
from matplotlib.dates import DateFormatter, HourLocator, MinuteLocator
from matplotlib.ticker import EngFormatter


MONITOR_RE =3D re.compile('\n'.join((
    '(?P<time>.+)',
    r' *(?P<seconds>\d+) +(?P<vsz>\d+) +(?P<rss>\d+) +(?P<args>.+)',
    ' *(?P<memheader>.+)',
    'Mem: *(?P<memvalues>.+)',
    'Swap: *(?P<swapvalues>.+)',
    ''
)), flags=3Dre.MULTILINE)


def list_snapshots(monitor_log):
    snapshots =3D []

    for match in MONITOR_RE.finditer(monitor_log):
        md =3D match.groupdict()

        memkeys =3D md['memheader'].split()
        memvalues =3D md['memvalues'].split()
        swapvalues =3D md['swapvalues'].split()

        snapshot =3D {
            'time': datetime.strptime(md['time'], '%Y-%m-%d-%H:%M:%S'),
            'uptime': int(md['seconds']),
            'vsz': int(md['vsz'])*1024,
            'rss': int(md['rss'])*1024,
            'process': md['args'],
            'mem': {memkeys[i]: int(val)*1024 for i, val in enumerate(memva=
lues)},
            'swap': {memkeys[i]: int(val)*1024 for i, val in enumerate(swap=
values)}
        }

        snapshots.append(snapshot)

    return snapshots


LOADDEFS_RE =3D re.compile(
    r'--eval \(setq generated-autoload-file'
    r' \(expand-file-name \(unmsys--file-name "([^"]+)"\)\)\)'
    r' -f batch-update-autoloads'
)

SEMANTIC_RE =3D re.compile(
    r'-l semantic/(?:wisent|bovine)/grammar -f (?:wisent|bovine)-batch-make=
-parser'
    r' -o (.+) .+\.[wb]y'
)

ELCELN_RE =3D re.compile(
    r'\.\./src/(?:bootstrap-)?emacs -batch --no-site-file --no-site-lisp'
    r' --eval \(setq load-prefer-newer t\) -l comp'
    r'(?: -f byte-compile-refresh-preloaded)?'
    r' -f batch-byte-native-compile-for-bootstrap'
    r' (.+\.el)'
)

SHORTENED_NAMES =3D {
    LOADDEFS_RE: 'GEN',
    SEMANTIC_RE: 'GEN',
    ELCELN_RE: 'ELC+ELN'
}

QUAIL_TIT_RE =3D re.compile(
    r'-l titdic-cnv -f batch-titdic-convert'
    r' -dir \./\.\./lisp/leim/quail CXTERM-DIC/(.+)\.tit'
)

QUAIL_MISC_RE =3D re.compile(
    r'-l titdic-cnv -f batch-miscdic-convert'
    r' -dir \./\.\./lisp/leim/quail MISC-DIC/(.+\.(html|map|cin|cns|b5))'
)

QUAIL_JA_RE =3D re.compile(
    r'-l ja-dic-cnv -f batch-skkdic-convert'
)

PARENT_RE =3D re.compile(
    r'$^'                       # Adjust to match parent process.
)

TRANSFORMED_NAMES =3D {
    QUAIL_TIT_RE: lambda m: f'GEN ../lisp/leim/quail/{m.group(1)}.el',
    QUAIL_MISC_RE: lambda m: f'GEN from {m.group(1)}',
    QUAIL_JA_RE: lambda m: f'GEN ../lisp/leim/ja-dic/ja-dic.el',
    PARENT_RE: lambda _: '(noise floor)'
}

def shorten(process):
    for r, name in SHORTENED_NAMES.items():
        match =3D r.search(process)
        if match is not None:
            return f'{name} {match.group(1)}'

    for r, transform in TRANSFORMED_NAMES.items():
        match =3D r.search(process)
        if match is not None:
            return transform(match)

    if len(process) > 40:
        return f'{process[:20]}=E2=80=A6{process[-20:]}'
    return process


def list_processes(snapshots):
    t0 =3D snapshots[0]['time']
    current_process =3D snapshots[0]['process']
    current_process_start =3D t0

    processes =3D []

    for s in snapshots[1:]:
        if s['process'] =3D=3D current_process:
            continue

        s_start =3D s['time']
        processes.append((
            current_process, current_process_start, s_start-current_process=
_start
        ))

        current_process =3D s['process']
        current_process_start =3D s_start

    processes.append((
        current_process,
        current_process_start,
        snapshots[-1]['time']-current_process_start
    ))

    return processes


snapshots =3D list_snapshots(Path('monitor.log').read_text())

xs =3D tuple(s['time'] for s in snapshots)
vsz =3D tuple(s['vsz'] for s in snapshots)
rss =3D tuple(s['rss'] for s in snapshots)
memavail =3D tuple(s['mem']['available'] for s in snapshots)
swapused =3D tuple(s['swap']['used'] for s in snapshots)

matplotlib.use('TkAgg')
fig, axes =3D pyplot.subplots(figsize=3D(128, 9.6))
axes.plot(xs, vsz, label=3D'VSZ (process)')
axes.plot(xs, rss, label=3D'RSS (process)')
axes.plot(xs, memavail, label=3D'available memory (system)', linewidth=3D0.=
5)
axes.plot(xs, swapused, label=3D'used swap (system)')

axes.set_xlim(snapshots[0]['time'], snapshots[-1]['time'])
axes.xaxis.set_major_formatter(DateFormatter('%H:%M'))
axes.xaxis.set_major_locator(HourLocator())
axes.xaxis.set_minor_locator(MinuteLocator(tuple(5*i for i in range(1, 12))=
))
axes.xaxis.set_label_text('Hours')
axes.set_ylim(0)
axes.yaxis.set_major_formatter(EngFormatter(unit=3D'B'))
axes.legend()

for p, start, duration in list_processes(snapshots):
    if duration < timedelta(minutes=3D2):
        continue
    pyplot.text(start, 1e9, shorten(p), rotation=3D45)
    pyplot.plot(
        (start, start+duration), (1e9, 1e9),
        marker=3D'|', linewidth=3D0.5, linestyle=3D'--',
        color=3D'black', alpha=3D0.8
    )

pyplot.savefig('monitor.pdf')
pyplot.show()

--=-=-=--