From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Larger GC thresholds for non-interactive Emacs Date: Tue, 21 Jun 2022 20:01:43 -0400 Message-ID: References: <83y1y2utnd.fsf@gnu.org> <87r13up587.fsf@localhost> <83o7yyur0l.fsf@gnu.org> <87leu2p3nu.fsf@localhost> <83leu2uewn.fsf@gnu.org> <87r13qv701.fsf@localhost> <83bkuursya.fsf@gnu.org> <87h74l9jk8.fsf@localhost> <83bkutqb3z.fsf@gnu.org> <9778F176-E724-4E61-B0FB-327BCDD316C0@acm.org> <87sfo4epeo.fsf@localhost> <87bkurrc5e.fsf@localhost> <87bkur72b7.fsf@gnus.org> <874k0j40e7.fsf@gnus.org> <871qvm16he.fsf@gnus.org> <83a6aanm5j.fsf@gnu.org> <87o7yozzj0.fsf@gnus.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000009c8c4f05e1fe0b35" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36827"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , Stefan Monnier , Ihor Radchenko , =?UTF-8?Q?Mattias_Engdeg=C3=A5rd?= , Tim Cross , rms@gnu.org, Alan Mackenzie , emacs-devel To: Lars Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Jun 22 02:03:07 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o3npr-0009PR-Ar for ged-emacs-devel@m.gmane-mx.org; Wed, 22 Jun 2022 02:03:07 +0200 Original-Received: from localhost ([::1]:45378 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o3npn-0006OM-Uq for ged-emacs-devel@m.gmane-mx.org; Tue, 21 Jun 2022 20:03:03 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:52828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o3nom-0005V1-UF for emacs-devel@gnu.org; Tue, 21 Jun 2022 20:02:00 -0400 Original-Received: from mail-oi1-x22c.google.com ([2607:f8b0:4864:20::22c]:45716) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o3nok-0002fj-6z; Tue, 21 Jun 2022 20:01:59 -0400 Original-Received: by mail-oi1-x22c.google.com with SMTP id u9so19104674oiv.12; Tue, 21 Jun 2022 17:01:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YLYnkaQr3mTu85oMpQNOa2SYLs3MSfNrvOsbeIN+nF8=; b=q3a/Hl4hDHxFndQtxXiR/nJj06tryV1kfTns+iC2QqoOWhLE3FlRC08CQ0FGz79Ahv j2E+J2BHro3cvRPVg0Kvv5op2mQeZW6oUPvaCXpqMcG6vep0pEXQXUZSBnUwc3rcbXvS 19FDrpx37bJNMp7TcDhewlO9jnnIaCddeusAfieF2sq4G0V7CmWtbAU7tdc+KcoqxZ3I N8CuKiLWBWhuCktTf/5A1KtJvWKUqwsgheB9i37ox9A9zHWNiwlyNjLByWVdMiWC6nBj GFMYf7oUa8ZARcqdRjXclP/u6U7RLmbogJKxyuNNMucTvLE/RbJwcINWyv5jKGVkP7AR zvKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YLYnkaQr3mTu85oMpQNOa2SYLs3MSfNrvOsbeIN+nF8=; b=K5fRyzhLW2eJxvnnUTFZvyOHuysMv0uOaFRoU3frRHMKSiEErnZ2HklkdjYmMaeGRl 9zUTPvJRjFfhzpPBdbyZeawrMS++s3+4J5aO+dRwV6wMKYoRwFu2iTRL7OE6MLOT9SKW 48uv+voDFZ/YXi7muyHnckyywTzUUsPwYNHkil8wOwnp5aQhUptZsE7Hnyu+421Gdj/+ 6d42jlIlj/+NltZErUYnaU+F7pnpLBwCp3lk+2xz10jp559EK64bbdfcH4C3MQHAQaQu 7l5o2jgMe8hSRYjty8U3RzlsH8AzEPHLQ7ghOc5GMLF1w5rZttfjxOQpX+aS3iCP3RQV eFKg== X-Gm-Message-State: AJIora9kdp4ouzhvFzyO4MiXW0xpJ/98UnWU27rOBhLLHhDfvtRVW5RP Zv1+/oh5kDyQ6OuyA2AqjxUwrUWCq92ClTabUxc= X-Google-Smtp-Source: AGRyM1vL6GM1cJjMKfi49rqp4zE0HCDyb0hUAW4hpR1a4wcPNHNpD+djs6LEsjMSIASC71TjZNd3Bsn+i/0cnVni+0U= X-Received: by 2002:a05:6808:144f:b0:32f:56f5:7754 with SMTP id x15-20020a056808144f00b0032f56f57754mr396740oiv.162.1655856116303; Tue, 21 Jun 2022 17:01:56 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::22c; envelope-from=owinebar@gmail.com; helo=mail-oi1-x22c.google.com X-Spam_score_int: -9 X-Spam_score: -1.0 X-Spam_bar: - X-Spam_report: (-1.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, THIS_AD=1.098, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:291502 Archived-At: --0000000000009c8c4f05e1fe0b35 Content-Type: text/plain; charset="UTF-8" On Mon, Jun 20, 2022 at 10:01 PM Lynn Winebarger wrote: > On Sun, Jun 19, 2022, 7:05 AM Lars Ingebrigtsen wrote: > >> Eli Zaretskii writes: >> >> > AFAIK, that's not really true. We call 'free' and glibc does release >> > to the OS when it can. And don't forget that Emacs calls 'malloc' a >> > lot not just for conses and other Lisp data. >> >> Wasn't there a huge discussion about this a couple months ago and the >> conclusion was that glibc rarely releases memory back to the OS? That's >> why we added `malloc-trim'. >> > > Was there? I see:. > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71 > but it's a little more than a couple of months ago. > I'm pretty curious because if I accumulate a large buffer of trace output > (running memory up to 100s of MB), killing the buffer doesn't seem to > impact gc time substantially. > I would think using mmap-based allocation would make release of memory to > the system more likely. But mmap is only used for 64+KiB size blocks, and > the trim threshold is 128KiB blocks (shouldn't it be lower than the > allocation size?) > I'm also interested in the improvement in performance reported in that > thread from using the jemalloc implementation. Has that been investigated > further? > > I tried jemalloc per Ihor's description, using LD_PRELOAD to have it replace the system malloc without having to change or even recompile any emacs code. My test was to start a shell in a buffer, and then cat about 32MB of csv data to the buffer. It seemed like a reasonable approximation of generating large buffers of tracing output for this purpose. I started emacs 28.1 from a build directory with MALLOC_CONF=background_thread:true,retain:false ./src/emacs As expected, the emacs process slowly added about 50MB of memory to RSS. After the cat had completed, I killed the buffer and ran (garbage-collect). The RSS and VSZ of the process dropped about 40MB. Not complete reclamation, but I expect that it would solve the problem of a permanent bump in gc pause time. For some reason, the configure script chooses not to use mmap for malloc. jemalloc is designed to use mmap as much as possible so it isn't limited to freeing only the uppermost regions. I can't tell what (if any) option allows me to overrule the configure scripts decision to use sbrk instead of mmap. That might help make the mallopt knobs more effective. Are there any formal memory-allocation/reclamation benchmarks that I should use instead of this ad hoc task? Lynn --0000000000009c8c4f05e1fe0b35 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, Jun 20, 2022 at 10:01 PM Lynn Win= ebarger <owinebar@gmail.com>= ; wrote:
On Sun, Jun 19, 2022, 7:05 AM Lars Ingebri= gtsen <larsi@gnus.or= g> wrote:
Eli Zaretskii <eliz@gnu.org> writes:

> AFAIK, that's not really true.=C2=A0 We call 'free' and gl= ibc does release
> to the OS when it can.=C2=A0 And don't forget that Emacs calls = 9;malloc' a
> lot not just for conses and other Lisp data.

Wasn't there a huge discussion about this a couple months ago and the conclusion was that glibc rarely releases memory back to the OS?=C2=A0 That= 's
why we added `malloc-trim'.

but it's a little more than a couple of months ago.
I'm pretty curious because if I accumulate a large buffer= of trace output (running memory up to 100s of MB), killing the buffer does= n't seem to impact gc time substantially.=C2=A0=C2=A0
I would think using mmap-based allocation would make release of memor= y to the system more likely.=C2=A0 But mmap is only used for 64+KiB size bl= ocks, and the trim threshold is 128KiB blocks (shouldn't it be lower th= an the allocation size?)
I'm also interested in = the improvement in performance reported in that thread from using the jemal= loc implementation.=C2=A0 Has that been investigated further?

I tried jemalloc per Ihor's= description, using LD_PRELOAD to have it replace the system malloc without= =C2=A0
having to change or even recompile any emacs code.

My test was to start a shell in a buffer, and then cat ab= out 32MB of csv data to the buffer.=C2=A0 It seemed=C2=A0
like a = reasonable approximation of generating large buffers of tracing output for = this purpose.
I started emacs 28.1 from a build directory with
MALLOC_CONF=3Dbackground_thread:true,retain:false ./src/emacs
=

As expected, the emacs process slowly added about 50MB = of memory to RSS.=C2=A0 After the cat had completed,=C2=A0
I kill= ed the buffer and ran (garbage-collect).
The RSS and VSZ of the p= rocess dropped about 40MB.
Not complete reclamation, but I expect= that it would solve the problem of a permanent bump in gc pause time.
For some reason, the configure script chooses not to use mmap for mal= loc.=C2=A0 jemalloc is designed to use mmap
as much as possible s= o it isn't limited to freeing only the uppermost regions.
I c= an't tell what (if any) option allows me to overrule the configure scri= pts decision to use sbrk instead of mmap.
That might help make th= e mallopt knobs more effective.

Are there any form= al memory-allocation/reclamation benchmarks that I should use instead of th= is ad hoc task?

Lynn=C2=A0

--0000000000009c8c4f05e1fe0b35--