From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Larger GC thresholds for non-interactive Emacs (was: Org mode and Emacs) Date: Fri, 17 Jun 2022 15:18:21 +0800 Message-ID: <87bkurrc5e.fsf@localhost> References: <875yl9e7zm.fsf@gmail.com> <83czfh12kp.fsf@gnu.org> <87pmjhghu2.fsf@localhost> <835yl910gp.fsf@gnu.org> <87wndndbhq.fsf@gmail.com> <83bkuzznws.fsf@gnu.org> <877d5mqmkh.fsf@localhost> <83y1y2utnd.fsf@gnu.org> <87r13up587.fsf@localhost> <83o7yyur0l.fsf@gnu.org> <87leu2p3nu.fsf@localhost> <83leu2uewn.fsf@gnu.org> <87r13qv701.fsf@localhost> <83bkuursya.fsf@gnu.org> <87h74l9jk8.fsf@localhost> <83bkutqb3z.fsf@gnu.org> <9778F176-E724-4E61-B0FB-327BCDD316C0@acm.org> <87sfo4epeo.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="20928"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Mattias =?utf-8?Q?Engdeg=C3=A5rd?= , Eli Zaretskii , Tim Cross , rms@gnu.org, Alan Mackenzie , emacs-devel To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Jun 17 09:19:52 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o26Gm-0005HG-5O for ged-emacs-devel@m.gmane-mx.org; Fri, 17 Jun 2022 09:19:52 +0200 Original-Received: from localhost ([::1]:38848 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o26Gk-0006Pl-JU for ged-emacs-devel@m.gmane-mx.org; Fri, 17 Jun 2022 03:19:50 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:33030) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o26EK-0005H9-KG for emacs-devel@gnu.org; Fri, 17 Jun 2022 03:17:20 -0400 Original-Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]:44858) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o26EH-0001n6-P7; Fri, 17 Jun 2022 03:17:20 -0400 Original-Received: by mail-pl1-x629.google.com with SMTP id h1so3164334plf.11; Fri, 17 Jun 2022 00:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=4Asmy7wLenuQ6ST0O0COGwdMQVtaCBnbeCaFMNmkcH0=; b=jeOl8hJAf+UCyHZlfehZMdXLc7zJlerBLXLL4bcHaK1P+CLC+u5SxKQyNkTDC0NvIz DoEwH7AQuNLo1BHzOvyfMTjrF4TwSesfnVCk4cYeKx8H4U20Py9E0eeJYiAVu44IRmkP TbhntcAZ1b1RfmrRT+2tNwZf44vIlDxf3cotMICmJwo6YYGoTZt18G04t0hjj8OmgBOc ZjXEa3Kz0SSjRP8La9JP9k0446pPII1LiFT2gE5ObAzZoIt5rHch81Ut2TwQEYfEuzl7 s+P671vlejC/UdDqit98YVMuFhyxWP8Ib7gwHCEnaPIj9+wjdd7bY1wZ8Ee04pg8Rs3q hedg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=4Asmy7wLenuQ6ST0O0COGwdMQVtaCBnbeCaFMNmkcH0=; b=dF6Ni+Cnofj1A0Ezex1sn1ALH0vljBl91F5YPu/ygoxIsTx+eICl8T+oFBMUV7USpm yFxGgkEA3KFD/qF9Wu5iBsADIKOvmr12UA1v9+Dliu6kU06VY9IITz7fy7+pUDIXiHzD UktbYkyabn29w93p/gv7qtXjMu0WF9oTV8NLIoxsv7UNDIl+WpAcTmqd1RP3qg7H4hSv cOMUQpWdJk7ckIwmeXPB6FFNeBVQXsdpWFskDtBkQiR8HXE68r2LMpfEWqi+YKTwEDM+ Jp2gu+IQYrD+uK6xnERuOx2iijpFSBiuLtXx9ldvEJldMzloGVHljIptGBhWlESVWXFg dqeQ== X-Gm-Message-State: AJIora9J05XHgBeOEQSy89uvrrk0dATbkBBPOHzHtCXSip7frYu6k/2l cyLGCIuQ5TEem82qqwR5cdc= X-Google-Smtp-Source: AGRyM1twgPRBYEVYCPwOsTN6N+qaaOsRZWn9SweNK7JqFHNP5NlCA3b+luHZE/q/5bpm+F9lXbKZow== X-Received: by 2002:a17:90b:380b:b0:1e6:67f6:c5b4 with SMTP id mq11-20020a17090b380b00b001e667f6c5b4mr20157092pjb.48.1655450235479; Fri, 17 Jun 2022 00:17:15 -0700 (PDT) Original-Received: from localhost ([66.154.105.4]) by smtp.gmail.com with ESMTPSA id h187-20020a6283c4000000b0051c01aa7d31sm2944612pfe.46.2022.06.17.00.17.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jun 2022 00:17:15 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::629; envelope-from=yantar92@gmail.com; helo=mail-pl1-x629.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:291279 Archived-At: Stefan Monnier writes: >>> thus Eli's choice is very good, and we really should do something about that GC of ours. >> >> I am wondering if there is a "universal" value suitable for one-off Emacs >> batch invocations. > > I doubt that's the case, but of course we should try and use values that > work "overall best" on average. Maybe we should revisit the current > knobs and their default values. > > Currently the two important knobs we have are: > > gc-cons-threshold > gc-cons-percentage > > Maybe it's time for some benchmarks with various values for these knobs > to see if the values we're using now are "good enough" or might need to > be changed. Note that I did not initially ask to change the GC defaults for normal Emacs session. What I am trying to suggest is: 1. Have some conservative GC settings in interactive Emacs session (as we do now; I do _not_ suggest changing it) 2. Increase GC thresholds in non-interactive --batch sessions, which tend to not run for a long time. I am a bit afraid to touch the topic of changing the interactive GC defaults. People tend to have contradicting opinions on this. I'd like to ask one thing from the very beginning here: Please spin a separate thread if you have objections on actual GC defaults applicable to normal Emacs sessions. > We could also try and be bit more discerning. E.g. currently when the > program is in a phase where it builds a lot of new data-structures, we > run the GC everytime some amount of new memory has been allocated, but > presumably almost none of those objects have died yet (we're still > building the overall datastructure) so the GC will be a pure waste of > time. OTOH in other phases the code allocates objects which are used > only temporarily, in which case it's beneficial to run the GC somewhat > frequently to avoid growing our heap unnecessarily (and suffering > fragmentation as a result). AFAIK, there is no way to determine such stage automatically. It's usually up to the program. For example, org-element-parse-buffer (function that creates buffer AST in Org) does the following: (let (... (gc-cons-threshold #x40000000)) ... (org-element--parse-elements ...)) A possible useful thing Emacs could help with would be a macro dedicated to let-binds like the above. Something like: (with-reduced-gc ;; Do staff) with-reduced-gc could take care about determining the inner specifics on what alternative gc-cons-threshold value should be used (possibly depending on the system memory information). > It's hard to know beforehand whether a GC will be useful or not, tho. > But maybe we can find good heuristics. E.g. have something like > a `gc-cons-percentage` which depends on how much garbage we collected in > the last GC: if a GC doesn't collect any garbage (or very little of it), > it's a sign that we're in a phase where running the GC is not very > useful so we should wait a bit more until the next GC, whereas if the GC > collected a fair bit of garbage, it's a sign that we're in a phase where > running the GC is beneficial and we can run it a bit more often. FYI, I have played with this approach making use of https://gitlab.com/koral/gcmh In practice, relying on recent GC history is often misleading. GC is normally not very frequent - tens of seconds to minutes. And the practical Emacs memory usage patterns are often irregular: long periods of "idleness"/low intensity usage vs. short bursts of high memory allocations when some "heavy" commands are called. The adjustment you propose will suffer from the following: 1. There is long "idle" period and GC gets less frequent 2. User runs "heavy" command, GC frequency not yet adjusted 3. Command allocates a lots of memory - GC is finally triggered and has to go through a lot of garbage; It often happens _after_ the command finishes. 4. GC re-adjusts the frequency, but Emacs is again "idle" with no "heavy" commands being used. Best, Ihor