From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Re: Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs) Date: Fri, 01 Jul 2022 19:12:05 +0800 Message-ID: <877d4xjddm.fsf@localhost> References: <83bkuzznws.fsf@gnu.org> <877d5mqmkh.fsf@localhost> <83y1y2utnd.fsf@gnu.org> <87r13up587.fsf@localhost> <83o7yyur0l.fsf@gnu.org> <87leu2p3nu.fsf@localhost> <83leu2uewn.fsf@gnu.org> <87r13qv701.fsf@localhost> <83bkuursya.fsf@gnu.org> <87h74l9jk8.fsf@localhost> <83bkutqb3z.fsf@gnu.org> <9778F176-E724-4E61-B0FB-327BCDD316C0@acm.org> <87sfo4epeo.fsf@localhost> <87bkurrc5e.fsf@localhost> <87v8shk1c5.fsf@localhost> <83wncxe4pr.fsf@gnu.org> <87k08xjmlm.fsf@localhost> <83r135dsbn.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="34236"; mail-complaints-to="usenet@ciao.gmane.io" Cc: monnier@iro.umontreal.ca, mattiase@acm.org, theophilusx@gmail.com, rms@gnu.org, acm@muc.de, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Jul 01 13:20:51 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o7Ehf-0008nX-8v for ged-emacs-devel@m.gmane-mx.org; Fri, 01 Jul 2022 13:20:51 +0200 Original-Received: from localhost ([::1]:51822 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o7Ehe-0007df-53 for ged-emacs-devel@m.gmane-mx.org; Fri, 01 Jul 2022 07:20:50 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:53150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o7EYH-0004Rc-Df for emacs-devel@gnu.org; Fri, 01 Jul 2022 07:11:10 -0400 Original-Received: from mail-oi1-x235.google.com ([2607:f8b0:4864:20::235]:43639) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o7EYE-0007bL-Is; Fri, 01 Jul 2022 07:11:08 -0400 Original-Received: by mail-oi1-x235.google.com with SMTP id q11so3038136oih.10; Fri, 01 Jul 2022 04:11:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=noMrei37U+4+WCcHHo2xxTx3JfxQthMxyLOP9+n3PzA=; b=LSp4euVzPNAXa6bQinl+FY6/DLtEjIB3TRoAsknchVywlzprc+FbQxhzfGX8IUFKle PcZdumMEbbXG0+/P1DkU82jw/6j/DVjzj8NJsyGEC99GMlswh58hWbA/SnQvtw40h2R0 6d8UmALDZsmCh0BnJpbs3+PPRKlTTSlDW1LD1t7bnvrIMHL+lhIMe5waAUaL44EpbyWc frmYyZXfdHm8pwbj08jnX9d56daqV+ywVH2GBmnPrBtQy0l25R67+JdDY/cMX18b3O8+ 7AQemMfyGT7eqj36smbWTFyi/CN26WDPGqTIKiQaaGcxvNqYJsnWLhd3tiPO358ohlhf rJWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=noMrei37U+4+WCcHHo2xxTx3JfxQthMxyLOP9+n3PzA=; b=ulbryR8u0wK9V6nmGhpjePBqCkDUtggvtnOAyPiRWDQ6ZJw53mToSK0Zq0ctPWdgyX dFThwLEF9kTCfWYYqCPLcuJG2bZ/EeXFxnT4WMvZ03tt0wGBaNNRpN2QD83bGhFZeOiw geEuasNojKcNTztwllbERFTV0pBIEQRq1yAHeo7zRdUhU0Q2YTMx2jTmnxChYsNo6IWK kkSXLit5L2yD7V2tL9ldShcZ3cHdoKKxWR3nBxNpZ++VjcWjp37NAsJ30TktKuA2Wc7H 2oHBmqMsLewMCthvgnpIKv5ZFI0gHib9pR5DVqc/DEamZIqzK7W2lD0HkGeYXU7e8jDu 7XWA== X-Gm-Message-State: AJIora/51YJ9mnhQWWkkudl2iWw0ko+h6dNlLWel1Xp1w461g7t7mzmP 9tL0nVFvzd8ancynExmLKU0vg02CdRmBZQ== X-Google-Smtp-Source: AGRyM1s8tBKvIvpDjDtoEHn3+WMmThxOXNTyoihCWyx0WdUQSDxk04UGbAiF8QrfYUfXG2xy0A8VwQ== X-Received: by 2002:a05:6808:1718:b0:333:5794:5572 with SMTP id bc24-20020a056808171800b0033357945572mr8288986oib.115.1656673859741; Fri, 01 Jul 2022 04:10:59 -0700 (PDT) Original-Received: from localhost ([207.126.88.10]) by smtp.gmail.com with ESMTPSA id g14-20020a056870340e00b00101afc7b263sm14252591oah.9.2022.07.01.04.10.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Jul 2022 04:10:58 -0700 (PDT) In-Reply-To: <83r135dsbn.fsf@gnu.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::235; envelope-from=yantar92@gmail.com; helo=mail-oi1-x235.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:291775 Archived-At: Eli Zaretskii writes: >> > Please don't forget that GC doesn't only collects unused Lisp objects, >> > it also does other useful memory-management related tasks. It >> > compacts buffer text and strings, and it frees unused slots in various >> > caches (font cache, image cache, etc.). You can find in the archives >> > discussions where innocently-looking code could cause Emacs run out of >> > memory because it used too many fonts without flushing the font cache >> > (any program that works on the list of fonts returned by the likes of >> > x-list-fonts is in danger of bumping into that). >> >> Then, if we decide to implement the macro I am suggesting, such macro >> should not affect memory allocation of such sensitive objects: font >> cache, image cache, etc. > > Why not? The same could happen with those cached objects. And doing > these jobs does take CPU time. I do not have enough knowledge on the topic to argue on this. What I wanted to say is that we can make some objects to be not affected by the suggested macro if there are reasons to believe that it could be too dangerous. >> 1. In addition to directly bumping the TOTAL counter of newly allocated >> memory, we can introduce a new LOCAL counter holding recent >> allocations within current sexp. >> 2. Every time we return from a sexp/self-quoting object into assignment, >> if we are inside the proposed macro and also assigning value to one >> of the pre-defined symbols, increase the upper LOCAL counter in the >> parent sexp. Otherwise, do not change the upper LOCAL counter. >> 3. Perform GC according to TOTAL-LOCAL threshold value. >> 4. When exiting the macro, set LOCAL to 0, unless inside another such >> macro. > > Is this aligned with what the implementation of the Lisp interpreter > actually does? I'm not sure we know, where we allocate memory for > Lisp data, whether we are going to bind some variable to the produced > data. Thus "count recent allocations within the current sexp" sounds > like non-trivial to implement. I am not 100% sure, but AFAIK lisp interpreter always knows the current lisp nesting level. The extra C-level variable I suggest may be an array of max-lisp-eval-depth size. Then, at every given sexp nesting level, if we know the current level, we automatically gain access to the appropriate extra in-sexp memory allocation info. It is also trivial to ensure that all the deeper nesting levels in the array are set to 0 (after appropriate update of the value corresponding to the current nesting) > And what do you mean by "pre-defined symbols"? what makes a symbol > "pre-defined"? I was referring to (with-no-gc '(pre-defined-symbol1 symbol2 ...) ...) I am not sure if my description above is 100% accurate, but the general idea is: 1. Introduce LOCAL variable holding total memory used to allocate objects within current lisp nesting. This variable can be represented by an array of max-lisp-eval-depth size. 2. GC is triggered based on TOTAL-LOCAL instead of currently used TOTAL. 3. LOCAL[i]+=LOCAL[i+1]; LOCAL[i+1]=0 should be done after anything potentially involving recursive sexp call, except symbol slot assignments to non-"pre-defined" symbols. 4. If we are assigning symbol value slot and the symbol value is not "pre-defined" in the macro, LOCAL is not incremented in 3, which will make all the memory allocated inside that sexp to be counted towards next GC threshold. The above effectively allows GCs only upon returning from symbol assignments (in let/setq/setf/etc). But I believe that it should not be a big deal. > I also am not sure we have the "parent sexp" in hand, but I'll let > experts on the Lisp interpreter to comment on that. Agree. I am nowhere expert. Mostly tried to look into setq implementation I come up with something. Hopefully people more familiar with the interpreter chime in. Best, Ihor