From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Indentation and gc Date: Sun, 12 Mar 2023 16:10:38 +0200 Message-ID: <83356aukkh.fsf@gnu.org> References: <20230310110747.4hytasakomvdyf7i.ref@Ergus> <20230310110747.4hytasakomvdyf7i@Ergus> <87a60k657y.fsf@web.de> <838rg4zmg9.fsf@gnu.org> <87ttys4dge.fsf@web.de> <83sfebyepp.fsf@gnu.org> <87ttyru4zt.fsf@web.de> <83fsabyb41.fsf@gnu.org> <87mt4jtpqf.fsf@web.de> <83ilf7wi48.fsf@gnu.org> <878rg3wh2f.fsf@localhost> <87a60jtg0z.fsf@web.de> <877cvmumjq.fsf@localhost> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26955"; mail-complaints-to="usenet@ciao.gmane.io" Cc: arne_bab@web.de, spacibba@aol.com, emacs-devel@gnu.org To: Ihor Radchenko Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Mar 12 15:11:38 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pbMQE-0006oU-Mm for ged-emacs-devel@m.gmane-mx.org; Sun, 12 Mar 2023 15:11:38 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pbMPX-0004HO-2C; Sun, 12 Mar 2023 10:10:55 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pbMPU-0004HF-HE for emacs-devel@gnu.org; Sun, 12 Mar 2023 10:10:52 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pbMPT-0005fx-AX; Sun, 12 Mar 2023 10:10:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=kfKjeDmPNRbxD0So/Lnd59jTa+gVLGf9OXT0AgWMc1U=; b=ZRI7R/jKDxnH rmnxmXLiUd99Hi6iPInL0R5nSXDkmUnY5jR//208DebjaMtCciI54W2bwfc+hyvjemLg70OWWjvjv IBdgc78BLBseVClTktUjuQWwnoFVrOgrwCaXRIG2tZNDfPpAjITTuU0NYbSdkPdhr4TDnVzB0nN7h +81VhvRIfmRhdY1z8DLAua3Yt7S2/oVq0AaqVZrlVnPphIGEjJgIesLxcjgA10S07IFEL0IVk0D9R mR2duiTBfJ0yWsTnjxm7+Ka/+UwBVfgaOMEiset0kKMPpOrC9KxfJ/wuKN9NvlkHPughalCkeF5+F px2N266G3Z3yOeSve+fq5A==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pbMPS-0007p7-QO; Sun, 12 Mar 2023 10:10:51 -0400 In-Reply-To: <877cvmumjq.fsf@localhost> (message from Ihor Radchenko on Sun, 12 Mar 2023 13:27:53 +0000) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:304368 Archived-At: > From: Ihor Radchenko > Cc: Eli Zaretskii , spacibba@aol.com, emacs-devel@gnu.org > Date: Sun, 12 Mar 2023 13:27:53 +0000 > > > So there should likely be also a hard upper limit to ensure that the > > pauses are unnoticeable. > > Well. I do realize that there should be a limit, which is why I put it > as 100Mb. Yes, but what is that 100 MiB number based on? any measurements of the time it takes to run GC behind that number? or just a more-or-less arbitrary value that "seems right"? > Strictly speaking, GC pauses scale with heap size. If by "heap size" you mean the total size of heap-allocated memory in the Emacs process, then this is inaccurate. GC traverses only the Lisp objects, whereas Emacs also allocates memory from the heap for other purposes. It also allocates memory from the VM outside of the "normal" heap -- that's where the buffer text memory usually comes from, as well as any large enough chunk of memory Emacs needs. > Increasing GC threshold > will have two effects on the heap size: (1) thresholds lager than normal > heap size will dominate the GC time - Emacs will need to traverse all > the newly added data to be GCed; You seem to assume that GC traverses only the Lisp objects newly-allocated since the previous GC. This is incorrect: it traverses _all_ of the Lisp objects, both old and new. > (2) too large thresholds will cause heap fragmentation, also > increasing the GC times as the heap will expand. Not sure why do you think heap fragmentation increases monotonically with larger thresholds. Maybe you should explain what you consider "heap fragmentation" for the purposes of this discussion. > I think that (2) is the most important factor for real world scenarios Not sure why you think so. Maybe because I don't have a clear idea what kind of fragmentation you have in mind here. > Emacs' default gives some lower safe bound on the threshold - it is > `gc-cons-percentage', defaulting to 1% of the heap size. Actually, the default value of gc-cons-percentage is 0.1, i.e. 10%. And it's 10% of the sum total of all live Lisp objects plus the number of bytes allocated for Lisp objects since the last GC. Not 10% of the heap size. > However, being safe is unfortunately not enough - apparently 1% heap > size is too less in real world scenarios when 1% is routinely and > frequently allocated, triggering GC rescans too many times. How large is what you call "heap size" in your production session, may I ask? > AFAIU, routine throw-away memory allocation in Emacs is not directly > correlated with the memory usage - it rather depends on the usage > patterns and the packages being used. For example, it takes about 10 > complex helm searches for me to trigger my 250Mb threshold - 25Mb per > helm command. This calculation is only valid if each of these 10 commands conses approximately the same amount of Lisp data. If that is not so, you cannot really divide 250 MiB by 10 and claim that each command used up that much Lisp memory. That's because GC is _not_ triggered as soon as Emacs crosses the threshold, it is triggered when Emacs _checks_ how much was consed since last GC and discovers it consed more than the threshold. The trigger for testing is unrelated to crossing the threshold. > To get some idea about the impact of gc-cons-threshold on memory > fragmentation, I compared the output of `memory-limit' with 250Mb vs. > default 800kb threshold: > > 250Mb threshold - 689520 kb memory > 800kb threshold - 531548 kb memory > > The memory usage is clearly increased, but not catastrophically, despite > using rather large threshold. > > Of course, it is just init.el, which is loaded once. Correction: it is _your_ init.el. We need similar statistics from many users and many different usage patterns; only then we will be able to draw valid conclusions. > Memory fragmentation as a result of routine Emacs usage may cause > more significant memory usage increase. Actually, Emacs tries very hard to avoid fragmentation. That's why it compacts buffers, and that's why it can relocate buffer text and string data. > As long as gc-cons-threshold is significantly (~10x) lower than normal > Emacs heap size, I expect the GC frequency to scale down the same 100x > at very little cost. What is the "normal heap size" in your production sessions? And how did you measure it? > For me, Emacs memory usage typically settles around 1Gb, which is why I > chose 100Mb as an upper limit (1Gb/10). Once again, the threshold value is not necessarily directly derived from the total memory footprint of the Emacs process.