From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#49656: more data on #43389 Date: Wed, 21 Jul 2021 15:21:40 +0300 Message-ID: <83y29z502z.fsf@gnu.org> References: <83o8ax5hxp.fsf@gnu.org> <20210720.213837.660237047276474473.enometh@meer.net> <837dhk6hby.fsf@gnu.org> <20210721.071623.1810463348128672347.enometh@meer.net> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11532"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 49656@debbugs.gnu.org To: Madhu Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Jul 21 14:22:11 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1m6BEo-0002lH-Nf for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 21 Jul 2021 14:22:10 +0200 Original-Received: from localhost ([::1]:44082 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6BEn-0000lR-PQ for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 21 Jul 2021 08:22:09 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:46486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6BEh-0000jh-C2 for bug-gnu-emacs@gnu.org; Wed, 21 Jul 2021 08:22:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:53067) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m6BEh-0004WR-49 for bug-gnu-emacs@gnu.org; Wed, 21 Jul 2021 08:22:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m6BEg-0004t0-IR for bug-gnu-emacs@gnu.org; Wed, 21 Jul 2021 08:22:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 21 Jul 2021 12:22:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49656 X-GNU-PR-Package: emacs Original-Received: via spool by 49656-submit@debbugs.gnu.org id=B49656.162687011018755 (code B ref 49656); Wed, 21 Jul 2021 12:22:02 +0000 Original-Received: (at 49656) by debbugs.gnu.org; 21 Jul 2021 12:21:50 +0000 Original-Received: from localhost ([127.0.0.1]:36380 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m6BEU-0004sO-5r for submit@debbugs.gnu.org; Wed, 21 Jul 2021 08:21:50 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:48624) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m6BES-0004sA-5n for 49656@debbugs.gnu.org; Wed, 21 Jul 2021 08:21:48 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:46014) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6BEM-0004R0-Dj; Wed, 21 Jul 2021 08:21:42 -0400 Original-Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2273 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6BEM-00087z-1q; Wed, 21 Jul 2021 08:21:42 -0400 In-Reply-To: <20210721.071623.1810463348128672347.enometh@meer.net> (message from Madhu on Wed, 21 Jul 2021 07:16:23 +0530 (IST)) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:210426 Archived-At: > Date: Wed, 21 Jul 2021 07:16:23 +0530 (IST) > Cc: 49656@debbugs.gnu.org > From: Madhu > > > It could be if, for example, you are using packages or your own code > > which plays with GC thresholds. E.g., I'm told that lsp-mode does > > that; are you per chance using it? > > No I haven't gotten around to trying that i'm sure none of my packages > mess with gc-threshold. I will double check that. Thanks, please do. It's important to know that. > > How much memory and swap do you have there? Can you enlarge the total > > VM size (by enlarging swap) so that you could run Emacs longer when it > > gets to such large sizes? > > The problem is not that emacs won't run. I am unable to use the > memory in the machine for other purposes and am obliged to kill emacs. If you enlarge the swap space, your system will still be workable even when Emacs has such a large footprint. That's the purpose of my suggestion. It is important to have a usable system when these situations happen because if we come up with some experiments to conduct in such cases, you need a usable system to be able to do that. > > Btw, 1.5GB is not too large for several days worth of work. > > I also have emacs running for several weeks when the problem doesn't > happen with RES never above say 500M for the same workloads and usage > patterns. Then one thing to try to figure out is what was the difference between these two classes of sessions -- what and how did you do differently in one that you didn't in the other? That could point us in the right direction. > > Was the value of gcs-done increasing with time, or did it stay put > > (which would be an evidence that Emacs doesn't run GC at all)? If > > GC was running, how frequently did it run? (You could answer the > > last question either by looking at the rate of growth in gcs-done, > > or by setting garbage-collection-messages non-nil, which will cause > > Emacs announce ever GC in the echo area.) > > There is nothing exceptional with GC. GC usually completes quickly > because it doesnt access much memory. I'd like quantitative measures, please: what is the rate of GC cycles when the memory footprint is measurd in GBs, and how much time does each GC cycle take? Also, does the memory footprint becomes smaller after a GC cycle, and by how much? > >> But doesn't the malloc info allocation and the gc reports indicate a > >> clear discrepancy? > > > > Not to me, it doesn't. It means the malloc arena holds to a lot of > > memory that it cannot release to the OS, but it is known that this can > > happen for certain patterns of memory allocation and deallocation. > > You can find explanations of why this happens on the net. > > But that is the point - I've been asking from the first time I posted > on this - WHAT is emacs allocating in these exceptional cases. It's impractical to try to answer these questions. If you put a breakpoint on 'malloc' and 'free' and then run Emacs, you will see that it calls these functions extremely frequently, with widely different block sizes. We don't have infrastructure that would record each allocation and deallocation with enough info to be able to analyze that, and even if we did, finding the callers which are responsible for the memory that's not returned to the OS is a very large and hard job. We tried that previously, with the help of the glibc developers using their debugging features -- it didn't really help us with the diagnostics. > I understand my usage patterns over the years of constant emacs use > and the "random" memory bloat in some sessions makes no sense and it > only suggests a memory leak in emacs code. It is impossible to look for memory leaks in Emacs without some clues about where those leaks could be. If you can provide the information I asked above, we might be able to come up with such clues. Alternatively, you could try running Emacs under Valgrind (see instructions in etc/DEBUG), although this will probably catch memory leaks only on the C level, not on the Lisp level. > >> Does it not indicate a leak which the GC is not aware of? > > > > No. Emacs allocates a lot of memory for purposes other than Lisp > > objects, and that memory is not visible to GC nor to memory-report. > > This is too vague. But that's all I can say, given the information you provided till now. > This 2GB active memory is not legitimate and the point is to figure > out where and why emacs is holding on to this memory. > > This is where I need assistance. This is me assisting you. I'm asking you to help us analyze this problem by providing more information. I don't think we will be able to make progress here without that additional info. > SO what could that be, which it is NOT allocating under identical > usage patterns? I don't know, sorry. > I'm sure others should be hitting the problem too We could, of course, wait for others to report similar problems, and then hope the information they provide would point us to the right direction. IME, though, this is not a very efficient method, because in many cases the reasons for the memory growth are not the same.