From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#43389: 28.0.50; Emacs memory leaks using hard disk all time Date: Wed, 25 Nov 2020 19:47:16 +0200 Message-ID: <83d001s50b.fsf@gnu.org> References: <86y2j2brg2.fsf@protected.rcdrun.com> <83blfxth7c.fsf@gnu.org> <83y2j0qb2v.fsf@gnu.org> <831rgppg3w.fsf@gnu.org> <83zh3czbvz.fsf@gnu.org> <83blfovzxz.fsf@gnu.org> <87o8jnu5f2.fsf@mail.trevorbentley.com> <83o8jmu49z.fsf@gnu.org> <87ft4ytw2c.fsf@mail.trevorbentley.com> <83d002tuoo.fsf@gnu.org> <87d001u46f.fsf@mail.trevorbentley.com> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="37208"; mail-complaints-to="usenet@ciao.gmane.io" Cc: fweimer@redhat.com, 43389@debbugs.gnu.org, bugs@gnu.support, dj@redhat.com, carlos@redhat.com, michael_heerdegen@web.de To: Trevor Bentley Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Nov 25 18:48:33 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1khyu9-0009Xu-3U for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 25 Nov 2020 18:48:33 +0100 Original-Received: from localhost ([::1]:56128 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1khyu7-0004o0-Lp for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 25 Nov 2020 12:48:31 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:52360) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1khytd-0004nm-R8 for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 12:48:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:53552) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1khytd-0004Om-K0 for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 12:48:01 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1khytd-0004XK-IW for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 12:48:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 25 Nov 2020 17:48:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43389 X-GNU-PR-Package: emacs Original-Received: via spool by 43389-submit@debbugs.gnu.org id=B43389.160632644717392 (code B ref 43389); Wed, 25 Nov 2020 17:48:01 +0000 Original-Received: (at 43389) by debbugs.gnu.org; 25 Nov 2020 17:47:27 +0000 Original-Received: from localhost ([127.0.0.1]:36864 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1khyt5-0004WR-15 for submit@debbugs.gnu.org; Wed, 25 Nov 2020 12:47:27 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:33882) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1khyt3-0004WF-W9 for 43389@debbugs.gnu.org; Wed, 25 Nov 2020 12:47:26 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:49075) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1khysw-0004Ix-MV; Wed, 25 Nov 2020 12:47:19 -0500 Original-Received: from [176.228.60.248] (port=3766 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1khysu-0002fS-7R; Wed, 25 Nov 2020 12:47:16 -0500 In-Reply-To: <87d001u46f.fsf@mail.trevorbentley.com> (message from Trevor Bentley on Wed, 25 Nov 2020 11:22:16 +0100) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:194212 Archived-At: > From: Trevor Bentley > Cc: bugs@gnu.support, fweimer@redhat.com, 43389@debbugs.gnu.org, > dj@redhat.com, michael_heerdegen@web.de, carlos@redhat.com > Date: Wed, 25 Nov 2020 11:22:16 +0100 > > >> - The leaking stops for a while after (garbage-collect). It > >> was leaking 1MB per second for this last log, and stopped > >> growing after the garbage collection. > > > > Now, what happens in that session once per second (in an > > otherwise idle Emacs, I presume?) to cause such memory > > consumption? Some timers? If you run with a breakpoint in > > malloc that just shows the backtrace and continues, do you see > > what could consume 1MB every second? > > Not an idle emacs at all, in this case. I have seen the memory > growth in an idle emacs, but the only one I can reproduce it on is > the emacs-slack one, which is connected to a corporate Slack > account. Tons of short messages streaming in over the network and > being displayed in rotating buffers, with images mixed in. It's a > big 'ol "web 2.0" API... it can easily pass 1MB/s of bloated JSON > messages through. This is one _very active_ emacs. Then I don't think we will be able to understand what consumes memory at such high rate without some debugging. Have you considered using breakpoints and collecting backtraces, as I suggested earlier? The hard problem is to understand which memory is allocated and not freed "soon enough", but for such a high rate of memory consumption perhaps just knowing which code request so much memory would be an important clue. > The original strace logs and valgrind output I posted before > showed a random assortment of calls from gnutls, imagemagick, and > lisp strings, with lisp strings dominating the malloc calls > (enlarge_buffer_text, mostly). Enlarging buffer text generally causes malloc to call mmap (as opposed to brk/sbrk), so this cannot cause the situation where a lot of unused memory that is not returned to the OS. And we already saw that just by summing up the buffer text memory we never get even close to the VM size of the process. > > What do you mean by "reaping dead references" here? > > > >> It could be that there really were 4.3GB of dead references. > > > > Not sure I understand what are you trying to establish here. > > GC is running through a list of active allocations and freeing the > ones with no remaining references, right? Presumably, if a lot of > active malloc() allocations are no longer refernced, and > (garbage-collect) calls free() on a bunch of blocks. We only call free on "unfragmented" Lisp data, e.g. if some block of Lisp strings was freed in its entirety. If some Lisp objects in a block are still alive, we don't free the block, we just mark the freed Lisp objects as being free and available for reuse. So the result of GC shows only tells you how much of the memory was freed but NOT returned to glibc, it doesn't show how much was actually free'd. > I'm wondering how to figure out how much memory a call to > (garbage-collect) has actually freed. Possibly a sort of "dry run" > where it performs the GC algorithm, but doesn't release any memory. "Freed" in what sense? returned to glibc? > > There's only one garbage-collect, it is called for _any_ GC. > > > > What do you mean by "during normal use" in this sentence: > > > > I certainly don't notice 5-10 minute long pauses during normal > > use, though "gcs-done" is incrementing. > > > > How is what you did here, where GC took several minutes, > > different from "normal usage"? > > In this log, I am explicitly executing "(garbage-collect)", and it > takes 10 minutes, during which the UI is unresponsive and > sometimes even turns grey when the window stops redrawing. > > By "normal use", I mean that I use this emacs instance on-and-off > all day long. I would notice if it were freezing for minutes at a > time, and it definitely is not. > > As far as I understand, garbage collection is supposed to happen > automatically during idle. I would certainly notice if it locked > up the whole instance for 10 minutes from an idle GC. I think > this means the automatic garbage collection is either not > happening, or running on a different thread, or being interrupted, > or simply works differently. I have no idea, hence asking you :) That is very strange. There's only one function to perform GC, and it is called both from garbage-collect and from an internal function called when Emacs is idle or when it calls interpreter functions like 'eval' or 'funcall'. The only thing garbage-collect does that the internal function doesn't is generate the list that is the return value of garbage-collect, but that cannot possibly take minutes. I suggest to set garbage-collection-messages non-nil, then you should see when each GC, whether the one you invoke interactively or the automatic one, starts and ends. maybe the minutes you wait are not directly related to GC, but to something else that is triggered by GC?