From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Russell Adams Newsgroups: gmane.emacs.bugs Subject: bug#43389: 28.0.50; Emacs memory leaks Date: Thu, 26 Nov 2020 16:42:19 +0100 Message-ID: <20201126154219.GA16802@maokai> References: <87r1r5428d.fsf@web.de> <87mu1sry72.fsf@mail.linkov.net> <875z8fc224.fsf@web.de> <20200915175418.GV20869@maokai> <838sda98jm.fsf@gnu.org> <20200915211209.GW20869@maokai> <83pn6l7ozj.fsf@gnu.org> <20200917204704.GA20217@maokai> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="7544"; mail-complaints-to="usenet@ciao.gmane.io" To: 43389@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Nov 26 16:53:23 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kiJaF-0001mL-8O for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 26 Nov 2020 16:53:23 +0100 Original-Received: from localhost ([::1]:40510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kiJaD-0005HY-Tb for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 26 Nov 2020 10:53:21 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:56210) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiJSA-0002L7-DD for bug-gnu-emacs@gnu.org; Thu, 26 Nov 2020 10:45:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:59219) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kiJSA-0000Nz-4j for bug-gnu-emacs@gnu.org; Thu, 26 Nov 2020 10:45:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kiJSA-0004W0-18 for bug-gnu-emacs@gnu.org; Thu, 26 Nov 2020 10:45:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Russell Adams Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 26 Nov 2020 15:45:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43389 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.160640546117272 (code B ref -1); Thu, 26 Nov 2020 15:45:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 26 Nov 2020 15:44:21 +0000 Original-Received: from localhost ([127.0.0.1]:42529 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kiJRL-0004UN-Ob for submit@debbugs.gnu.org; Thu, 26 Nov 2020 10:44:21 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:33420) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kiJRI-0004UD-Ef for submit@debbugs.gnu.org; Thu, 26 Nov 2020 10:44:11 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:55786) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiJRG-00028X-Pf for bug-gnu-emacs@gnu.org; Thu, 26 Nov 2020 10:44:07 -0500 Original-Received: from se15.route25.eu ([2a00:f10:402:781:1c00:45ff:fe00:24f8]:50160) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiJR9-0008Cl-VG for bug-gnu-emacs@gnu.org; Thu, 26 Nov 2020 10:44:05 -0500 Mail-Followup-To: bug-gnu-emacs@gnu.org Content-Disposition: inline In-Reply-To: <20200917204704.GA20217@maokai> X-Originating-IP: 185.87.184.57 X-SpamExperts-Domain: out.pcextreme.nl X-SpamExperts-Username: 185.87.184.57 Authentication-Results: route25.eu; auth=pass smtp.auth=185.87.184.57@out.pcextreme.nl X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: SB/global_tokens (0.00791682965805) X-Recommended-Action: accept X-Filter-ID: Mvzo4OR0dZXEDF/gcnlw0fJVsTFCFA1YlcTpjJcy6PSpSDasLI4SayDByyq9LIhVtcWfpvqLS60e BNTMwtOw90TNWdUk1Ol2OGx3IfrIJKywOmJyM1qr8uRnWBrbSAGD0VsY0HzEkBzUTjuObUvkgtQ5 3pMmTXeVbblO3cSIiVD+lPsJl3NQibG9+awNzLmjJODXbtOodkPED+RkHjVGH9gpfPe2lDQBFQc5 pWKjU8U1TnR+f0U9cu064nZcMieLAqdkk6hppCDsqsBwLjHur1XxB26GtR7aQdGQ3RxVousDFpkW mgyT/YeLVvI3HPYSaHwJ2eESrUp0Iw/gSJ3HqMuR600P9eQ3vodupN36MrkzGQZS068e3EYTgNAB jZkD8TzPF4eG61o+cxFZrUIXowfXJosMX5ZQSlYSVlCDu2na8Lc2iK0mXNgodhRh0E6+UMVFei1G 2PrSIU9EwioS70yFBF2moUWeHNDS4qZDIDB7XoQijaYHvr9sPSt0aWxBU8NAmhVjxOcflCqTdpCd nz3dKxLhoxcmaInYbR5vlqF9UZSlUPITrUlXo0KweE5K01CkZQcRJUiIO9z1A2bbodvpo2UgV9PP 0GtOSQCD3Wp6EipRzMVZ5LqwTx7Vvn9SWQRihqWRcCaa7yMGNcHiKiC9NjNYY5WoFXFyoMFsG38I aa05P3P4eYW86qsHunbFcho0P7VyKpXK1A14c1ki+AeYUOp7A73HI6oJg7w/Vod4U0DAmRXz28Ch LUng6DAB7o8DebwK9n0BDYMYTfyprxMqABCuK7DVpOJr7QXPUVFWWG/qurFASmykSe7dic5Ihrj9 GCF/KjAOTCzc+K6CedMMKG5rbgpB8 X-Report-Abuse-To: spam@semaster01.route25.eu Received-SPF: pass client-ip=2a00:f10:402:781:1c00:45ff:fe00:24f8; envelope-from=rladams@adamsinfoserv.com; helo=se15.route25.eu X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:194348 Archived-At: On Thu, Sep 17, 2020 at 10:47:04PM +0200, Russell Adams wrote: > From Emacs memory-usage package: > > Garbage collection stats: > ((conses 16 1912248 251798) (symbols 48 54872 19) (strings 32 327552 81803) (string-bytes 1 12344346) (vectors 16 158994) (vector-slots 8 2973919 339416) (floats 8 992 4604) (intervals 56 182607 7492) (buffers 1000 195)) > > => 29.2MB (+ 3.84MB dead) in conses > 2.51MB (+ 0.89kB dead) in symbols > 10.00MB (+ 2.50MB dead) in strings > 11.8MB in string-bytes > 2.43MB in vectors > 22.7MB (+ 2.59MB dead) in vector-slots > 7.75kB (+ 36.0kB dead) in floats > 9.75MB (+ 410kB dead) in intervals > 190kB in buffers > > Total in lisp objects: 97.9MB (live 88.5MB, dead 9.36MB) I had the memory leak occur again and this time I had the glibc-malloc-trace-utils loaded and running from the start. So my emacs grew to 8GB in RAM, and what was curious is if it was a background task (not window focused on an emacsclient), then the memory stayed the same. When I had the window focused, I could watch the memory constantly increasing in htop a few megs at a time. Garbage collection stats: ((conses 16 1749077 1176908) (symbols 48 47530 38) (strings 32 307123 144020) (string-bytes 1 10062511) (vectors 16 113172) (vector-slots 8 2105205 486800) (floats 8 709 1719) (intervals 56 174593 44804) (buffers 1000 71)) => 26.7MB (+ 18.0MB dead) in conses 2.18MB (+ 1.78kB dead) in symbols 9.37MB (+ 4.40MB dead) in strings 9.60MB in string-bytes 1.73MB in vectors 16.1MB (+ 3.71MB dead) in vector-slots 5.54kB (+ 13.4kB dead) in floats 9.32MB (+ 2.39MB dead) in intervals 69.3kB in buffers Total in lisp objects: 103MB (live 75.0MB, dead 28.5MB) Buffer ralloc memory usage: 47 buffers 3.36MB total ( 232kB in gaps) Size Gap Name 926626 1504 AIS.org 690050 1933 Personal.org 553850 2000 Abuffer.org 490398 3851 *Packages* 215653 2000 KB.org 76686 1708 X230.org 59841 2123 Agenda.org 51375 51076 *sly-events for sbcl* 51060 1902 ASC.org 44596 2000 Contacts.org 36825 1792 *Messages* 23882 2309 *org-caldav-debug* 22867 2000 rgb.lisp 14678 746 *sly-mrepl for sbcl* 6640 1173 VirtualFCMap.lisp 4096 2000 *code-converting-work* 3409 16717 *http orgmode.org:443* 1946 104 *Org Agenda* 1528 2028 *http gaming.demosthenes.org*-491231 1524 2028 *http gaming.demosthenes.org*-15349 1518 2028 *http gaming.demosthenes.org* 1276 1368 *sly-inferior-lisp for sbcl* 1231 2026 *http gaming.demosthenes.org*-464306 1208 825 *Help* 679 1574 *Buffer Details* 641 1975 *Agenda Commands* 531 1494 *Calendar* 324 2008 *http melpa.org:443* 278 3775 *helm M-x* 185 1838 *org caldav sync result* 144 2000 *scratch* 57 21434 *helm find files* 44 5610 *icalendar-work* 30 2000 *sly-fontify* 21 2000 *log-edit-files* 20 0 *pdf-info-query--escape* 18 4077 *helm mini* 12 8630 *code-conversion-work* 5 4065 *Echo Area 1* 0 2033 *Minibuf-1* 0 20 *Minibuf-0* 0 20 *server* 0 4060 *Echo Area 0* 0 61547 *sly-1* 0 20 *sly-dds-1-1* 0 20 *changes to ~/ASC/Software/Snaps/* 0 20 *vc* I started emacs with: MTRACE_CTL_FILE=mtraceEMACS.mtr LD_PRELOAD=~/software/glibc-malloc-trace-utils/libmtrace.so ~/.local/bin/emacs --daemon >> ~/.config/emacs/emacs.log 2>&1 This created some huge files. By the time I reached 8GB in RAM, the mtr file for the main process (I think) was 53 GB. I also have little mtrace files littered everywhere in different project directories. -rw-r--r-- 1 adamsrl adamsrl 53G Nov 26 13:23 mtraceEMACS.mtr.15236 -rw-r--r-- 1 adamsrl adamsrl 4.2G Nov 26 13:36 my.wl -rw-r--r-- 1 adamsrl adamsrl 1.3G Nov 26 13:50 mtraceEMACS.mtr.15236.allocs -rw-r--r-- 1 adamsrl adamsrl 32K Nov 26 13:55 mtraceEMACS.mtr.15236.binnedallocs.log -rw-r--r-- 1 adamsrl adamsrl 6.0G Nov 26 15:12 vmrssout -rw-r--r-- 1 adamsrl adamsrl 6.0G Nov 26 15:12 vmout -rw-r--r-- 1 adamsrl adamsrl 8.6G Nov 26 15:12 idealrssout I converted the mtraceEMACS.mtr.15236 to my.wl using trace2wl. The trace_run command did this output: % ~/software/glibc-malloc-trace-utils/trace_run ./my.wl vmout vmrssout idealrssout 11,757,635,230,744 cycles 4,532,472,554 usec wall time 5,966,752,470 usec across 3 threads 8,461,721,600 bytes Max RSS (218,308,608 -> 8,680,030,208) Starting VmRSS 218308608 (bytes) Starting VmSize 219549696 (bytes) Starting MaxRSS 218308608 (bytes) Ending VmRSS 8680030208 (bytes) Ending VmSize 8903626752 (bytes) Ending MaxRSS 8680030208 (bytes) 8,131,008 Kb Max Ideal RSS sizeof ticks_t is 8 Avg malloc time: 145 in 422,186,832 calls Avg calloc time: 12,538 in 1,164,584 calls Avg realloc time: 566 in 3,294,165 calls Avg free time: 110 in 449,397,629 calls Total call time: 127,318,389,383 cycles These files are impossible to share around, is there anything I can run to extract anything else useful from them? % ~/software/glibc-malloc-trace-utils/trace_statistics mtraceEMACS.mtr.15236 Min allocation size: 0 Max allocation size: 1603869 Mean allocation size: 128 I did follow the instructions for downsampling, but I haven't a clue what to do in Octave. Is it worth posting those files? I have the impression this is more about how often more RAM was requested, and not the source of the call? I should mention I'm present in #emacs and happy to discuss there. ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3