From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Carlos O'Donell Newsgroups: gmane.emacs.bugs Subject: bug#43389: 28.0.50; Emacs memory leaks using hard disk all time Date: Wed, 25 Nov 2020 13:57:34 -0500 Organization: Red Hat Message-ID: <53fa9720-e709-81cc-dece-369f6136d048@redhat.com> References: <86y2j2brg2.fsf@protected.rcdrun.com> <83blfxth7c.fsf@gnu.org> <83y2j0qb2v.fsf@gnu.org> <831rgppg3w.fsf@gnu.org> <83zh3czbvz.fsf@gnu.org> <83blfovzxz.fsf@gnu.org> <87o8jnu5f2.fsf@mail.trevorbentley.com> <83o8jmu49z.fsf@gnu.org> <522e3cc0-c563-3308-7264-1b09cd5e264b@redhat.com> <83blfls494.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36369"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 Cc: fweimer@redhat.com, 43389@debbugs.gnu.org, bugs@gnu.support, dj@redhat.com, michael_heerdegen@web.de, trevor@trevorbentley.com To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Nov 25 19:58:23 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1khzzi-0009LA-T2 for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 25 Nov 2020 19:58:22 +0100 Original-Received: from localhost ([::1]:53594 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1khzzh-00072w-SU for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 25 Nov 2020 13:58:21 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:42574) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1khzzQ-00070h-KF for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 13:58:05 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:53673) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1khzzN-0005bu-On for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 13:58:04 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1khzzN-0006BK-N4 for bug-gnu-emacs@gnu.org; Wed, 25 Nov 2020 13:58:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Carlos O'Donell Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 25 Nov 2020 18:58:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43389 X-GNU-PR-Package: emacs Original-Received: via spool by 43389-submit@debbugs.gnu.org id=B43389.160633066423737 (code B ref 43389); Wed, 25 Nov 2020 18:58:01 +0000 Original-Received: (at 43389) by debbugs.gnu.org; 25 Nov 2020 18:57:44 +0000 Original-Received: from localhost ([127.0.0.1]:36986 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1khzz5-0006An-KN for submit@debbugs.gnu.org; Wed, 25 Nov 2020 13:57:44 -0500 Original-Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:45025) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1khzz3-0006Af-Ei for 43389@debbugs.gnu.org; Wed, 25 Nov 2020 13:57:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606330661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yiwqv0Wq3VOPx65/MwCXAOWYgMoiJD5S3QuXyarLc8E=; b=MBZSBl/f4KUXODpqOMLBY6/qfIRYjasEY5gXDEWRkzBSbAvaTWEB7HZAuVQcjvNt3z+GOw jfDTGa7JIv+eaIR1CefUL3X3+fnY5b/VZTs2n0ZYa74qYCdN1OU/ZGao42shu+l7tnD0MO Snn+ZbLeXwW7qS+iMtJ81no9qWGuP1U= Original-Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-529-RWbtKlkNPPeAY3b9-L7X7g-1; Wed, 25 Nov 2020 13:57:37 -0500 X-MC-Unique: RWbtKlkNPPeAY3b9-L7X7g-1 Original-Received: by mail-qk1-f199.google.com with SMTP id 198so3179310qkj.7 for <43389@debbugs.gnu.org>; Wed, 25 Nov 2020 10:57:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:organization :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=Yiwqv0Wq3VOPx65/MwCXAOWYgMoiJD5S3QuXyarLc8E=; b=mEJLwKOPt6HFayRmaqU7WGhQ4pgyaVR2N4XmLKDVL6pCOGXkI3B7LP0/wBjWQi2Fz1 rvYGVX4cr3bTR3w+C2QgJKMbr5rBXyMlQSrAWMtoVRBI9URqJTD5Z5C8XlvYHqrsxAhs 8DzIq4ToCggnmYtwESdyYna9/H+2GO1e8hLA9yuV9knrZr+OMNmwYIM2daPfwpLxxsOk u4grk0VDkgn+3RJXj+mL/6qZKx2+HrCItakL+fuQ1qtX76wsdAtAjzRONYqGS0koLTGP gzC5tTDhMoKRplPfQObBzbdrIxANg5sdg+2Vrl0n9fa2ECaVhd1PY2tI3wCZmlnBqaee Zgyw== X-Gm-Message-State: AOAM530F38SSHKeuAYmi8MzFqrDVqsbdFiDOU7gcOPxLL8B6ZlwL/oM9 Ms+vjRCONBLG0JuCocgZ19U2v0wvc7BCcBM5T1X95N9MqwYWYlOfp+2ONXww+EaZRXCHdUpfJS9 AgiJA7fjEHNf27GY= X-Received: by 2002:ac8:4e87:: with SMTP id 7mr278510qtp.310.1606330656639; Wed, 25 Nov 2020 10:57:36 -0800 (PST) X-Google-Smtp-Source: ABdhPJzgJPwqjAJF3vWyhouo7nNxGU8lfvFLi0lQXuA/cHNky67mP02q7t0aD7FHBI1i+CdtswqXFQ== X-Received: by 2002:ac8:4e87:: with SMTP id 7mr278493qtp.310.1606330656373; Wed, 25 Nov 2020 10:57:36 -0800 (PST) Original-Received: from [192.168.1.16] (198-84-214-74.cpe.teksavvy.com. [198.84.214.74]) by smtp.gmail.com with ESMTPSA id c6sm175704qkg.54.2020.11.25.10.57.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 25 Nov 2020 10:57:35 -0800 (PST) In-Reply-To: <83blfls494.fsf@gnu.org> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=carlos@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:194221 Archived-At: On 11/25/20 1:03 PM, Eli Zaretskii wrote: >> Cc: bugs@gnu.support, fweimer@redhat.com, 43389@debbugs.gnu.org, >> dj@redhat.com, michael_heerdegen@web.de >> From: Carlos O'Donell >> Date: Wed, 25 Nov 2020 12:45:04 -0500 >> >> On 11/24/20 11:07 AM, Eli Zaretskii wrote: >>> Look at the large chunks in the tail of this. Together, they do >>> account for ~2GB. >>> >>> Carlos, are these chunks in use (i.e. allocated and not freed), or are >>> they the free chunks that are available for allocation, but not >>> released to the OS? If the former, then it sounds like this session >>> does have around 2GB of allocated heap data, so either there's some >>> allocated memory we don't account for, or there is indeed a memory >>> leak in Emacs. If these are the free chunks, then the way glibc >>> manages free'd memory is indeed an issue. >> >> These chunks are all free and mapped for use by the algorithm to satisfy >> a request by the application. > > So we have more than 1.5GB free memory available for allocation, is > that right? There are 3 malloc_info traces in the log. 1. Lines 47-219. Day 1: 1100MiB of RSS. 2. Lines 386-556. Day 4: 2.3GiB of RSS. 3. Lines 744-792. Day 5: 4.2GiB of RSS. Lines are numbered for the log starting at 1. > To make sure there are no misunderstandings, I'm talking about this > part of the log: Your analysis is for trace #2, lines 386-556. My analysis was for trace #3, lines 744-792. > > > [...] > > > > > > > > > > > > > > > > > > > If I sum up the "total=" parts of these large numbers, I get 1.6GB. > Is this free memory, given back to glibc for future allocations from > this arena, and if so, are those 1.6GB part of the 4.2GB total? In trace #2 we have these final statistics: 549 550 551 552 553 554 555 556 This shows ~1.7GiB of unused free chunks. Keep in mind glibc malloc is a heap-based allocator so if you have FIFO usage pattern you won't see the kernel heap decrease until you free the most recently allocated chunk. In trace #3 we *do* see that application demand consumes all these free chunks again, so something is using them in the application. There are none left reported in the malloc_info statistics (could also be chunk corruption). During trace #2 the only way to free some of the ~1.7GiB in-use by the algorithm is to call malloc_trim() to free back unused pages (requires free/unsorted chunk walk and mmumap() calls to the kernel to reduce RSS accounting). Calling malloc_trim is expensive, particularly if you're just going to use the chunks again, as appears to be happening the next day. In trace #3, for which we are at 4.2GiB of RSS usage, we see the following: 742 ;; malloc-info 743 (malloc-info) 744 745 746 747 748 749 a. Arena 0 (kernel heap) shows 0KiB of unused fast bins, 112KiB of other in 1 bin (probably top-chunk). 750 751 752 753 b. Arena 0 (kernel heap) shows 4.2GiB "current" which means that the sbrk-extended kernel heap is in use up to 4.2GiB. WARNING: We count "foreign" uses of sbrk as brk space, so looking for sbrk or brk by a foreign source is useful. 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 c. Arena 1 has 42MiB of free'd chunks for use. 784 785 786 787 d. We have: - 912KiB of fast bins. - 42MiB of regular bins. - 200MiB of mmap'd large chunks. 788 789 790 e. Total allocated space is 4.2GiB. 791 792 Something is using the kernel heap chunks, or calling sbrk/brk directly (since foreign brks are counted by our statistics). >> This shows the application is USING memory on the main system heap. >> >> It might not be "leaked" memory since the application might be using it. >> >> You want visibility into what is USING that memory. >> >> With glibc-malloc-trace-utils you can try to do that with: >> >> LD_PRELOAD=libmtrace.so \ >> MTRACE_CTL_FILE=/home/user/app.mtr \ >> MTRACE_CTL_BACKTRACE=1 \ >> ./app >> >> This will use libgcc's unwinder to get a copy of the malloc caller >> address and then we'll have to decode that based on a /proc/self/maps. >> >> Next steps: >> - Get a glibc-malloc-trace-utils trace of the application ratcheting. >> - Get a copy of /proc/$PID/maps for the application (shorter version of smaps). >> >> Then we might be able to correlate where all the kernel heap data went? > > Thanks for the instructions. Would people please try that and report > the results? > -- Cheers, Carlos.