From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.bugs Subject: bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs) Date: Thu, 05 Dec 2019 14:48:03 +0800 Message-ID: <87zhg7jmjg.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> References: <87sgme1ww7.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <83o8x0rl6d.fsf@gnu.org> <87lfs2mzo8.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <87lfs19shb.fsf@mail.linkov.net> <87a78gn5k5.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <83sgm8ox3g.fsf@gnu.org> <87zhgflxmc.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <83lfrzq1s1.fsf@gnu.org> <87d0d7w3tt.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <83zhgaloc5.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="138115"; mail-complaints-to="usenet@blaine.gmane.org" Cc: 38345@debbugs.gnu.org, juri@linkov.net To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Thu Dec 05 07:51:16 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1ickyq-000ZoI-4h for geb-bug-gnu-emacs@m.gmane.org; Thu, 05 Dec 2019 07:51:16 +0100 Original-Received: from localhost ([::1]:50756 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ickyo-0001a4-US for geb-bug-gnu-emacs@m.gmane.org; Thu, 05 Dec 2019 01:51:14 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40343) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ickyi-0001YP-19 for bug-gnu-emacs@gnu.org; Thu, 05 Dec 2019 01:51:09 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ickyg-0000Ju-Hi for bug-gnu-emacs@gnu.org; Thu, 05 Dec 2019 01:51:07 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:36946) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ickyd-0000IR-LX for bug-gnu-emacs@gnu.org; Thu, 05 Dec 2019 01:51:04 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ickyd-0007UA-JE for bug-gnu-emacs@gnu.org; Thu, 05 Dec 2019 01:51:03 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Ihor Radchenko Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 05 Dec 2019 06:51:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38345 X-GNU-PR-Package: emacs Original-Received: via spool by 38345-submit@debbugs.gnu.org id=B38345.157552861428620 (code B ref 38345); Thu, 05 Dec 2019 06:51:03 +0000 Original-Received: (at 38345) by debbugs.gnu.org; 5 Dec 2019 06:50:14 +0000 Original-Received: from localhost ([127.0.0.1]:42919 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ickxq-0007RY-5i for submit@debbugs.gnu.org; Thu, 05 Dec 2019 01:50:14 -0500 Original-Received: from mail-yw1-f41.google.com ([209.85.161.41]:35569) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ickxo-0007R3-4v for 38345@debbugs.gnu.org; Thu, 05 Dec 2019 01:50:12 -0500 Original-Received: by mail-yw1-f41.google.com with SMTP id i190so831235ywc.2 for <38345@debbugs.gnu.org>; Wed, 04 Dec 2019 22:50:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=Y0f150BrR9AyaIeu+i0j0CJAMN8OBtMOPnT99/Yy5sg=; b=ZdlWt1am7gl4wmv+UCxY5l12Wp8GcnQw3zYyAjYz/7xWtW7XjdiB93ZLKtT8XnomC+ l1g9PXijV9ArmM+0P4FkIFEZuYNY2oB2NFJvFddR9Dw5tvf739y8jb+fOIVINkEU/+36 HMpMETOS9oqWd6lZL9KevmJfRbP9u7D93ksj26X4kcATqi+6V+5r17SUmeDgR+bIOArR ak5Q6EIWuTJkPpwnbZe2r0lFaD+XNo3uECaEl9IMDsUID3mIE4MlvN6kTSDa5lhWizQu /7jJrA1BFGJoIhEuw8xYUSn087qqz3yVFzCfXTgsnRBrzPs7SLLYZCDSU7tCfZGTkrD2 Txdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=Y0f150BrR9AyaIeu+i0j0CJAMN8OBtMOPnT99/Yy5sg=; b=AclYVo1MZNzMJNlQrSZ6eJ9VEwrNgBgHKtPzEIhgGk0jsNDTWot3+GYhjE2/gLSn9G ZP981uoZiYHJeCdXsL/fr4zQk/wZVKp+MvD1RBs2F4N63Q+NjbnaXWX9bXhdeYpmRuS2 E9s2KAEHWM61yFglCkSnnbnPUJpHKE5ENxEcbnCyb/Yp8EYaUDIjMoVZZmF/NoNNBdqY plcSGP+WanUSgZ8IuKSDCSC5EH9sTDSU3qn9CFxFmhSt7mHUWRKKZhtkj8YEdvSCTWp6 Xz+kC0oyvHJGjc7+yvGKymF9k3JjgCxKv5RgtNcYI8dOl4JnJw6Yn3t3XLt0grzFshTj qroA== X-Gm-Message-State: APjAAAXLUDOPio1tWRGtQlJCEKAJQOY+4w1uA1VcXY4OYl5S3QkyzBGT 5u7jPjfzx1JHsaOlS2wIqO8= X-Google-Smtp-Source: APXvYqxmT1jxNKyz1bHYzvh2m3u3fwljPxRJKSiAswVQpPJCTqLXbcYdM0GG4oZjX7Q59enhbzdUnQ== X-Received: by 2002:a81:65c4:: with SMTP id z187mr4433487ywb.191.1575528606447; Wed, 04 Dec 2019 22:50:06 -0800 (PST) Original-Received: from localhost ([5.226.137.4]) by smtp.gmail.com with ESMTPSA id l68sm4167423ywf.89.2019.12.04.22.50.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2019 22:50:05 -0800 (PST) In-Reply-To: <83zhgaloc5.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:172871 Archived-At: > Feel free to ask questions about the details you don't understand. I tried to figure out why the memory consumption is increasing with less frequent cache clearing. As I understand, there may be two general reasons of this: 1. Increasing of memory used by the image cache data structure 2. Imperfect implementation of xfree or img->type->free, which are called by free_image xfree and img->type->free seem to rely on system libraries and are unlikely to have such a noticeable memory leaks. For the image cache data structure, I can only see one place where it may request extra memory allocation. It is when the number of images in the cache exceeds the size of c->images array (= c->size, which is 50 by default). I observed memory consumption increase even when the frequency of cache clearing in my test was <50, which makes it unclear for me where the extra memory consumption is coming from. > I don't think replacing the system malloc on GNU/Linux systems (let > alone on others) is an idea we'd like to pursue. You may have more > luck playing with mallopt calls in init_alloc_once_for_pdumper > (assuming your build defines DOUG_LEA_MALLOC). Do I understand correctly that you refer to emacs compiled with alternative Doug Lea's implementation of malloc? Actually, I tried to find a way to compile emacs with alternative variants of malloc, but I did not find how to do it. P.S. I am running emacs with jemalloc for a few days and the overall impression is that emacs became a lot more responsive. Previously, I had a slow overtime degradation of delay between commands as emacs process ran for a long time, up to the point when I can type a whole sentence until emacs finally displays it on screen. Now, I do not see so much degradation. Best, Ihor Eli Zaretskii writes: >> From: Ihor Radchenko >> Cc: juri@linkov.net, 38345@debbugs.gnu.org >> Date: Mon, 02 Dec 2019 16:04:46 +0800 >> >> > The doc string assumes "normal" operation, it doesn't assume some Lisp >> > command runs for a long time, thus preventing redisplay. >> >> Sounds counter-intuitive. Elisp commands can run for a long time >> (pointing at org-agenda). > > Commands that run for a long time make Emacs unresponsive, and > therefore are a misfeature or a bug. > > That said, a command that runs a long time is unlikely to create many > images, destroying each one before creating the next one, because > what would be the point of that? > > IOW, yours is a scenario that is infrequent at best, IMO. > >> I tried to look at the code, but I have to admit that I don't understand >> the reason. Probably because of my poor C. > > Feel free to ask questions about the details you don't understand. > >> In any case, I tried different approach trying to reduce memory >> fragmentation. Instead of using the default malloc/free, I used jemalloc >> through LD_PRELOAD. For periodic cache clearing (every 2,10,50, and 100 >> opened/killed images), there was no noticeable difference in the memory >> consumption. However, when I tried to open a bunch of images all >> together, killed them, and cleared the cache (as in my initial bug >> report), the jemalloc instance of emacs showed much lower memory >> consumption (~240Mb vs 1200Mb). > > I don't think replacing the system malloc on GNU/Linux systems (let > alone on others) is an idea we'd like to pursue. You may have more > luck playing with mallopt calls in init_alloc_once_for_pdumper > (assuming your build defines DOUG_LEA_MALLOC).