From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#33275: 27.0.50; Image cache pruning Date: Tue, 06 Nov 2018 17:40:36 +0200 Message-ID: <83bm72ciyz.fsf@gnu.org> References: <83va5bcxxu.fsf@gnu.org> <83sh0fcx2r.fsf@gnu.org> <83o9b3cu4f.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org X-Trace: blaine.gmane.org 1541518873 1137 195.159.176.226 (6 Nov 2018 15:41:13 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 6 Nov 2018 15:41:13 +0000 (UTC) Cc: 33275@debbugs.gnu.org To: Lars Ingebrigtsen Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Nov 06 16:41:09 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK3TY-0000CO-Vz for geb-bug-gnu-emacs@m.gmane.org; Tue, 06 Nov 2018 16:41:09 +0100 Original-Received: from localhost ([::1]:41774 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gK3Vf-0008KW-DC for geb-bug-gnu-emacs@m.gmane.org; Tue, 06 Nov 2018 10:43:19 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:43932) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gK3TW-0005TN-QR for bug-gnu-emacs@gnu.org; Tue, 06 Nov 2018 10:41:11 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gK3TS-0002df-Oq for bug-gnu-emacs@gnu.org; Tue, 06 Nov 2018 10:41:06 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:33221) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gK3TS-0002db-LI for bug-gnu-emacs@gnu.org; Tue, 06 Nov 2018 10:41:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gK3TS-0003Wv-Hj for bug-gnu-emacs@gnu.org; Tue, 06 Nov 2018 10:41:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 06 Nov 2018 15:41:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 33275 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 33275-submit@debbugs.gnu.org id=B33275.154151885713545 (code B ref 33275); Tue, 06 Nov 2018 15:41:02 +0000 Original-Received: (at 33275) by debbugs.gnu.org; 6 Nov 2018 15:40:57 +0000 Original-Received: from localhost ([127.0.0.1]:37478 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK3TJ-0003WL-V3 for submit@debbugs.gnu.org; Tue, 06 Nov 2018 10:40:57 -0500 Original-Received: from eggs.gnu.org ([208.118.235.92]:34173) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK3TE-0003Vt-QU for 33275@debbugs.gnu.org; Tue, 06 Nov 2018 10:40:52 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gK3T4-0002L5-Vn for 33275@debbugs.gnu.org; Tue, 06 Nov 2018 10:40:43 -0500 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:55586) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gK3T4-0002KU-R1; Tue, 06 Nov 2018 10:40:38 -0500 Original-Received: from [176.228.60.248] (port=1731 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1gK3T4-0006dl-A5; Tue, 06 Nov 2018 10:40:38 -0500 In-reply-to: (message from Lars Ingebrigtsen on Mon, 05 Nov 2018 18:48:33 +0100) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:152103 Archived-At: > From: Lars Ingebrigtsen > Cc: 33275@debbugs.gnu.org > Date: Mon, 05 Nov 2018 18:48:33 +0100 > > At present, the image cache is purely based on timestamps, if I recall > correctly? If we wish to fix this unexpected behaviour, we could also > look at the size of the image cache and perhaps try to limit it, or if > we wanted to be really fancy, we could somehow check whether there's any > references to the images anywhere (beyond the image cache itself). > > I think the latter would mean adding some... fun... to the gc system? > > Or does Emacs have weak hashes? In which case the image cache could be > a weak hash and the problem would resolve itself? > [...] > Emacs does have weak hashes... Hm... > > Anyway, there's code that tries to keep the cache small: > > /* If the number of cached images has grown unusually large, > decrease the cache eviction delay (Bug#6230). */ > delay = XFIXNUM (Vimage_cache_eviction_delay); > if (nimages > 40) > delay = 1600 * delay / nimages / nimages; > > But it works solely on the number of images in the cache, and not the > size of the images. So it's a heuristic that could be tweaked pretty > easily, I think? After thinking about this and looking at the code, I don't think I understand your proposal(s). Here are the issues that I saw with what I think you proposed: . The hash table used by the image cache is not a Lisp hash-table, so weakness doesn't apply. We cache pointers to C structs, so using a Lisp hash-tables would need "some work". . The code you show above is called from redisplay, so it will not be executed as long as a Lisp program (such as the one you show at the beginning of this discussion) runs, and you will still have the OOM killer get you before that code get a chance to clear the cache. . Running such code from GC could be tricky, because freeing images needs to remove references of those images from the glyph matrices, and that cannot be safely done from arbitrary places. . I don't think you can ensure GC will run during your program anyway, because most of the memory allocated by your program is not used to cons Lisp objects, so maybe_gc called by the Lisp interpreter will most probably decide there's no need for GC and do nothing. Therefore, I don't think that avoiding to cache the images would prevent the OOM situation in your case. I think if we want to prevent the OOM killer from killing Emacs due to many cached images, we should inspect inside cache_image the current VM size of the process (like system_process_attributes does), compare it to the memory limits, and display a warning and/or signal an error when we get dangerously close to the limit. If we want to avoid similar situations with other objects, we could do these tests inside xmalloc and mmap_alloc/mmap_realloc. (We used to do something similar in the past, but that code needed a libc hook for memory allocation from the OS, something I understand we cannot rely on now.) IOW, I don't see how we could prevent being OOM-killed, except if we monitor the committed memory at the point where it is allocated, as otherwise the opportunistic memory allocation strategy can always trip us.