From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Paul Eggert Newsgroups: gmane.emacs.bugs Subject: bug#37006: 27.0.50; garbage collection not happening after 26de2d42 Date: Sat, 14 Sep 2019 00:51:18 -0700 Organization: UCLA Computer Science Department Message-ID: References: <5075406D-6DB8-4560-BB64-7198526FCF9F@acm.org> <83h86nu0pq.fsf@gnu.org> <86pnlbphus.fsf@phe.ftfl.ca> <83a7cft8qx.fsf@gnu.org> <868srysb9x.fsf@phe.ftfl.ca> <83wofis508.fsf@gnu.org> <2687613f-ba89-25cf-9517-5311106aef9a@cs.ucla.edu> <83ef1prly5.fsf@gnu.org> <0bc956d1-4cf5-a886-1703-49ee0aeb3d58@cs.ucla.edu> <83v9uzrasm.fsf@gnu.org> <18886155-d96b-ae07-1df2-1b1d58a8bbb2@cs.ucla.edu> <83o90qqzqw.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------AE78A0F38C0C63807462A40B" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="120518"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 Cc: jrm@ftfl.ca, mattiase@acm.org, 37006@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Sep 14 09:52:14 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1i92qs-000VBf-38 for geb-bug-gnu-emacs@m.gmane.org; Sat, 14 Sep 2019 09:52:14 +0200 Original-Received: from localhost ([::1]:48924 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i92qq-0005v2-MZ for geb-bug-gnu-emacs@m.gmane.org; Sat, 14 Sep 2019 03:52:12 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:58643) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1i92qi-0005us-EB for bug-gnu-emacs@gnu.org; Sat, 14 Sep 2019 03:52:06 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1i92qg-0007QX-FA for bug-gnu-emacs@gnu.org; Sat, 14 Sep 2019 03:52:04 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:36986) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1i92qg-0007QM-Ak for bug-gnu-emacs@gnu.org; Sat, 14 Sep 2019 03:52:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1i92qg-0007U4-7H for bug-gnu-emacs@gnu.org; Sat, 14 Sep 2019 03:52:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Paul Eggert Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 14 Sep 2019 07:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 37006 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 37006-submit@debbugs.gnu.org id=B37006.156844748928730 (code B ref 37006); Sat, 14 Sep 2019 07:52:02 +0000 Original-Received: (at 37006) by debbugs.gnu.org; 14 Sep 2019 07:51:29 +0000 Original-Received: from localhost ([127.0.0.1]:45807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i92q9-0007TJ-4Y for submit@debbugs.gnu.org; Sat, 14 Sep 2019 03:51:29 -0400 Original-Received: from zimbra.cs.ucla.edu ([131.179.128.68]:56718) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i92q6-0007T3-1p for 37006@debbugs.gnu.org; Sat, 14 Sep 2019 03:51:27 -0400 Original-Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 5A8301601EE; Sat, 14 Sep 2019 00:51:20 -0700 (PDT) Original-Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id P0-SyMoqbXSG; Sat, 14 Sep 2019 00:51:18 -0700 (PDT) Original-Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id DCE3C16021D; Sat, 14 Sep 2019 00:51:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Original-Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id KSZztKKhX2l6; Sat, 14 Sep 2019 00:51:18 -0700 (PDT) Original-Received: from [192.168.1.9] (cpe-23-242-74-103.socal.res.rr.com [23.242.74.103]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 9BB4F1601EE; Sat, 14 Sep 2019 00:51:18 -0700 (PDT) In-Reply-To: <83o90qqzqw.fsf@gnu.org> Content-Language: en-US X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:166427 Archived-At: This is a multi-part message in MIME format. --------------AE78A0F38C0C63807462A40B Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 8/15/19 7:17 AM, Eli Zaretskii wrote: > OK, but I still hope we will switch to EMACS_INT instead. I installed the attached patch into master, and it changes the relevant threshold variables to EMACS_INT, as part of a somewhat-larger cleanup. This should make a practical difference only on 32-bit Emacs without --with-wide-int. --------------AE78A0F38C0C63807462A40B Content-Type: text/x-patch; name="0001-Improve-gc-cons-percentage-calculation.patch" Content-Disposition: attachment; filename="0001-Improve-gc-cons-percentage-calculation.patch" Content-Transfer-Encoding: quoted-printable >From bac66302e92bdd3a353102d2076548e7e83d92e5 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Sat, 14 Sep 2019 00:32:01 -0700 Subject: [PATCH] Improve gc-cons-percentage calculation MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit The old calculation relied on a hodgpodge of partly updated GC stats to find a number to multiply gc-cons-percentage by. The new one counts data found by the previous GC, plus half of the data allocated since then; this is more systematic albeit still ad hoc. * src/alloc.c (consing_until_gc, gc_threshold, consing_threshold): Now EMACS_INT, not intmax_t. (HI_THRESHOLD): New macro. (tally_consing): New function. (make_interval, allocate_string, allocate_string_data) (make_float, free_cons, allocate_vectorlike, Fmake_symbol): Use it. (allow_garbage_collection, inhibit_garbage_collection) (consing_threshold, garbage_collect): Use HI_THRESHOLD rather than INTMAX_MAX. (consing_threshold): New arg SINCE_GC. All callers changed. (bump_consing_until_gc): Return new consing_until_gc, instead of nil. All callers changed. Don=E2=80=99t worry about overflow since we now saturate at HI_THRESHOLD. Guess that half of recently-allocated objects are still alive, instead of relying on the previous (even less-accurate) hodgepodge. (maybe_garbage_collect): New function. (garbage_collect): Work even if a finalizer disables or enables memory profiling. Do not use malloc_probe if GC reclaimed nothing. * src/lisp.h (maybe_gc): Call maybe_garbage_collect instead of garbage_collect. --- src/alloc.c | 127 ++++++++++++++++++++++++++++++---------------------- src/lisp.h | 5 ++- 2 files changed, 77 insertions(+), 55 deletions(-) diff --git a/src/alloc.c b/src/alloc.c index ca8311cc00..497f600551 100644 --- a/src/alloc.c +++ b/src/alloc.c @@ -224,7 +224,7 @@ struct emacs_globals globals; =20 /* maybe_gc collects garbage if this goes negative. */ =20 -intmax_t consing_until_gc; +EMACS_INT consing_until_gc; =20 #ifdef HAVE_PDUMPER /* Number of finalizers run: used to loop over GC until we stop @@ -238,9 +238,16 @@ bool gc_in_progress; =20 /* System byte and object counts reported by GC. */ =20 +/* Assume byte counts fit in uintptr_t and object counts fit into + intptr_t. */ typedef uintptr_t byte_ct; typedef intptr_t object_ct; =20 +/* Large-magnitude value for a threshold count, which fits in EMACS_INT. + Using only half the EMACS_INT range avoids overflow hassles. + There is no need to fit these counts into fixnums. */ +#define HI_THRESHOLD (EMACS_INT_MAX / 2) + /* Number of live and free conses etc. counted by the most-recent GC. *= / =20 static struct gcstat @@ -299,7 +306,7 @@ static intptr_t garbage_collection_inhibited; =20 /* The GC threshold in bytes, the last time it was calculated from gc-cons-threshold and gc-cons-percentage. */ -static intmax_t gc_threshold; +static EMACS_INT gc_threshold; =20 /* If nonzero, this is a warning delivered by malloc and not yet displayed. */ @@ -536,6 +543,15 @@ XFLOAT_INIT (Lisp_Object f, double n) XFLOAT (f)->u.data =3D n; } =20 +/* Account for allocation of NBYTES in the heap. This is a separate + function to avoid hassles with implementation-defined conversion + from unsigned to signed types. */ +static void +tally_consing (ptrdiff_t nbytes) +{ + consing_until_gc -=3D nbytes; +} + #ifdef DOUG_LEA_MALLOC static bool pointers_fit_in_lispobj_p (void) @@ -1372,7 +1388,7 @@ make_interval (void) =20 MALLOC_UNBLOCK_INPUT; =20 - consing_until_gc -=3D sizeof (struct interval); + tally_consing (sizeof (struct interval)); intervals_consed++; RESET_INTERVAL (val); val->gcmarkbit =3D 0; @@ -1739,7 +1755,7 @@ allocate_string (void) MALLOC_UNBLOCK_INPUT; =20 ++strings_consed; - consing_until_gc -=3D sizeof *s; + tally_consing (sizeof *s); =20 #ifdef GC_CHECK_STRING_BYTES if (!noninteractive) @@ -1859,7 +1875,7 @@ allocate_string_data (struct Lisp_String *s, old_data->string =3D NULL; } =20 - consing_until_gc -=3D needed; + tally_consing (needed); } =20 =20 @@ -2464,7 +2480,7 @@ make_float (double float_value) =20 XFLOAT_INIT (val, float_value); eassert (!XFLOAT_MARKED_P (XFLOAT (val))); - consing_until_gc -=3D sizeof (struct Lisp_Float); + tally_consing (sizeof (struct Lisp_Float)); floats_consed++; return val; } @@ -2535,8 +2551,8 @@ free_cons (struct Lisp_Cons *ptr) ptr->u.s.u.chain =3D cons_free_list; ptr->u.s.car =3D dead_object (); cons_free_list =3D ptr; - if (INT_ADD_WRAPV (consing_until_gc, sizeof *ptr, &consing_until_gc)) - consing_until_gc =3D INTMAX_MAX; + ptrdiff_t nbytes =3D sizeof *ptr; + tally_consing (-nbytes); } =20 DEFUN ("cons", Fcons, Scons, 2, 2, 0, @@ -3153,7 +3169,7 @@ allocate_vectorlike (ptrdiff_t len) if (find_suspicious_object_in_range (p, (char *) p + nbytes)) emacs_abort (); =20 - consing_until_gc -=3D nbytes; + tally_consing (nbytes); vector_cells_consed +=3D len; =20 MALLOC_UNBLOCK_INPUT; @@ -3438,7 +3454,7 @@ Its value is void, and its function definition and = property list are nil. */) MALLOC_UNBLOCK_INPUT; =20 init_symbol (val, name); - consing_until_gc -=3D sizeof (struct Lisp_Symbol); + tally_consing (sizeof (struct Lisp_Symbol)); symbols_consed++; return val; } @@ -5477,7 +5493,7 @@ staticpro (Lisp_Object const *varaddress) static void allow_garbage_collection (intmax_t consing) { - consing_until_gc =3D consing - (INTMAX_MAX - consing_until_gc); + consing_until_gc =3D consing - (HI_THRESHOLD - consing_until_gc); garbage_collection_inhibited--; } =20 @@ -5487,7 +5503,7 @@ inhibit_garbage_collection (void) ptrdiff_t count =3D SPECPDL_INDEX (); record_unwind_protect_intmax (allow_garbage_collection, consing_until_= gc); garbage_collection_inhibited++; - consing_until_gc =3D INTMAX_MAX; + consing_until_gc =3D HI_THRESHOLD; return count; } =20 @@ -5761,11 +5777,13 @@ mark_and_sweep_weak_table_contents (void) } } =20 -/* Return the number of bytes to cons between GCs, assuming - gc-cons-threshold is THRESHOLD and gc-cons-percentage is - PERCENTAGE. */ -static intmax_t -consing_threshold (intmax_t threshold, Lisp_Object percentage) +/* Return the number of bytes to cons between GCs, given THRESHOLD and + PERCENTAGE. When calculating a threshold based on PERCENTAGE, + assume SINCE_GC bytes have been allocated since the most recent GC. + The returned value is positive and no greater than HI_THRESHOLD. */ +static EMACS_INT +consing_threshold (intmax_t threshold, Lisp_Object percentage, + intmax_t since_gc) { if (!NILP (Vmemory_full)) return memory_full_cons_threshold; @@ -5775,42 +5793,33 @@ consing_threshold (intmax_t threshold, Lisp_Objec= t percentage) if (FLOATP (percentage)) { double tot =3D (XFLOAT_DATA (percentage) - * total_bytes_of_live_objects ()); + * (total_bytes_of_live_objects () + since_gc)); if (threshold < tot) { - if (tot < INTMAX_MAX) - threshold =3D tot; + if (tot < HI_THRESHOLD) + return tot; else - threshold =3D INTMAX_MAX; + return HI_THRESHOLD; } } - return threshold; + return min (threshold, HI_THRESHOLD); } } =20 -/* Adjust consing_until_gc, assuming gc-cons-threshold is THRESHOLD and - gc-cons-percentage is PERCENTAGE. */ -static Lisp_Object +/* Adjust consing_until_gc and gc_threshold, given THRESHOLD and PERCENT= AGE. + Return the updated consing_until_gc. */ + +static EMACS_INT bump_consing_until_gc (intmax_t threshold, Lisp_Object percentage) { - /* If consing_until_gc is negative leave it alone, since this prevents - negative integer overflow and a GC would have been done soon anyway= . */ - if (0 <=3D consing_until_gc) - { - threshold =3D consing_threshold (threshold, percentage); - intmax_t sum; - if (INT_ADD_WRAPV (consing_until_gc, threshold - gc_threshold, &su= m)) - { - /* Scale the threshold down so that consing_until_gc does - not overflow. */ - sum =3D INTMAX_MAX; - threshold =3D INTMAX_MAX - consing_until_gc + gc_threshold; - } - consing_until_gc =3D sum; - gc_threshold =3D threshold; - } - - return Qnil; + /* Guesstimate that half the bytes allocated since the most + recent GC are still in use. */ + EMACS_INT since_gc =3D (gc_threshold - consing_until_gc) >> 1; + EMACS_INT new_gc_threshold =3D consing_threshold (threshold, percentag= e, + since_gc); + consing_until_gc +=3D new_gc_threshold - gc_threshold; + gc_threshold =3D new_gc_threshold; + return consing_until_gc; } =20 /* Watch changes to gc-cons-threshold. */ @@ -5821,7 +5830,8 @@ watch_gc_cons_threshold (Lisp_Object symbol, Lisp_O= bject newval, intmax_t threshold; if (! (INTEGERP (newval) && integer_to_intmax (newval, &threshold))) return Qnil; - return bump_consing_until_gc (threshold, Vgc_cons_percentage); + bump_consing_until_gc (threshold, Vgc_cons_percentage); + return Qnil; } =20 /* Watch changes to gc-cons-percentage. */ @@ -5829,7 +5839,18 @@ static Lisp_Object watch_gc_cons_percentage (Lisp_Object symbol, Lisp_Object newval, Lisp_Object operation, Lisp_Object where) { - return bump_consing_until_gc (gc_cons_threshold, newval); + bump_consing_until_gc (gc_cons_threshold, newval); + return Qnil; +} + +/* It may be time to collect garbage. Recalculate consing_until_gc, + since it might depend on current usage, and do the garbage + collection if the recalculation says so. */ +void +maybe_garbage_collect (void) +{ + if (bump_consing_until_gc (gc_cons_threshold, Vgc_cons_percentage) < 0= ) + garbage_collect (); } =20 /* Subroutine of Fgarbage_collect that does most of the work. */ @@ -5841,7 +5862,6 @@ garbage_collect (void) bool message_p; ptrdiff_t count =3D SPECPDL_INDEX (); struct timespec start; - byte_ct tot_before =3D 0; =20 eassert (weak_hash_tables =3D=3D NULL); =20 @@ -5856,14 +5876,15 @@ garbage_collect (void) FOR_EACH_BUFFER (nextb) compact_buffer (nextb); =20 - if (profiler_memory_running) - tot_before =3D total_bytes_of_live_objects (); + byte_ct tot_before =3D (profiler_memory_running + ? total_bytes_of_live_objects () + : (byte_ct) -1); =20 start =3D current_timespec (); =20 /* In case user calls debug_print during GC, don't let that cause a recursive GC. */ - consing_until_gc =3D INTMAX_MAX; + consing_until_gc =3D HI_THRESHOLD; =20 /* Save what's currently displayed in the echo area. Don't do that if we are GC'ing because we've run out of memory, since @@ -5975,7 +5996,7 @@ garbage_collect (void) unblock_input (); =20 consing_until_gc =3D gc_threshold - =3D consing_threshold (gc_cons_threshold, Vgc_cons_percentage); + =3D consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0); =20 if (garbage_collection_messages && NILP (Vmemory_full)) { @@ -6008,11 +6029,11 @@ garbage_collect (void) gcs_done++; =20 /* Collect profiling data. */ - if (profiler_memory_running) + if (tot_before !=3D (byte_ct) -1) { byte_ct tot_after =3D total_bytes_of_live_objects (); - byte_ct swept =3D tot_before <=3D tot_after ? 0 : tot_before - tot= _after; - malloc_probe (min (swept, SIZE_MAX)); + if (tot_after < tot_before) + malloc_probe (min (tot_before - tot_after, SIZE_MAX)); } } =20 diff --git a/src/lisp.h b/src/lisp.h index 024e5edb26..02f8a7b668 100644 --- a/src/lisp.h +++ b/src/lisp.h @@ -3824,9 +3824,10 @@ extern void mark_maybe_objects (Lisp_Object const = *, ptrdiff_t); extern void mark_stack (char const *, char const *); extern void flush_stack_call_func (void (*func) (void *arg), void *arg); extern void garbage_collect (void); +extern void maybe_garbage_collect (void); extern const char *pending_malloc_warning; extern Lisp_Object zero_vector; -extern intmax_t consing_until_gc; +extern EMACS_INT consing_until_gc; #ifdef HAVE_PDUMPER extern int number_finalizers_run; #endif @@ -5056,7 +5057,7 @@ INLINE void maybe_gc (void) { if (consing_until_gc < 0) - garbage_collect (); + maybe_garbage_collect (); } =20 INLINE_HEADER_END --=20 2.17.1 --------------AE78A0F38C0C63807462A40B--