bug#37006: 27.0.50; garbage collection not happening after 26de2d42

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

Observed on macOS as well. Reason: free_cons has the condition

 if (INT_ADD_WRAPV (consing_until_gc, sizeof *ptr, &consing_until_gc))

which will return true (overflow) if consing_until_gc is negative, which is kind of defensible since sizeof is unsigned which causes the sum (consing_until_gc + sizeof *ptr) to be a large unsigned number that doesn't fit into consing_until_gc.

Clang 10 defines __GNUC__ to 4 which causes intprops.h to not use __builtin_add_overflow despite that being present and working.

Casting the sizeof should fix it; patch attached.


[-- Attachment #2: 0001-Avoid-unsigned-addend-in-overflow-check-bug-37006.patch --]
[-- Type: application/octet-stream, Size: 989 bytes --]

From f733339714cab022cbbdea06148795d452b183fe Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mattias=20Engdeg=C3=A5rd?= <mattiase@acm.org>
Date: Sun, 11 Aug 2019 18:11:54 +0200
Subject: [PATCH] Avoid unsigned addend in overflow check (bug#37006)

* src/alloc.c (free_cons): Cast addend to avoid spurious overflow when
consing_until_gc is negative.
---
 src/alloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/alloc.c b/src/alloc.c
index 5e089311a2..d416d32743 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -2542,7 +2542,8 @@ free_cons (struct Lisp_Cons *ptr)
   ptr->u.s.u.chain = cons_free_list;
   ptr->u.s.car = dead_object ();
   cons_free_list = ptr;
-  if (INT_ADD_WRAPV (consing_until_gc, sizeof *ptr, &consing_until_gc))
+  if (INT_ADD_WRAPV (consing_until_gc, (object_ct) sizeof *ptr,
+                     &consing_until_gc))
     consing_until_gc = OBJECT_CT_MAX;
   gcstat.total_free_conses++;
 }
-- 
2.20.1 (Apple Git-117)

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sun, 11 Aug 2019 18:23:28 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>,
>         37006@debbugs.gnu.org
> 
> Observed on macOS as well. Reason: free_cons has the condition
> 
>  if (INT_ADD_WRAPV (consing_until_gc, sizeof *ptr, &consing_until_gc))
> 
> which will return true (overflow) if consing_until_gc is negative, which is kind of defensible since sizeof is unsigned which causes the sum (consing_until_gc + sizeof *ptr) to be a large unsigned number that doesn't fit into consing_until_gc.
> 
> Clang 10 defines __GNUC__ to 4 which causes intprops.h to not use __builtin_add_overflow despite that being present and working.
> 
> Casting the sizeof should fix it; patch attached.

I believe it should be fixed now.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: Mattias Engdegård <mattiase@acm.org>,
>   eggert@cs.ucla.edu
> Date: Sun, 11 Aug 2019 17:28:11 -0300
> 
> > Thanks, I fixed this slightly differently, in a way that makes it more
> > explicit why we need a non-trivial code there.  Joseph, please see if
> > the latest master fixes the problem.
> 
> > (IMNSHO, this issue makes INT_ADD_WRAPV and friends unsafe; at the
> > very least this caveat should be prominently documented in Gnulib's
> > intprops.h.)
> 
> I have been running 94644d8 for the past hour or so and resident memory for the Emacs process is up over 1300 MB.  Also with `garbage-collection-messages' set to t, I do not see any messages about garbage collection.

Are you saying that the fix didn't solve the problem for you?  I
definitely saw a lot of GC messages after the fix where I didn't
before.  For example, if you visit xdisp.c from the Emacs sources and
page through it with C-v, don't you see a lot of GC messages?

> I should also add that after my initial report, running 26de2d42, I did eventually start seeing garbage collection messages and the memory usage stopped increasing.  Something must have triggered garbage collection to start again.

After a lot of consing, the GC would come back for a while, until it
would be effectively disabled again by some opportune code path.

If you see no GC messages for a long time, attach a debugger and look
at the value of consing_until_gc.  If its value is huge, around
LONG_MAX, the problem is still not completely solved.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Mingrone <jrm@ftfl.ca>
>> Cc: Mattias Engdegård <mattiase@acm.org>,
>>   eggert@cs.ucla.edu
>> Date: Sun, 11 Aug 2019 17:28:11 -0300

>> > Thanks, I fixed this slightly differently, in a way that makes it more
>> > explicit why we need a non-trivial code there.  Joseph, please see if
>> > the latest master fixes the problem.

>> > (IMNSHO, this issue makes INT_ADD_WRAPV and friends unsafe; at the
>> > very least this caveat should be prominently documented in Gnulib's
>> > intprops.h.)

>> I have been running 94644d8 for the past hour or so and resident
>> memory for the Emacs process is up over 1300 MB.  Also with
>> `garbage-collection-messages' set to t, I do not see any messages
>> about garbage collection.

> Are you saying that the fix didn't solve the problem for you?  I
> definitely saw a lot of GC messages after the fix where I didn't
> before.  For example, if you visit xdisp.c from the Emacs sources and
> page through it with C-v, don't you see a lot of GC messages?

>> I should also add that after my initial report, running 26de2d42, I
>> did eventually start seeing garbage collection messages and the
>> memory usage stopped increasing.  Something must have triggered
>> garbage collection to start again.

> After a lot of consing, the GC would come back for a while, until it
> would be effectively disabled again by some opportune code path.

> If you see no GC messages for a long time, attach a debugger and look
> at the value of consing_until_gc.  If its value is huge, around
> LONG_MAX, the problem is still not completely solved.

The fix did not initially work for me.  I tested a bit more.  With

1. emacs -Q
2. (setq garbage-collection-messages t)
3. page through xdisp.c

I saw lots of garbage collection messages.  But, with my init.el there
were no such messages.  My init.el looked like this.

----------------------------------------------------
(setq gc-cons-threshold most-positive-fixnum)

;; contents of init.el here

(setq gc-cons-threshold 800000) ;; default value
----------------------------------------------------

When I removed the surrounding setqs, garbage collection message were
shown again when paging through xdisp.c.

I assume that temporarily setting `gc-cons-threshold' to a large number
to temporarily prevent garbage collection, then setting it back to a
reasonable value should be acceptable.  Help for `gc-cons-threshold'
says

      By binding this temporarily to a large number, you can effectively
      prevent garbage collection during a part of the program.

--
Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: mattiase@acm.org,  eggert@cs.ucla.edu,  37006@debbugs.gnu.org
> Date: Mon, 12 Aug 2019 11:34:18 -0300
> 
> The fix did not initially work for me.  I tested a bit more.  With
> 
> 1. emacs -Q
> 2. (setq garbage-collection-messages t)
> 3. page through xdisp.c
> 
> I saw lots of garbage collection messages.  But, with my init.el there
> were no such messages.  My init.el looked like this.
> 
> ----------------------------------------------------
> (setq gc-cons-threshold most-positive-fixnum)
> 
> ;; contents of init.el here
> 
> (setq gc-cons-threshold 800000) ;; default value
> ----------------------------------------------------
> 
> When I removed the surrounding setqs, garbage collection message were
> shown again when paging through xdisp.c.
> 
> I assume that temporarily setting `gc-cons-threshold' to a large number
> to temporarily prevent garbage collection, then setting it back to a
> reasonable value should be acceptable.

Yes, of course.  There's a separate bug in the recent GC-related
changes.  Thanks for pointing this out.

Paul, the current method of updating consing_until_gc only in
garbage_collect_1 isn't workable, because it doesn't support the (very
popular nowadays) paradigm of temporarily setting gc-cons-threshold
very high: doing so avoids calling garbage_collect_1, and thus the
change of the threshold back to a lower value is never seen.  We must
have something in maybe_gc to notice the change and recompute the
threshold.  We must also notice the memory-full condition there.

We need to fix this ASAP, please.

I also don't think I understand the details of the threshold
calculations:

  if (!NILP (Vmemory_full))
    consing_until_gc = memory_full_cons_threshold;
  else
    {
      intptr_t threshold = min (max (GC_DEFAULT_THRESHOLD,
				     gc_cons_threshold >> 3),
				OBJECT_CT_MAX);
      if (FLOATP (Vgc_cons_percentage))
	{
	  double tot = (XFLOAT_DATA (Vgc_cons_percentage)
			* total_bytes_of_live_objects ());
	  if (threshold < tot)
	    {
	      if (tot < OBJECT_CT_MAX)
		threshold = tot;
	      else
		threshold = OBJECT_CT_MAX;
	    }
	}
      consing_until_gc = threshold;
    }

First, gc_cons_threshold is an EMACS_INT, so putting its value into
intptr_t is wrong in 32-bit builds --with-wide-int, right?  For the
same reason, using intptr_t for OBJECT_CT_MAX is wrong in such a
build.

And second, why does the code divide gc_cons_threshold by 8?  If the
value of gc_cons_threshold is most-positive-fixnum, that is wrong, I
think.  Did you mean to divide GC_DEFAULT_THRESHOLD instead?

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Mattias Engdegård]

12 aug. 2019 kl. 18.49 skrev Eli Zaretskii <eliz@gnu.org>:

> [...] We must
> have something in maybe_gc to notice the change and recompute the
> threshold.  We must also notice the memory-full condition there.

What about biasing consing_until_gc by gc_cons_threshold, and change the condition in maybe_gc to (the moral equivalent of)

 if (consing_until_gc < gc_cons_threshold)

? It is practically as cheap as the current test against 0.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Mon, 12 Aug 2019 19:00:37 +0200
> Cc: Joseph Mingrone <jrm@ftfl.ca>, Paul Eggert <eggert@cs.ucla.edu>,
>         37006@debbugs.gnu.org
> 
> What about biasing consing_until_gc by gc_cons_threshold, and change the condition in maybe_gc to (the moral equivalent of)
> 
>  if (consing_until_gc < gc_cons_threshold)
> 
> ? It is practically as cheap as the current test against 0.

Yes, but the full calculation of the threshold is more complicated
than that.  For starters, how do you handle gc_cons_threshold values
that are smaller than GC_DEFAULT_THRESHOLD / 10 under your proposal?
There are use cases where the value was below that before and is above
now, or the other way around, or was below and stays below.

And that's even before we consider other complications: when
memory-full is non-nil, we should use a different threshold; and what
about user changes to gc-cons-percentage?

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Mattias Engdegård]

13 aug. 2019 kl. 17.37 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> Yes, but the full calculation of the threshold is more complicated
> than that.  For starters, how do you handle gc_cons_threshold values
> that are smaller than GC_DEFAULT_THRESHOLD / 10 under your proposal?
> There are use cases where the value was below that before and is above
> now, or the other way around, or was below and stays below.

If a change to gc_cons_threshold has us end up in garbage_collect too soon, we can just adjust the bias and consing_until_gc and continue; the cost for doing so is small and amortised. Conversely, if the user raises gc_cons_threshold beyond the limit (OBJECT_CT_MAX), the intention is clearly to inhibit GC anyway.

> And that's even before we consider other complications: when
> memory-full is non-nil, we should use a different threshold; and what
> about user changes to gc-cons-percentage?

The check for memory-full was already eliminated from maybe_gc by changing consing_until_gc when that condition occurs, which seems reasonable enough. Regarding changes gc-cons-percentage, the effect will just be delayed to next GC --- is this really harmful?

I could be wrong about all this; I'm a bit confused by the threshold computation, too. However, Paul's consolidation of conditions in the hot and inlined maybe_gc makes eminently sense to me.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Tue, 13 Aug 2019 18:48:05 +0200
> Cc: jrm@ftfl.ca, eggert@cs.ucla.edu, 37006@debbugs.gnu.org
> 
> > Yes, but the full calculation of the threshold is more complicated
> > than that.  For starters, how do you handle gc_cons_threshold values
> > that are smaller than GC_DEFAULT_THRESHOLD / 10 under your proposal?
> > There are use cases where the value was below that before and is above
> > now, or the other way around, or was below and stays below.
> 
> If a change to gc_cons_threshold has us end up in garbage_collect too soon, we can just adjust the bias and consing_until_gc and continue; the cost for doing so is small and amortised. Conversely, if the user raises gc_cons_threshold beyond the limit (OBJECT_CT_MAX), the intention is clearly to inhibit GC anyway.
> 
> > And that's even before we consider other complications: when
> > memory-full is non-nil, we should use a different threshold; and what
> > about user changes to gc-cons-percentage?
> 
> The check for memory-full was already eliminated from maybe_gc by changing consing_until_gc when that condition occurs, which seems reasonable enough. Regarding changes gc-cons-percentage, the effect will just be delayed to next GC --- is this really harmful?

IOW, you think that whatever this changes in user-facing behavior is
unimportant, and we shouldn't be bothered about it.  I happen to
disagree.

> I could be wrong about all this; I'm a bit confused by the threshold computation, too. However, Paul's consolidation of conditions in the hot and inlined maybe_gc makes eminently sense to me.

I cannot say the same thing myself, sorry.  Making a frequent
operation faster definitely makes sense, even if the speedup is minor.
But doing that and as result breaking use cases that worked for years,
or even just changing their effects in a tangible manner? what's our
excuse for that?

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Mattias Engdegård]

13 aug. 2019 kl. 19.04 skrev Eli Zaretskii <eliz@gnu.org>:
> 
>> Regarding changes gc-cons-percentage, the effect will just be delayed to next GC --- is this really harmful?
> 
> IOW, you think that whatever this changes in user-facing behavior is
> unimportant, and we shouldn't be bothered about it.  I happen to
> disagree.

In contrast to gc-cons-threshold, we never really promised the effect of gc-cons-percentage to be immediate, and it seems unlikely that anyone would use the latter variable in such a way.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

Eli Zaretskii wrote:

> We must
> have something in maybe_gc to notice the change and recompute the
> threshold.

I don't see why the threshold needs to be recomputed on each maybe_gc call. I 
suppose we could add a new builtin variable type, so that the threshold could be 
recomputed whenever GC-related builtin variables are changed; that should do the 
trick without slowing down maybe_gc. But is it that important to recalculate the 
GC threshold immediately? Variables like gc-cons-threshold aren't intended for 
fine-grained control over exactly when the GC is called; they're merely advice.

> We must also notice the memory-full condition there.

memory_full already does that, no? It sets consing_until_gc. Or are you thinking 
of some other memory-full condition?

>    if (!NILP (Vmemory_full))
>      consing_until_gc = memory_full_cons_threshold;
>    else
>      {
>        intptr_t threshold = min (max (GC_DEFAULT_THRESHOLD,
> 				     gc_cons_threshold >> 3),
> 				OBJECT_CT_MAX);
>        if (FLOATP (Vgc_cons_percentage))
> 	{
> 	  double tot = (XFLOAT_DATA (Vgc_cons_percentage)
> 			* total_bytes_of_live_objects ());
> 	  if (threshold < tot)
> 	    {
> 	      if (tot < OBJECT_CT_MAX)
> 		threshold = tot;
> 	      else
> 		threshold = OBJECT_CT_MAX;
> 	    }
> 	}
>        consing_until_gc = threshold;
>      }
> 
> First, gc_cons_threshold is an EMACS_INT, so putting its value into
> intptr_t is wrong in 32-bit builds --with-wide-int, right?

That's not a problem, since the above code does min (..., OBJECT_CT_MAX) on the 
result before storing it into intptr_t.

> using intptr_t for OBJECT_CT_MAX is wrong in such a build.

I don't see why it's wrong. The idea is to count the total number of object 
bytes in use. This cannot exceed the number of bytes addressable by pointers 
regardless of the width of EMACS_INT, so intptr_t is appropriate for such counts.
> And second, why does the code divide gc_cons_threshold by 8?  If the
> value of gc_cons_threshold is most-positive-fixnum, that is wrong, I
> think.  Did you mean to divide GC_DEFAULT_THRESHOLD instead?

Right you are; that's a typo. Thanks. I fixed that in master in the attached patch.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-GC-threshold-typo.patch --]
[-- Type: text/x-patch; name="0001-Fix-GC-threshold-typo.patch", Size: 1138 bytes --]

From 1e5bda273a67f960fb834007f4653a630cd67ce9 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 13 Aug 2019 10:03:41 -0700
Subject: [PATCH] Fix GC threshold typo
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Eli Zaretskii (Bug#37006#25).
* src/alloc.c (garbage_collect_1): Fix typo in threshold calc.
Go back to dividing by 10 since the numerator’s a constant now.
Problem introduced in 2019-07-21T02:40:03Z!eggert@cs.ucla.edu.
---
 src/alloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index 39833f8dec..c7419e2fa5 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -5932,8 +5932,8 @@ garbage_collect_1 (struct gcstat *gcst)
     consing_until_gc = memory_full_cons_threshold;
   else
     {
-      intptr_t threshold = min (max (GC_DEFAULT_THRESHOLD,
-				     gc_cons_threshold >> 3),
+      intptr_t threshold = min (max (GC_DEFAULT_THRESHOLD / 10,
+				     gc_cons_threshold),
 				OBJECT_CT_MAX);
       if (FLOATP (Vgc_cons_percentage))
 	{
-- 
2.17.1

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> Cc: mattiase@acm.org, 37006@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Aug 2019 10:21:51 -0700
> 
> > We must
> > have something in maybe_gc to notice the change and recompute the
> > threshold.
> 
> I don't see why the threshold needs to be recomputed on each maybe_gc call. I 
> suppose we could add a new builtin variable type, so that the threshold could be 
> recomputed whenever GC-related builtin variables are changed; that should do the 
> trick without slowing down maybe_gc.

I don't think I understand what this proposal means in practice.  Can
you elaborate, or show an example?

> But is it that important to recalculate the GC threshold
> immediately?

How else would you succeed in reacting to the change "soon enough"?
In the use case which triggered this bug report, setting
gc-cons-threshold to a very large number disables GC for an extremely
long time, and all that time the gc-cons-threshold changes made by the
user are not acted upon.

IOW, we could perhaps explain away a delay of seconds in acting upon
the change, but we surely cannot explain away a delay of hours, let
alone days.

> Variables like gc-cons-threshold aren't intended for fine-grained
> control over exactly when the GC is called; they're merely advice.

Yes, but abrupt changes, like to most-positive-fixnum and then back to
much smaller values, should produce at least approximately the
expected behavior.  In particular, changing from most-positive-fixnum
to a value like 800000 should cause a GC "soon enough".  If we don't
test the value inside maybe_gc, what alternative mechanisms would you
suggest to produce such an effect?

> > We must also notice the memory-full condition there.
> 
> memory_full already does that, no? It sets consing_until_gc.

It sets it to a positive value, so no immediate GC will follow.  The
original code was setting the threshold to a very small value, so GC
would happen immediately.  I think the code in memory_full which sets
consing_until_gc should be amended to (a) not raise consing_until_gc
if the current value is already below memory_full_cons_threshold, and
(b) probably even set it to the negative of sizeof (struct cons_block)
so as to cause an immediate GC.

> > First, gc_cons_threshold is an EMACS_INT, so putting its value into
> > intptr_t is wrong in 32-bit builds --with-wide-int, right?
> 
> That's not a problem, since the above code does min (..., OBJECT_CT_MAX) on the 
> result before storing it into intptr_t.

??? But that effectively disallows/ignores values of gc-cons-threshold
above LONG_MAX.  Why would we make such a backward-incompatible change?
When the user sets the value of that variable, the variable should
keep its value, and we should be able to compare the threshold with
the value of the user variable.  If nothing else, arbitrarily throwing
away high-order bits of the value is a sure path towards bugs due to
confusion between these two value ranges.

> > using intptr_t for OBJECT_CT_MAX is wrong in such a build.
> 
> I don't see why it's wrong.

Because OBJECT_CT_MAX should have the value EMACS_INT_MAX.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2458 bytes --]

Eli Zaretskii wrote:

> OBJECT_CT_MAX should have the value EMACS_INT_MAX.

Not if EMACS_INT_MAX < INTPTR_MAX, since object counts might overflow in that 
case. However, I take your point that consing_until_gc can easily be made to 
hold any fixnum value, so I installed the first attached patch. This is to some 
extent overkill, since these variables should not be assumed to have this sort 
of fine-grained control, but the change is tiny so should be fine.

Come to think of it, the limit should be INTMAX_MAX not EMACS_INT_MAX since 
gc-cons-threshold could exceed EMACS_INT_MAX. So I installed the second attached 
patch to do that.

>> I don't see why the threshold needs to be recomputed on each maybe_gc call. I
>> suppose we could add a new builtin variable type, so that the threshold could be
>> recomputed whenever GC-related builtin variables are changed; that should do the
>> trick without slowing down maybe_gc.
> 
> I don't think I understand what this proposal means in practice.  Can
> you elaborate, or show an example?

The idea would be to have a type that is like struct Lisp_Objfwd but with an 
extra member, a function to be called whenever the variable is accessed. (Or 
perhaps two extra members, a getter and a setter.) This could be useful for 
other builtin vars, I suspect.

> How else would you succeed in reacting to the change "soon enough"?

There are other possibilities. We could have a timer, for example.
>>> We must also notice the memory-full condition there.
>>
>> memory_full already does that, no? It sets consing_until_gc.
> 
> It sets it to a positive value, so no immediate GC will follow.  The
> original code was setting the threshold to a very small value, so GC
> would happen immediately.

Are you talking about the change in commit 
2019-07-20T02:40:03Z!eggert@cs.ucla.edu 
(26de2d42d0460c5b193456950a568cb04a29dc00)? If so, I'm not quite following, as 
the old code did not GC until consing_since_gc > memory_full_cons_threshold. I 
expect that the idea was to not thrash doing GCs when memory is full.

I think the code in memory_full which sets
> consing_until_gc should be amended to (a) not raise consing_until_gc
> if the current value is already below memory_full_cons_threshold, and
> (b) probably even set it to the negative of sizeof (struct cons_block)
> so as to cause an immediate GC.

Immediate-GC might cause GC thrashing, no? But (a) makes sense so I installed 
the third attached patch.

[-- Attachment #2: 0001-Let-consing_until_gc-exceed-INTPTR_MAX.patch --]
[-- Type: text/x-patch, Size: 4076 bytes --]

From a354736e1dfe5a7e4ddbb1ee7f1373be2b5bbe09 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 13 Aug 2019 12:11:35 -0700
Subject: [PATCH 1/3] Let consing_until_gc exceed INTPTR_MAX

Suggested by Eli Zaretskii (Bug#37006#46).
* src/alloc.c (consing_until_gc): Now of type consing_ct.
All uses changed, so gc-cons-threshold no longer saturates
against OBJECT_CT_MAX.
(object_ct): Move typedef here from lisp.h.
* src/lisp.h (consing_ct, CONSING_CT_MAX): New type and macro.
(OBJECT_CT_MAX): Remove.  Replace all uses with CONSING_CT_MAX.
---
 src/alloc.c | 21 ++++++++++-----------
 src/lisp.h  | 10 +++++++---
 2 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index c7419e2fa5..7bed3f4488 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -224,7 +224,7 @@ struct emacs_globals globals;
 
 /* maybe_gc collects garbage if this goes negative.  */
 
-object_ct consing_until_gc;
+consing_ct consing_until_gc;
 
 #ifdef HAVE_PDUMPER
 /* Number of finalizers run: used to loop over GC until we stop
@@ -236,9 +236,10 @@ int number_finalizers_run;
 
 bool gc_in_progress;
 
-/* System byte counts reported by GC.  */
+/* System byte and object counts reported by GC.  */
 
 typedef uintptr_t byte_ct;
+typedef intptr_t object_ct;
 
 /* Number of live and free conses etc.  */
 
@@ -2546,7 +2547,7 @@ free_cons (struct Lisp_Cons *ptr)
      might incorrectly return non-zero.  */
   int incr = sizeof *ptr;
   if (INT_ADD_WRAPV (consing_until_gc, incr, &consing_until_gc))
-    consing_until_gc = OBJECT_CT_MAX;
+    consing_until_gc = CONSING_CT_MAX;
   gcstat.total_free_conses++;
 }
 
@@ -5501,7 +5502,7 @@ staticpro (Lisp_Object const *varaddress)
 static void
 allow_garbage_collection (intmax_t consing)
 {
-  consing_until_gc = consing - (OBJECT_CT_MAX - consing_until_gc);
+  consing_until_gc = consing - (CONSING_CT_MAX - consing_until_gc);
   garbage_collection_inhibited--;
 }
 
@@ -5511,7 +5512,7 @@ inhibit_garbage_collection (void)
   ptrdiff_t count = SPECPDL_INDEX ();
   record_unwind_protect_intmax (allow_garbage_collection, consing_until_gc);
   garbage_collection_inhibited++;
-  consing_until_gc = OBJECT_CT_MAX;
+  consing_until_gc = CONSING_CT_MAX;
   return count;
 }
 
@@ -5817,7 +5818,7 @@ garbage_collect_1 (struct gcstat *gcst)
 
   /* In case user calls debug_print during GC,
      don't let that cause a recursive GC.  */
-  consing_until_gc = OBJECT_CT_MAX;
+  consing_until_gc = CONSING_CT_MAX;
 
   /* Save what's currently displayed in the echo area.  Don't do that
      if we are GC'ing because we've run out of memory, since
@@ -5932,19 +5933,17 @@ garbage_collect_1 (struct gcstat *gcst)
     consing_until_gc = memory_full_cons_threshold;
   else
     {
-      intptr_t threshold = min (max (GC_DEFAULT_THRESHOLD / 10,
-				     gc_cons_threshold),
-				OBJECT_CT_MAX);
+      consing_ct threshold = max (gc_cons_threshold, GC_DEFAULT_THRESHOLD / 10);
       if (FLOATP (Vgc_cons_percentage))
 	{
 	  double tot = (XFLOAT_DATA (Vgc_cons_percentage)
 			* total_bytes_of_live_objects ());
 	  if (threshold < tot)
 	    {
-	      if (tot < OBJECT_CT_MAX)
+	      if (tot < CONSING_CT_MAX)
 		threshold = tot;
 	      else
-		threshold = OBJECT_CT_MAX;
+		threshold = CONSING_CT_MAX;
 	    }
 	}
       consing_until_gc = threshold;
diff --git a/src/lisp.h b/src/lisp.h
index 63baab5d63..043f2f738e 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3793,9 +3793,13 @@ extern void flush_stack_call_func (void (*func) (void *arg), void *arg);
 extern void garbage_collect (void);
 extern const char *pending_malloc_warning;
 extern Lisp_Object zero_vector;
-typedef intptr_t object_ct; /* Signed type of object counts reported by GC.  */
-#define OBJECT_CT_MAX INTPTR_MAX
-extern object_ct consing_until_gc;
+#define CONSING_CT_MAX max (INTPTR_MAX, EMACS_INT_MAX)
+#if CONSING_CT_MAX == INTPTR_MAX
+typedef intptr_t consing_ct;
+#else
+typedef EMACS_INT consing_ct;
+#endif
+extern consing_ct consing_until_gc;
 #ifdef HAVE_PDUMPER
 extern int number_finalizers_run;
 #endif
-- 
2.17.1


[-- Attachment #3: 0002-Let-consing_until_gc-exceed-EMACS_INT_MAX.patch --]
[-- Type: text/x-patch, Size: 3478 bytes --]

From b80559be212292d44ce14ca5e94505cab4d9a868 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 13 Aug 2019 12:20:40 -0700
Subject: [PATCH 2/3] Let consing_until_gc exceed EMACS_INT_MAX

This builds on the previous patch.
* src/alloc.c (consing_until_gc): Now of type intmax_t,
since gc-cons-threshold can be up to INTMAX_MAX.  All uses changed.
* src/lisp.h (CONSING_CT_MAX, consing_ct): Remove.
---
 src/alloc.c | 16 ++++++++--------
 src/lisp.h  |  8 +-------
 2 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index 7bed3f4488..14b0a7b838 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -224,7 +224,7 @@ struct emacs_globals globals;
 
 /* maybe_gc collects garbage if this goes negative.  */
 
-consing_ct consing_until_gc;
+intmax_t consing_until_gc;
 
 #ifdef HAVE_PDUMPER
 /* Number of finalizers run: used to loop over GC until we stop
@@ -2547,7 +2547,7 @@ free_cons (struct Lisp_Cons *ptr)
      might incorrectly return non-zero.  */
   int incr = sizeof *ptr;
   if (INT_ADD_WRAPV (consing_until_gc, incr, &consing_until_gc))
-    consing_until_gc = CONSING_CT_MAX;
+    consing_until_gc = INTMAX_MAX;
   gcstat.total_free_conses++;
 }
 
@@ -5502,7 +5502,7 @@ staticpro (Lisp_Object const *varaddress)
 static void
 allow_garbage_collection (intmax_t consing)
 {
-  consing_until_gc = consing - (CONSING_CT_MAX - consing_until_gc);
+  consing_until_gc = consing - (INTMAX_MAX - consing_until_gc);
   garbage_collection_inhibited--;
 }
 
@@ -5512,7 +5512,7 @@ inhibit_garbage_collection (void)
   ptrdiff_t count = SPECPDL_INDEX ();
   record_unwind_protect_intmax (allow_garbage_collection, consing_until_gc);
   garbage_collection_inhibited++;
-  consing_until_gc = CONSING_CT_MAX;
+  consing_until_gc = INTMAX_MAX;
   return count;
 }
 
@@ -5818,7 +5818,7 @@ garbage_collect_1 (struct gcstat *gcst)
 
   /* In case user calls debug_print during GC,
      don't let that cause a recursive GC.  */
-  consing_until_gc = CONSING_CT_MAX;
+  consing_until_gc = INTMAX_MAX;
 
   /* Save what's currently displayed in the echo area.  Don't do that
      if we are GC'ing because we've run out of memory, since
@@ -5933,17 +5933,17 @@ garbage_collect_1 (struct gcstat *gcst)
     consing_until_gc = memory_full_cons_threshold;
   else
     {
-      consing_ct threshold = max (gc_cons_threshold, GC_DEFAULT_THRESHOLD / 10);
+      intmax_t threshold = max (gc_cons_threshold, GC_DEFAULT_THRESHOLD / 10);
       if (FLOATP (Vgc_cons_percentage))
 	{
 	  double tot = (XFLOAT_DATA (Vgc_cons_percentage)
 			* total_bytes_of_live_objects ());
 	  if (threshold < tot)
 	    {
-	      if (tot < CONSING_CT_MAX)
+	      if (tot < INTMAX_MAX)
 		threshold = tot;
 	      else
-		threshold = CONSING_CT_MAX;
+		threshold = INTMAX_MAX;
 	    }
 	}
       consing_until_gc = threshold;
diff --git a/src/lisp.h b/src/lisp.h
index 043f2f738e..0370c52fad 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3793,13 +3793,7 @@ extern void flush_stack_call_func (void (*func) (void *arg), void *arg);
 extern void garbage_collect (void);
 extern const char *pending_malloc_warning;
 extern Lisp_Object zero_vector;
-#define CONSING_CT_MAX max (INTPTR_MAX, EMACS_INT_MAX)
-#if CONSING_CT_MAX == INTPTR_MAX
-typedef intptr_t consing_ct;
-#else
-typedef EMACS_INT consing_ct;
-#endif
-extern consing_ct consing_until_gc;
+extern intmax_t consing_until_gc;
 #ifdef HAVE_PDUMPER
 extern int number_finalizers_run;
 #endif
-- 
2.17.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Don-t-increase-consing_until_gc-when-out-of-memory.patch --]
[-- Type: text/x-patch; name="0003-Don-t-increase-consing_until_gc-when-out-of-memory.patch", Size: 1063 bytes --]

From f4974d6fe6137f436763998be27afafea9866098 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 13 Aug 2019 12:28:53 -0700
Subject: [PATCH 3/3] =?UTF-8?q?Don=E2=80=99t=20increase=20consing=5Funtil?=
 =?UTF-8?q?=5Fgc=20when=20out=20of=20memory?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* src/alloc.c (memory_full): Don’t increase consing_until_gc.
Suggested by Eli Zaretskii (Bug#37006#46).
---
 src/alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/alloc.c b/src/alloc.c
index 14b0a7b838..0548a09cb8 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3866,7 +3866,7 @@ memory_full (size_t nbytes)
   if (! enough_free_memory)
     {
       Vmemory_full = Qt;
-      consing_until_gc = memory_full_cons_threshold;
+      consing_until_gc = min (consing_until_gc, memory_full_cons_threshold);
 
       /* The first time we get here, free the spare memory.  */
       for (int i = 0; i < ARRAYELTS (spare_memory); i++)
-- 
2.17.1

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> Cc: jrm@ftfl.ca, mattiase@acm.org, 37006@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Aug 2019 12:32:24 -0700
> 
> > OBJECT_CT_MAX should have the value EMACS_INT_MAX.
> 
> Not if EMACS_INT_MAX < INTPTR_MAX, since object counts might overflow in that 
> case. However, I take your point that consing_until_gc can easily be made to 
> hold any fixnum value, so I installed the first attached patch. This is to some 
> extent overkill, since these variables should not be assumed to have this sort 
> of fine-grained control, but the change is tiny so should be fine.

Thanks.

However, I'd rather we don't invent new data types unless really
necessary.  I think we should simply use EMACS_INT (see below), but
even if we end up using intptr_max, let's just use that directly, not
introduce yet another type which we will have to look up every time we
read this code.  And likewise with the corresponding _MAX value.
Using a non-standard data type makes the code harder to read.

> Come to think of it, the limit should be INTMAX_MAX not EMACS_INT_MAX since 
> gc-cons-threshold could exceed EMACS_INT_MAX.

Sorry, I don't think I follow.  gc-cons-threshold is a Lisp integer, a
fixnum, so it cannot exceed EMACS_INT_MAX, I think.

> The idea would be to have a type that is like struct Lisp_Objfwd but with an 
> extra member, a function to be called whenever the variable is accessed. (Or 
> perhaps two extra members, a getter and a setter.) This could be useful for 
> other builtin vars, I suspect.

Ah, okay.  Can we use for this purpose the existing trapped_write
field of Lisp_Symbol that is the base for implementing Lisp watcher
functions?

> > How else would you succeed in reacting to the change "soon enough"?
> 
> There are other possibilities. We could have a timer, for example.

I don't think timers are reliable enough, as they can be deferred for
arbitrarily long time interval by some Lisp that takes a long time to
finish.

> >>> We must also notice the memory-full condition there.
> >>
> >> memory_full already does that, no? It sets consing_until_gc.
> > 
> > It sets it to a positive value, so no immediate GC will follow.  The
> > original code was setting the threshold to a very small value, so GC
> > would happen immediately.
> 
> Are you talking about the change in commit 
> 2019-07-20T02:40:03Z!eggert@cs.ucla.edu 
> (26de2d42d0460c5b193456950a568cb04a29dc00)? If so, I'm not quite following, as 
> the old code did not GC until consing_since_gc > memory_full_cons_threshold. I 
> expect that the idea was to not thrash doing GCs when memory is full.

With the old code, whenever memory-full was non-nil, and
consing_since_gc was more than the size of cons_block (about 1KB on my
system), the very next maybe_gc call would actually trigger GC.  With
the new code, no matter how much consing happened before memory-full
became non-nil, we still need to cons 1KB worth of objects before GC
happens.  This 1KB might be critical when we are out of memory.

> Immediate-GC might cause GC thrashing, no?

Not sure how, can you elaborate?

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Paul Eggert]

Eli Zaretskii wrote:

> However, I'd rather we don't invent new data types unless really
> necessary.

I did that yesterday, in commit 2019-08-13T19:20:40Z!eggert@cs.ucla.edu 
(b80559be212292d44ce14ca5e94505cab4d9a868).

> gc-cons-threshold is a Lisp integer, a
> fixnum, so it cannot exceed EMACS_INT_MAX, I think.

No, (setq gc-cons-threshold (1+ most-positive-fixnum)) works and does the right 
thing. The variable's value can be any intmax_t value. This is useful for 
quantities like GC object byte counts that might not fit into fixnums.

> Can we use for this purpose the existing trapped_write
> field of Lisp_Symbol that is the base for implementing Lisp watcher
> functions?

Don't see why not.

> With the old code, whenever memory-full was non-nil, and
> consing_since_gc was more than the size of cons_block (about 1KB on my
> system), the very next maybe_gc call would actually trigger GC.  With
> the new code, no matter how much consing happened before memory-full
> became non-nil, we still need to cons 1KB worth of objects before GC
> happens.  This 1KB might be critical when we are out of memory.

I don't think the scenario is worth worrying about doing a GC now rather than 
later. But if we go the trapped_write route, this issue won't matter since the 
GC will be done quickly.

>> Immediate-GC might cause GC thrashing, no?
> 
> Not sure how, can you elaborate?

When EMacs is low on memory, if we're not careful Emacs could GC every time 
maybe_gc is called, which will be roughly equivalent to Emacs hanging and doing 
nothing.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> Cc: jrm@ftfl.ca, mattiase@acm.org, 37006@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 14 Aug 2019 18:37:44 -0700
> 
> > However, I'd rather we don't invent new data types unless really
> > necessary.
> 
> I did that yesterday, in commit 2019-08-13T19:20:40Z!eggert@cs.ucla.edu 
> (b80559be212292d44ce14ca5e94505cab4d9a868).

OK, but I still hope we will switch to EMACS_INT instead.

> > gc-cons-threshold is a Lisp integer, a
> > fixnum, so it cannot exceed EMACS_INT_MAX, I think.
> 
> No, (setq gc-cons-threshold (1+ most-positive-fixnum)) works and does the right 
> thing.

That makes gc-cons-threshold a bignum, right?  I don't see why we
should support such large values of the threshold, we never did
before.  most-positive-fixnum on 32-bit systems is large enough for
every practical purpose.  Also, supporting the full 32 bits (and 64
bits in 64-bit builds) will also allow contradictory situation whereby
gc-cons-threshold is higher than what we say should be used to disable
GC.

> The variable's value can be any intmax_t value. This is useful for 
> quantities like GC object byte counts that might not fit into fixnums.

Why do we need to talk about how many objects are there?  GC threshold
is not about counting objects, it's about deciding when to GC.

> > Can we use for this purpose the existing trapped_write
> > field of Lisp_Symbol that is the base for implementing Lisp watcher
> > functions?
> 
> Don't see why not.

Are you working on that, or should someone else do it?

> > With the old code, whenever memory-full was non-nil, and
> > consing_since_gc was more than the size of cons_block (about 1KB on my
> > system), the very next maybe_gc call would actually trigger GC.  With
> > the new code, no matter how much consing happened before memory-full
> > became non-nil, we still need to cons 1KB worth of objects before GC
> > happens.  This 1KB might be critical when we are out of memory.
> 
> I don't think the scenario is worth worrying about doing a GC now rather than 
> later. But if we go the trapped_write route, this issue won't matter since the 
> GC will be done quickly.

I don't think I understand how trapped writes could help in the case
of memory-full, since that situation happens independently of setting
the value of gc-cons-threshold.

> >> Immediate-GC might cause GC thrashing, no?
> > 
> > Not sure how, can you elaborate?
> 
> When EMacs is low on memory, if we're not careful Emacs could GC every time 
> maybe_gc is called, which will be roughly equivalent to Emacs hanging and doing 
> nothing.

Right, but that's not what I proposed.  I proposed to trigger an
immediate GC only the first time we detect memory-full.  Thereafter,
the threshold will be set to 1KB, which should prevent thrashing.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Paul Eggert]

Eli Zaretskii wrote:

> That makes gc-cons-threshold a bignum, right?

If the user sets it to a bignum, yes. Ordinarily it's not.

> most-positive-fixnum on 32-bit systems is large enough for
> every practical purpose.

It's not that hard for the number of consed bytes to exceed most-positive-fixnum 
on a 32-bit Emacs. Here's a simple test case to illustrate the phenomenon:

(let* ((cons-size (car (cdr (car (garbage-collect)))))
        (long-length (1+ (/ most-positive-fixnum cons-size)))
        (l (make-list long-length nil)))
   (cons most-positive-fixnum (* cons-size (length l))))

This yielded (536870911 . 536870912) on the 32-bit Emacs that I just built. Of 
course a practical application would likely have a bunch of smaller lists, but 
the same basic idea applies. On such a platform, a user who wants to disable GC 
while fiddling with a bunch of large lists will need to set gc-cons-threshold to 
a bignum.

> supporting the full 32 bits (and 64
> bits in 64-bit builds) will also allow contradictory situation whereby
> gc-cons-threshold is higher than what we say should be used to disable
> GC.

Sorry, I'm not following. If setting gc-cons-threshold to a large value 
effectively disables GC, then setting gc-cons-threshold to a larger value should 
do the same thing. This is independent of any particular large value that we 
suggest in the manual.
>> The variable's value can be any intmax_t value. This is useful for
>> quantities like GC object byte counts that might not fit into fixnums.
> 
> Why do we need to talk about how many objects are there?  GC threshold
> is not about counting objects, it's about deciding when to GC.

The GC threshold is part of a related set of integers that count objects and 
bytes, for use in the returned value of garbage-collect among other things. It's 
convenient for it to be as least as wide as those other integers, so that 
calculations involving it do not overflow.

>> Don't see why not.
> 
> Are you working on that, or should someone else do it?

I can add it to my list of things to do. To my mind, getting the timestamp API 
nailed down is more urgent, though, because fiddling with GC heuristics doesn't 
affect the API.

> Right, but that's not what I proposed.  I proposed to trigger an
> immediate GC only the first time we detect memory-full.  Thereafter,
> the threshold will be set to 1KB, which should prevent thrashing.

Isn't it more complicated than that? Emacs can be low on memory, but can then 
get more and not be low on memory, and then be low on memory again later.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> Cc: jrm@ftfl.ca, mattiase@acm.org, 37006@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 15 Aug 2019 11:51:06 -0700
> 
> > most-positive-fixnum on 32-bit systems is large enough for
> > every practical purpose.
> 
> It's not that hard for the number of consed bytes to exceed most-positive-fixnum 
> on a 32-bit Emacs. Here's a simple test case to illustrate the phenomenon:
> 
> (let* ((cons-size (car (cdr (car (garbage-collect)))))
>         (long-length (1+ (/ most-positive-fixnum cons-size)))
>         (l (make-list long-length nil)))
>    (cons most-positive-fixnum (* cons-size (length l))))
> 
> This yielded (536870911 . 536870912) on the 32-bit Emacs that I just built. Of 
> course a practical application would likely have a bunch of smaller lists, but 
> the same basic idea applies. On such a platform, a user who wants to disable GC 
> while fiddling with a bunch of large lists will need to set gc-cons-threshold to 
> a bignum.

I don't see why we must complicate our code to support such a use
case.  Users who want to disable GC while consing so much object will
have to learn that they cannot, not on a 32-bit machine.

> > supporting the full 32 bits (and 64
> > bits in 64-bit builds) will also allow contradictory situation whereby
> > gc-cons-threshold is higher than what we say should be used to disable
> > GC.
> 
> Sorry, I'm not following. If setting gc-cons-threshold to a large value 
> effectively disables GC, then setting gc-cons-threshold to a larger value should 
> do the same thing.

most-positive-fixnum is infinity for this purpose, and there should
not be numbers greater than infinity.  It's confusing and hard to
explain.

> > Why do we need to talk about how many objects are there?  GC threshold
> > is not about counting objects, it's about deciding when to GC.
> 
> The GC threshold is part of a related set of integers that count objects and 
> bytes, for use in the returned value of garbage-collect among other things.

No, they are just a means to decide when it's a good time to GC.  Very
large values are used to effectively disable GC, very small values to
cause GC "soon".  That's all, there are no requirements to count
objects or to have any accurate bookeeping of how many objects or
bytes were consed.

> > Are you working on that, or should someone else do it?
> 
> I can add it to my list of things to do. To my mind, getting the timestamp API 
> nailed down is more urgent, though, because fiddling with GC heuristics doesn't 
> affect the API.

Well, meanwhile we've broken a very popular use case, so I think
fixing this is rather urgent.

> > Right, but that's not what I proposed.  I proposed to trigger an
> > immediate GC only the first time we detect memory-full.  Thereafter,
> > the threshold will be set to 1KB, which should prevent thrashing.
> 
> Isn't it more complicated than that? Emacs can be low on memory, but can then 
> get more and not be low on memory, and then be low on memory again later.

Maybe so, but we had such code in Emacs for decades.  I just want to
avoid losing those features.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

On 8/15/19 7:17 AM, Eli Zaretskii wrote:
> OK, but I still hope we will switch to EMACS_INT instead.

I installed the attached patch into master, and it changes the relevant 
threshold variables to EMACS_INT, as part of a somewhat-larger cleanup. This 
should make a practical difference only on 32-bit Emacs without --with-wide-int.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Improve-gc-cons-percentage-calculation.patch --]
[-- Type: text/x-patch; name="0001-Improve-gc-cons-percentage-calculation.patch", Size: 12388 bytes --]

From bac66302e92bdd3a353102d2076548e7e83d92e5 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 14 Sep 2019 00:32:01 -0700
Subject: [PATCH] Improve gc-cons-percentage calculation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The old calculation relied on a hodgpodge of partly updated GC
stats to find a number to multiply gc-cons-percentage by.
The new one counts data found by the previous GC, plus half of
the data allocated since then; this is more systematic albeit
still ad hoc.
* src/alloc.c (consing_until_gc, gc_threshold, consing_threshold):
Now EMACS_INT, not intmax_t.
(HI_THRESHOLD): New macro.
(tally_consing): New function.
(make_interval, allocate_string, allocate_string_data)
(make_float, free_cons, allocate_vectorlike, Fmake_symbol): Use it.
(allow_garbage_collection, inhibit_garbage_collection)
(consing_threshold, garbage_collect):
Use HI_THRESHOLD rather than INTMAX_MAX.
(consing_threshold): New arg SINCE_GC.  All callers changed.
(bump_consing_until_gc): Return new consing_until_gc, instead of
nil.  All callers changed.  Don’t worry about overflow since we
now saturate at HI_THRESHOLD.  Guess that half of
recently-allocated objects are still alive, instead of relying on
the previous (even less-accurate) hodgepodge.
(maybe_garbage_collect): New function.
(garbage_collect): Work even if a finalizer disables or enables
memory profiling.  Do not use malloc_probe if GC reclaimed nothing.
* src/lisp.h (maybe_gc): Call maybe_garbage_collect instead
of garbage_collect.
---
 src/alloc.c | 127 ++++++++++++++++++++++++++++++----------------------
 src/lisp.h  |   5 ++-
 2 files changed, 77 insertions(+), 55 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index ca8311cc00..497f600551 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -224,7 +224,7 @@ struct emacs_globals globals;
 
 /* maybe_gc collects garbage if this goes negative.  */
 
-intmax_t consing_until_gc;
+EMACS_INT consing_until_gc;
 
 #ifdef HAVE_PDUMPER
 /* Number of finalizers run: used to loop over GC until we stop
@@ -238,9 +238,16 @@ bool gc_in_progress;
 
 /* System byte and object counts reported by GC.  */
 
+/* Assume byte counts fit in uintptr_t and object counts fit into
+   intptr_t.  */
 typedef uintptr_t byte_ct;
 typedef intptr_t object_ct;
 
+/* Large-magnitude value for a threshold count, which fits in EMACS_INT.
+   Using only half the EMACS_INT range avoids overflow hassles.
+   There is no need to fit these counts into fixnums.  */
+#define HI_THRESHOLD (EMACS_INT_MAX / 2)
+
 /* Number of live and free conses etc. counted by the most-recent GC.  */
 
 static struct gcstat
@@ -299,7 +306,7 @@ static intptr_t garbage_collection_inhibited;
 
 /* The GC threshold in bytes, the last time it was calculated
    from gc-cons-threshold and gc-cons-percentage.  */
-static intmax_t gc_threshold;
+static EMACS_INT gc_threshold;
 
 /* If nonzero, this is a warning delivered by malloc and not yet
    displayed.  */
@@ -536,6 +543,15 @@ XFLOAT_INIT (Lisp_Object f, double n)
   XFLOAT (f)->u.data = n;
 }
 
+/* Account for allocation of NBYTES in the heap.  This is a separate
+   function to avoid hassles with implementation-defined conversion
+   from unsigned to signed types.  */
+static void
+tally_consing (ptrdiff_t nbytes)
+{
+  consing_until_gc -= nbytes;
+}
+
 #ifdef DOUG_LEA_MALLOC
 static bool
 pointers_fit_in_lispobj_p (void)
@@ -1372,7 +1388,7 @@ make_interval (void)
 
   MALLOC_UNBLOCK_INPUT;
 
-  consing_until_gc -= sizeof (struct interval);
+  tally_consing (sizeof (struct interval));
   intervals_consed++;
   RESET_INTERVAL (val);
   val->gcmarkbit = 0;
@@ -1739,7 +1755,7 @@ allocate_string (void)
   MALLOC_UNBLOCK_INPUT;
 
   ++strings_consed;
-  consing_until_gc -= sizeof *s;
+  tally_consing (sizeof *s);
 
 #ifdef GC_CHECK_STRING_BYTES
   if (!noninteractive)
@@ -1859,7 +1875,7 @@ allocate_string_data (struct Lisp_String *s,
       old_data->string = NULL;
     }
 
-  consing_until_gc -= needed;
+  tally_consing (needed);
 }
 
 
@@ -2464,7 +2480,7 @@ make_float (double float_value)
 
   XFLOAT_INIT (val, float_value);
   eassert (!XFLOAT_MARKED_P (XFLOAT (val)));
-  consing_until_gc -= sizeof (struct Lisp_Float);
+  tally_consing (sizeof (struct Lisp_Float));
   floats_consed++;
   return val;
 }
@@ -2535,8 +2551,8 @@ free_cons (struct Lisp_Cons *ptr)
   ptr->u.s.u.chain = cons_free_list;
   ptr->u.s.car = dead_object ();
   cons_free_list = ptr;
-  if (INT_ADD_WRAPV (consing_until_gc, sizeof *ptr, &consing_until_gc))
-    consing_until_gc = INTMAX_MAX;
+  ptrdiff_t nbytes = sizeof *ptr;
+  tally_consing (-nbytes);
 }
 
 DEFUN ("cons", Fcons, Scons, 2, 2, 0,
@@ -3153,7 +3169,7 @@ allocate_vectorlike (ptrdiff_t len)
   if (find_suspicious_object_in_range (p, (char *) p + nbytes))
     emacs_abort ();
 
-  consing_until_gc -= nbytes;
+  tally_consing (nbytes);
   vector_cells_consed += len;
 
   MALLOC_UNBLOCK_INPUT;
@@ -3438,7 +3454,7 @@ Its value is void, and its function definition and property list are nil.  */)
   MALLOC_UNBLOCK_INPUT;
 
   init_symbol (val, name);
-  consing_until_gc -= sizeof (struct Lisp_Symbol);
+  tally_consing (sizeof (struct Lisp_Symbol));
   symbols_consed++;
   return val;
 }
@@ -5477,7 +5493,7 @@ staticpro (Lisp_Object const *varaddress)
 static void
 allow_garbage_collection (intmax_t consing)
 {
-  consing_until_gc = consing - (INTMAX_MAX - consing_until_gc);
+  consing_until_gc = consing - (HI_THRESHOLD - consing_until_gc);
   garbage_collection_inhibited--;
 }
 
@@ -5487,7 +5503,7 @@ inhibit_garbage_collection (void)
   ptrdiff_t count = SPECPDL_INDEX ();
   record_unwind_protect_intmax (allow_garbage_collection, consing_until_gc);
   garbage_collection_inhibited++;
-  consing_until_gc = INTMAX_MAX;
+  consing_until_gc = HI_THRESHOLD;
   return count;
 }
 
@@ -5761,11 +5777,13 @@ mark_and_sweep_weak_table_contents (void)
     }
 }
 
-/* Return the number of bytes to cons between GCs, assuming
-   gc-cons-threshold is THRESHOLD and gc-cons-percentage is
-   PERCENTAGE.  */
-static intmax_t
-consing_threshold (intmax_t threshold, Lisp_Object percentage)
+/* Return the number of bytes to cons between GCs, given THRESHOLD and
+   PERCENTAGE.  When calculating a threshold based on PERCENTAGE,
+   assume SINCE_GC bytes have been allocated since the most recent GC.
+   The returned value is positive and no greater than HI_THRESHOLD.  */
+static EMACS_INT
+consing_threshold (intmax_t threshold, Lisp_Object percentage,
+		   intmax_t since_gc)
 {
   if (!NILP (Vmemory_full))
     return memory_full_cons_threshold;
@@ -5775,42 +5793,33 @@ consing_threshold (intmax_t threshold, Lisp_Object percentage)
       if (FLOATP (percentage))
 	{
 	  double tot = (XFLOAT_DATA (percentage)
-			* total_bytes_of_live_objects ());
+			* (total_bytes_of_live_objects () + since_gc));
 	  if (threshold < tot)
 	    {
-	      if (tot < INTMAX_MAX)
-		threshold = tot;
+	      if (tot < HI_THRESHOLD)
+		return tot;
 	      else
-		threshold = INTMAX_MAX;
+		return HI_THRESHOLD;
 	    }
 	}
-      return threshold;
+      return min (threshold, HI_THRESHOLD);
     }
 }
 
-/* Adjust consing_until_gc, assuming gc-cons-threshold is THRESHOLD and
-   gc-cons-percentage is PERCENTAGE.  */
-static Lisp_Object
+/* Adjust consing_until_gc and gc_threshold, given THRESHOLD and PERCENTAGE.
+   Return the updated consing_until_gc.  */
+
+static EMACS_INT
 bump_consing_until_gc (intmax_t threshold, Lisp_Object percentage)
 {
-  /* If consing_until_gc is negative leave it alone, since this prevents
-     negative integer overflow and a GC would have been done soon anyway.  */
-  if (0 <= consing_until_gc)
-    {
-      threshold = consing_threshold (threshold, percentage);
-      intmax_t sum;
-      if (INT_ADD_WRAPV (consing_until_gc, threshold - gc_threshold, &sum))
-	{
-	  /* Scale the threshold down so that consing_until_gc does
-	     not overflow.  */
-	  sum = INTMAX_MAX;
-	  threshold = INTMAX_MAX - consing_until_gc + gc_threshold;
-	}
-      consing_until_gc = sum;
-      gc_threshold = threshold;
-    }
-
-  return Qnil;
+  /* Guesstimate that half the bytes allocated since the most
+     recent GC are still in use.  */
+  EMACS_INT since_gc = (gc_threshold - consing_until_gc) >> 1;
+  EMACS_INT new_gc_threshold = consing_threshold (threshold, percentage,
+						  since_gc);
+  consing_until_gc += new_gc_threshold - gc_threshold;
+  gc_threshold = new_gc_threshold;
+  return consing_until_gc;
 }
 
 /* Watch changes to gc-cons-threshold.  */
@@ -5821,7 +5830,8 @@ watch_gc_cons_threshold (Lisp_Object symbol, Lisp_Object newval,
   intmax_t threshold;
   if (! (INTEGERP (newval) && integer_to_intmax (newval, &threshold)))
     return Qnil;
-  return bump_consing_until_gc (threshold, Vgc_cons_percentage);
+  bump_consing_until_gc (threshold, Vgc_cons_percentage);
+  return Qnil;
 }
 
 /* Watch changes to gc-cons-percentage.  */
@@ -5829,7 +5839,18 @@ static Lisp_Object
 watch_gc_cons_percentage (Lisp_Object symbol, Lisp_Object newval,
 			  Lisp_Object operation, Lisp_Object where)
 {
-  return bump_consing_until_gc (gc_cons_threshold, newval);
+  bump_consing_until_gc (gc_cons_threshold, newval);
+  return Qnil;
+}
+
+/* It may be time to collect garbage.  Recalculate consing_until_gc,
+   since it might depend on current usage, and do the garbage
+   collection if the recalculation says so.  */
+void
+maybe_garbage_collect (void)
+{
+  if (bump_consing_until_gc (gc_cons_threshold, Vgc_cons_percentage) < 0)
+    garbage_collect ();
 }
 
 /* Subroutine of Fgarbage_collect that does most of the work.  */
@@ -5841,7 +5862,6 @@ garbage_collect (void)
   bool message_p;
   ptrdiff_t count = SPECPDL_INDEX ();
   struct timespec start;
-  byte_ct tot_before = 0;
 
   eassert (weak_hash_tables == NULL);
 
@@ -5856,14 +5876,15 @@ garbage_collect (void)
   FOR_EACH_BUFFER (nextb)
     compact_buffer (nextb);
 
-  if (profiler_memory_running)
-    tot_before = total_bytes_of_live_objects ();
+  byte_ct tot_before = (profiler_memory_running
+			? total_bytes_of_live_objects ()
+			: (byte_ct) -1);
 
   start = current_timespec ();
 
   /* In case user calls debug_print during GC,
      don't let that cause a recursive GC.  */
-  consing_until_gc = INTMAX_MAX;
+  consing_until_gc = HI_THRESHOLD;
 
   /* Save what's currently displayed in the echo area.  Don't do that
      if we are GC'ing because we've run out of memory, since
@@ -5975,7 +5996,7 @@ garbage_collect (void)
   unblock_input ();
 
   consing_until_gc = gc_threshold
-    = consing_threshold (gc_cons_threshold, Vgc_cons_percentage);
+    = consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0);
 
   if (garbage_collection_messages && NILP (Vmemory_full))
     {
@@ -6008,11 +6029,11 @@ garbage_collect (void)
   gcs_done++;
 
   /* Collect profiling data.  */
-  if (profiler_memory_running)
+  if (tot_before != (byte_ct) -1)
     {
       byte_ct tot_after = total_bytes_of_live_objects ();
-      byte_ct swept = tot_before <= tot_after ? 0 : tot_before - tot_after;
-      malloc_probe (min (swept, SIZE_MAX));
+      if (tot_after < tot_before)
+	malloc_probe (min (tot_before - tot_after, SIZE_MAX));
     }
 }
 
diff --git a/src/lisp.h b/src/lisp.h
index 024e5edb26..02f8a7b668 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3824,9 +3824,10 @@ extern void mark_maybe_objects (Lisp_Object const *, ptrdiff_t);
 extern void mark_stack (char const *, char const *);
 extern void flush_stack_call_func (void (*func) (void *arg), void *arg);
 extern void garbage_collect (void);
+extern void maybe_garbage_collect (void);
 extern const char *pending_malloc_warning;
 extern Lisp_Object zero_vector;
-extern intmax_t consing_until_gc;
+extern EMACS_INT consing_until_gc;
 #ifdef HAVE_PDUMPER
 extern int number_finalizers_run;
 #endif
@@ -5056,7 +5057,7 @@ INLINE void
 maybe_gc (void)
 {
   if (consing_until_gc < 0)
-    garbage_collect ();
+    maybe_garbage_collect ();
 }
 
 INLINE_HEADER_END
-- 
2.17.1

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> Cc: jrm@ftfl.ca, mattiase@acm.org, 37006@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 14 Sep 2019 00:51:18 -0700
> 
> On 8/15/19 7:17 AM, Eli Zaretskii wrote:
> > OK, but I still hope we will switch to EMACS_INT instead.
> 
> I installed the attached patch into master, and it changes the relevant 
> threshold variables to EMACS_INT, as part of a somewhat-larger cleanup.

Thanks.

> This should make a practical difference only on 32-bit Emacs without
> --with-wide-int.

Right.

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Joseph Mingrone]

After 26de2d42 from 2019-06-20, garbage collection is not happening and
allocated memory continues to grow until

Aug 10 21:39:02 <kern.err> phe kernel: pid 73800 (emacs-27.0.50), uid
1001, was killed: out of swap space

and

Aug 10 21:39:07 <kern.crit> phe kernel: swap_pager_getswapspace(3):
failed

In GNU Emacs 27.0.50 (build 1, amd64-portbld-freebsd12.0, GTK+ Version 3.24.10)
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: 12.0-RELEASE-p9

Configured using:
 'configure --disable-build-details --localstatedir=/var --with-x
 --enable-acl --without-cairo --without-dbus --without-gconf --with-gif
 --with-gnutls --without-gsettings --with-x-toolkit=gtk3
 --without-harfbuzz --with-jpeg --with-json
 --with-file-notification=kqueue --with-lcms2 --with-m17n-flt
 --with-imagemagick --with-mailutils --with-modules --with-sound=oss
 --with-libotf --with-png --with-toolkit-scroll-bars --with-rsvg
 --with-threads --with-tiff --with-xft --with-xim --with-xml2 --with-xpm
 --without-xwidgets --x-libraries=/usr/local/lib
 --x-includes=/usr/local/include --prefix=/usr/local
 --mandir=/usr/local/man --disable-silent-rules
 --infodir=/usr/local/share/emacs/info/
 --build=amd64-portbld-freebsd12.0 'CFLAGS=-O2 -pipe
 -fstack-protector-strong -isystem /usr/local/include
 -fno-strict-aliasing ' 'CPPFLAGS=-isystem /usr/local/include' 'LDFLAGS=
 -fstack-protector-strong -L/usr/local/lib ''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GLIB NOTIFY KQUEUE ACL
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 XDBE XIM MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Message

Minor modes in effect:
  gnus-message-citation-mode: t
  erc-spelling-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-netsplit-mode: t
  erc-menu-mode: t
  erc-log-mode: t
  erc-list-mode: t
  erc-pcomplete-mode: t
  erc-button-mode: t
  erc-stamp-mode: t
  erc-autojoin-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-match-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  mml-mode: t
  shell-dirtrack-mode: t
  flyspell-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  which-key-mode: t
  show-paren-mode: t
  savehist-mode: t
  global-company-mode: t
  company-mode: t
  global-auto-revert-mode: t
  counsel-mode: t
  ivy-mode: t
  cl-old-struct-compat-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
/home/jrm/.emacs.d/elpa/slime-20190724.1352/slime-autoloads hides /usr/local/share/emacs/27.0.50/site-lisp/slime/slime-autoloads
/home/jrm/.emacs.d/elpa/slime-20190724.1352/slime hides /usr/local/share/emacs/27.0.50/site-lisp/slime/slime
/home/jrm/.emacs.d/elpa/slime-20190724.1352/slime-tests hides /usr/local/share/emacs/27.0.50/site-lisp/slime/slime-tests
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-org hides /usr/local/share/emacs/27.0.50/lisp/org/ox-org
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-bibtex hides /usr/local/share/emacs/27.0.50/lisp/org/org-bibtex
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ruby hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ruby
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-table hides /usr/local/share/emacs/27.0.50/lisp/org/ob-table
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-awk hides /usr/local/share/emacs/27.0.50/lisp/org/ob-awk
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-element hides /usr/local/share/emacs/27.0.50/lisp/org/org-element
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ref hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ref
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-processing hides /usr/local/share/emacs/27.0.50/lisp/org/ob-processing
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ditaa hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ditaa
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-info hides /usr/local/share/emacs/27.0.50/lisp/org/org-info
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-md hides /usr/local/share/emacs/27.0.50/lisp/org/ox-md
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-colview hides /usr/local/share/emacs/27.0.50/lisp/org/org-colview
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-mhe hides /usr/local/share/emacs/27.0.50/lisp/org/org-mhe
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-python hides /usr/local/share/emacs/27.0.50/lisp/org/ob-python
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-shell hides /usr/local/share/emacs/27.0.50/lisp/org/ob-shell
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-ascii hides /usr/local/share/emacs/27.0.50/lisp/org/ox-ascii
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-lisp hides /usr/local/share/emacs/27.0.50/lisp/org/ob-lisp
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-capture hides /usr/local/share/emacs/27.0.50/lisp/org/org-capture
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-js hides /usr/local/share/emacs/27.0.50/lisp/org/ob-js
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-src hides /usr/local/share/emacs/27.0.50/lisp/org/org-src
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-sass hides /usr/local/share/emacs/27.0.50/lisp/org/ob-sass
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-footnote hides /usr/local/share/emacs/27.0.50/lisp/org/org-footnote
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-lob hides /usr/local/share/emacs/27.0.50/lisp/org/ob-lob
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-gnus hides /usr/local/share/emacs/27.0.50/lisp/org/org-gnus
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-picolisp hides /usr/local/share/emacs/27.0.50/lisp/org/ob-picolisp
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-scheme hides /usr/local/share/emacs/27.0.50/lisp/org/ob-scheme
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-mobile hides /usr/local/share/emacs/27.0.50/lisp/org/org-mobile
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-latex hides /usr/local/share/emacs/27.0.50/lisp/org/ob-latex
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-crypt hides /usr/local/share/emacs/27.0.50/lisp/org/org-crypt
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-docview hides /usr/local/share/emacs/27.0.50/lisp/org/org-docview
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob hides /usr/local/share/emacs/27.0.50/lisp/org/ob
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-icalendar hides /usr/local/share/emacs/27.0.50/lisp/org/ox-icalendar
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-rmail hides /usr/local/share/emacs/27.0.50/lisp/org/org-rmail
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-publish hides /usr/local/share/emacs/27.0.50/lisp/org/ox-publish
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-sed hides /usr/local/share/emacs/27.0.50/lisp/org/ob-sed
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-datetree hides /usr/local/share/emacs/27.0.50/lisp/org/org-datetree
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-protocol hides /usr/local/share/emacs/27.0.50/lisp/org/org-protocol
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-matlab hides /usr/local/share/emacs/27.0.50/lisp/org/ob-matlab
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-inlinetask hides /usr/local/share/emacs/27.0.50/lisp/org/org-inlinetask
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-shen hides /usr/local/share/emacs/27.0.50/lisp/org/ob-shen
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-core hides /usr/local/share/emacs/27.0.50/lisp/org/ob-core
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-entities hides /usr/local/share/emacs/27.0.50/lisp/org/org-entities
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ebnf hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ebnf
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-coq hides /usr/local/share/emacs/27.0.50/lisp/org/ob-coq
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-forth hides /usr/local/share/emacs/27.0.50/lisp/org/ob-forth
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox hides /usr/local/share/emacs/27.0.50/lisp/org/ox
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-hledger hides /usr/local/share/emacs/27.0.50/lisp/org/ob-hledger
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-abc hides /usr/local/share/emacs/27.0.50/lisp/org/ob-abc
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org hides /usr/local/share/emacs/27.0.50/lisp/org/org
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-archive hides /usr/local/share/emacs/27.0.50/lisp/org/org-archive
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-J hides /usr/local/share/emacs/27.0.50/lisp/org/ob-J
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-macs hides /usr/local/share/emacs/27.0.50/lisp/org/org-macs
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-version hides /usr/local/share/emacs/27.0.50/lisp/org/org-version
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-exp hides /usr/local/share/emacs/27.0.50/lisp/org/ob-exp
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-ctags hides /usr/local/share/emacs/27.0.50/lisp/org/org-ctags
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-java hides /usr/local/share/emacs/27.0.50/lisp/org/ob-java
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-sql hides /usr/local/share/emacs/27.0.50/lisp/org/ob-sql
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-lint hides /usr/local/share/emacs/27.0.50/lisp/org/org-lint
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-keys hides /usr/local/share/emacs/27.0.50/lisp/org/ob-keys
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-eshell hides /usr/local/share/emacs/27.0.50/lisp/org/org-eshell
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-plantuml hides /usr/local/share/emacs/27.0.50/lisp/org/ob-plantuml
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ocaml hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ocaml
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-sqlite hides /usr/local/share/emacs/27.0.50/lisp/org/ob-sqlite
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-list hides /usr/local/share/emacs/27.0.50/lisp/org/org-list
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-lua hides /usr/local/share/emacs/27.0.50/lisp/org/ob-lua
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-C hides /usr/local/share/emacs/27.0.50/lisp/org/ob-C
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-habit hides /usr/local/share/emacs/27.0.50/lisp/org/org-habit
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-bbdb hides /usr/local/share/emacs/27.0.50/lisp/org/org-bbdb
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-lilypond hides /usr/local/share/emacs/27.0.50/lisp/org/ob-lilypond
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-latex hides /usr/local/share/emacs/27.0.50/lisp/org/ox-latex
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-asymptote hides /usr/local/share/emacs/27.0.50/lisp/org/ob-asymptote
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-groovy hides /usr/local/share/emacs/27.0.50/lisp/org/ob-groovy
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-screen hides /usr/local/share/emacs/27.0.50/lisp/org/ob-screen
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-org hides /usr/local/share/emacs/27.0.50/lisp/org/ob-org
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-eww hides /usr/local/share/emacs/27.0.50/lisp/org/org-eww
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-octave hides /usr/local/share/emacs/27.0.50/lisp/org/ob-octave
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-vala hides /usr/local/share/emacs/27.0.50/lisp/org/ob-vala
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-indent hides /usr/local/share/emacs/27.0.50/lisp/org/org-indent
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-macro hides /usr/local/share/emacs/27.0.50/lisp/org/org-macro
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-gnuplot hides /usr/local/share/emacs/27.0.50/lisp/org/ob-gnuplot
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-haskell hides /usr/local/share/emacs/27.0.50/lisp/org/ob-haskell
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-dot hides /usr/local/share/emacs/27.0.50/lisp/org/ob-dot
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-html hides /usr/local/share/emacs/27.0.50/lisp/org/ox-html
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-calc hides /usr/local/share/emacs/27.0.50/lisp/org/ob-calc
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-fortran hides /usr/local/share/emacs/27.0.50/lisp/org/ob-fortran
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-stan hides /usr/local/share/emacs/27.0.50/lisp/org/ob-stan
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-ledger hides /usr/local/share/emacs/27.0.50/lisp/org/ob-ledger
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-io hides /usr/local/share/emacs/27.0.50/lisp/org/ob-io
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-install hides /usr/local/share/emacs/27.0.50/lisp/org/org-install
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-clock hides /usr/local/share/emacs/27.0.50/lisp/org/org-clock
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-agenda hides /usr/local/share/emacs/27.0.50/lisp/org/org-agenda
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-table hides /usr/local/share/emacs/27.0.50/lisp/org/org-table
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-attach hides /usr/local/share/emacs/27.0.50/lisp/org/org-attach
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-faces hides /usr/local/share/emacs/27.0.50/lisp/org/org-faces
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-beamer hides /usr/local/share/emacs/27.0.50/lisp/org/ox-beamer
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-comint hides /usr/local/share/emacs/27.0.50/lisp/org/ob-comint
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-mscgen hides /usr/local/share/emacs/27.0.50/lisp/org/ob-mscgen
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-w3m hides /usr/local/share/emacs/27.0.50/lisp/org/org-w3m
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-maxima hides /usr/local/share/emacs/27.0.50/lisp/org/ob-maxima
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-css hides /usr/local/share/emacs/27.0.50/lisp/org/ob-css
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-compat hides /usr/local/share/emacs/27.0.50/lisp/org/org-compat
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-feed hides /usr/local/share/emacs/27.0.50/lisp/org/org-feed
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-irc hides /usr/local/share/emacs/27.0.50/lisp/org/org-irc
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-clojure hides /usr/local/share/emacs/27.0.50/lisp/org/ob-clojure
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-odt hides /usr/local/share/emacs/27.0.50/lisp/org/ox-odt
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-man hides /usr/local/share/emacs/27.0.50/lisp/org/ox-man
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-emacs-lisp hides /usr/local/share/emacs/27.0.50/lisp/org/ob-emacs-lisp
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-tangle hides /usr/local/share/emacs/27.0.50/lisp/org/ob-tangle
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-R hides /usr/local/share/emacs/27.0.50/lisp/org/ob-R
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-makefile hides /usr/local/share/emacs/27.0.50/lisp/org/ob-makefile
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-duration hides /usr/local/share/emacs/27.0.50/lisp/org/org-duration
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ox-texinfo hides /usr/local/share/emacs/27.0.50/lisp/org/ox-texinfo
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-id hides /usr/local/share/emacs/27.0.50/lisp/org/org-id
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-eval hides /usr/local/share/emacs/27.0.50/lisp/org/ob-eval
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-timer hides /usr/local/share/emacs/27.0.50/lisp/org/org-timer
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-mouse hides /usr/local/share/emacs/27.0.50/lisp/org/org-mouse
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-pcomplete hides /usr/local/share/emacs/27.0.50/lisp/org/org-pcomplete
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-loaddefs hides /usr/local/share/emacs/27.0.50/lisp/org/org-loaddefs
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/org-plot hides /usr/local/share/emacs/27.0.50/lisp/org/org-plot
/home/jrm/.emacs.d/elpa/org-plus-contrib-20190805/ob-perl hides /usr/local/share/emacs/27.0.50/lisp/org/ob-perl
/home/jrm/.emacs.d/elpa/let-alist-1.0.6/let-alist hides /usr/local/share/emacs/27.0.50/lisp/emacs-lisp/let-alist

Features:
(shadow emacsbug bbdb-message sendmail nnir bbdb-mua bbdb-com crm sort
gnus-cite mail-extr gnus-async gnus-bcklg qp gnus-ml disp-table
ace-window cl-print help-fns radix-tree make-mode tramp-cache tramp-sh
bookmark mm-archive mule-util url-http url-gw url-cache url-auth url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util finder-inf gnutls network-stream nsm erc-spelling
erc-ring erc-networks erc-netsplit erc-menu erc-log erc-list
erc-pcomplete erc-button erc-fill erc-stamp erc-join znc warnings
erc-track erc-match erc-tex easy-mmode erc-goodies erc erc-backend
erc-compat pp erc-loaddefs cl hl-line gnus-topic nndraft nnmh nnagent
nnml gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg
gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap nntp
gnus-cache gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range message
rmc puny rfc822 mml mml-sec epa derived epg mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win
gnus nnheader wid-edit gnus-util rmail rmail-loaddefs rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils text-property-search time-date
tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell pcomplete parse-time ls-lisp format-spec rx server flyspell ispell
undo-tree diff smart-mode-line-dark-theme smart-mode-line rich-minority
s pdf-tools-loaddefs misc hydra lv ace-link avy bbdb bbdb-site timezone
company-oddmuse company-keywords company-etags etags fileloop generator
company-gtags company-dabbrev-code company-dabbrev company-files
company-capf company-cmake company-xcode company-clang company-semantic
company-eclim company-template company-bbdb which-key paren savehist
company edmacro kmacro pcase autorevert filenotify counsel xdg xref
project dired dired-loaddefs compile comint ansi-color swiper cl-extra
help-mode ivy delsel ring colir color ivy-overlay ffap thingatpt
cus-start cus-load benchmark-init benchmark-init-loaddefs tex-site
advice slime-autoloads info package easymenu epg-config url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads kqueue lcms2 dynamic-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 4672710 1249112)
 (symbols 48 33763 2)
 (strings 32 177500 153301)
 (string-bytes 1 6559318)
 (vectors 16 108474)
 (vector-slots 8 4931122 2331820)
 (floats 8 565 3210)
 (intervals 56 12670 13121)
 (buffers 992 90))
<#secure method=pgpmime mode=sign>

bug#37006: 27.0.50; garbage collection not happening after 26de2d42

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Date: Sun, 11 Aug 2019 09:39:58 -0300
> 
> After 26de2d42 from 2019-06-20, garbage collection is not happening and
> allocated memory continues to grow until
> 
> Aug 10 21:39:02 <kern.err> phe kernel: pid 73800 (emacs-27.0.50), uid
> 1001, was killed: out of swap space
> 
> and
> 
> Aug 10 21:39:07 <kern.crit> phe kernel: swap_pager_getswapspace(3):
> failed

I see this on GNU/Linux, but not on MS-Windows, FWIW.