Increase gc-cons-threshold during package-init?

Increase gc-cons-threshold during package-init?

[T.V Raman]

With Emacs now calling package-initialize on its own, I noticed (after
removing package-initialize from my emacs init file)
that emacs startup was now  entailing 3 times as many gc calls during
startup as before. One guess is that my init file was lexically
binding gc-cons-threshold to 64MB --- the default gc-cons-threshold
of 800K is likely from the past and could be increased. 

Even if we leave the default as is, it might make sense to temporarily
increase gc-cons-threshold within package-initialize 

-- 

-- 

Re: Increase gc-cons-threshold during package-init?

[Daniel Colascione]

On 02/19/2018 09:23 AM, T.V Raman wrote:
> With Emacs now calling package-initialize on its own, I noticed (after
> removing package-initialize from my emacs init file)
> that emacs startup was now  entailing 3 times as many gc calls during
> startup as before. One guess is that my init file was lexically
> binding gc-cons-threshold to 64MB --- the default gc-cons-threshold
> of 800K is likely from the past and could be increased.

There's a tradeoff though: increasing gc-cons-threshold means that the 
GC that eventually occurs takes longer. I wish we had generational GC.

Re: Increase gc-cons-threshold during package-init?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 19 Feb 2018 09:27:36 -0800
> 
> I wish we had generational GC.

Yep.  That job is still up for grabs.  A few years ago, someone was
working towards a more modern GC, but that job was never finished,
although some changes in the infrastructure were done to make that
easier.

Re: Increase gc-cons-threshold during package-init?

[Stefan Monnier]

> With Emacs now calling package-initialize on its own,

Not sure what you mean by "now": it's been the case since Emacs-24.1
(recent changes on `master` have only moved the time at which it takes
place).

> that emacs startup was now  entailing 3 times as many gc calls during
> startup as before.  One guess is that my init file was lexically
> binding gc-cons-threshold to 64MB ---

Lexically-binding it would have no effect on the GC, and since that var
is declared as dynamically scoped, it was probably dynamically bound.

Have you checked how much extra time is spent in the GC because of those
extra calls?

> the default gc-cons-threshold of 800K is likely from the past and
> could be increased.

This is a misunderstanding: the real gc-cons-threshold is computed
dynamically based on the heap size (see gc-cons-percentage).

Maybe those values need to be tweaked because tradeoffs have shifted
somewhat, but it doesn't seem very likely.

The problem with the above GC calls is much more fundamental and
unrelated to the actual values we choose: during periods of
initialization, programs typically allocate a lot of memory without
generating much garbage.  So during those times running a GC tends to be
a waste of effort (we scan the whole heap only to find a few crumbs here
and there).

People often look at situations like yours and conclude that
gc-cons-threshold is too small.  Yes, it's too small for those cases,
but bumping it once and forall makes things worse for the "usual" case,
leading to increased heap size, fragmentation, and longer pauses during
normal operation (tho this "usual case" is much more difficult to
measure).

There's no sadly way the GC can magically know that the last N megabytes
of allocation generated very little garbage, tho.

Maybe should bump those gc-knobs around the startup.el code (or maybe
more generally we should bump those knobs while loading a file).

> Even if we leave the default as is, it might make sense to temporarily
> increase gc-cons-threshold within package-initialize

Right.  Tho doing this will likely cause a GC to happen right after
package-initialize, i.e. just before loading your ~/.emacs.
Better would be to bump it around both, I guess.


        Stefan

Re: Increase gc-cons-threshold during package-init?

[T.V Raman]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

By Now I meant "earlier that package-initialize was happening in my
startup file where the numbers were tweaked".

Agree to the rest of what you say -- the right thing might be to tweak
those during startup as a whole.>> With Emacs now calling package-initialize on its own,
>
> Not sure what you mean by "now": it's been the case since Emacs-24.1
> (recent changes on `master` have only moved the time at which it takes
> place).
>
>> that emacs startup was now  entailing 3 times as many gc calls during
>> startup as before.  One guess is that my init file was lexically
>> binding gc-cons-threshold to 64MB ---
>
> Lexically-binding it would have no effect on the GC, and since that var
> is declared as dynamically scoped, it was probably dynamically bound.
>
> Have you checked how much extra time is spent in the GC because of those
> extra calls?
>
>> the default gc-cons-threshold of 800K is likely from the past and
>> could be increased.
>
> This is a misunderstanding: the real gc-cons-threshold is computed
> dynamically based on the heap size (see gc-cons-percentage).
>
> Maybe those values need to be tweaked because tradeoffs have shifted
> somewhat, but it doesn't seem very likely.
>
> The problem with the above GC calls is much more fundamental and
> unrelated to the actual values we choose: during periods of
> initialization, programs typically allocate a lot of memory without
> generating much garbage.  So during those times running a GC tends to be
> a waste of effort (we scan the whole heap only to find a few crumbs here
> and there).
>
> People often look at situations like yours and conclude that
> gc-cons-threshold is too small.  Yes, it's too small for those cases,
> but bumping it once and forall makes things worse for the "usual" case,
> leading to increased heap size, fragmentation, and longer pauses during
> normal operation (tho this "usual case" is much more difficult to
> measure).
>
> There's no sadly way the GC can magically know that the last N megabytes
> of allocation generated very little garbage, tho.
>
> Maybe should bump those gc-knobs around the startup.el code (or maybe
> more generally we should bump those knobs while loading a file).
>
>> Even if we leave the default as is, it might make sense to temporarily
>> increase gc-cons-threshold within package-initialize
>
> Right.  Tho doing this will likely cause a GC to happen right after
> package-initialize, i.e. just before loading your ~/.emacs.
> Better would be to bump it around both, I guess.
>
>
>         Stefan
>
>

-- 

Re: Increase gc-cons-threshold during package-init?

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 210 bytes --]

> With Emacs now calling package-initialize on its own

If you need to run (package-initialize) with special settings in your init
file,
just set `package-enable-at-startup' to nil in ~/.emacs.d/early-init.el.

[-- Attachment #2: Type: text/html, Size: 426 bytes --]