Re: out-of-control GC

[Mark H Weaver <mhw@netris.org>]

Linas Vepstas <linasvepstas@gmail.com> writes:
> The overall message seems to be that gc either runs too often, or not often
> enough.  Some sort of better planning is needed.

There several variables available to adjust libgc's heuristics, e.g.
GC_use_entire_heap, GC_free_space_divisor, GC_full_frequency, with
associated environment variables, described in:
<https://github.com/ivmai/bdwgc/blob/master/doc/README.environment>

> I'm motivated to explore this, as it actually impacts my work.

If you'd like to learn how Boehm GC decides when to run, I would
encourage you to read <http://www.hboehm.info/gc/gcdescr.html>.  I've
selected some relevant excerpts below.

      Mark


http://www.hboehm.info/gc/gcdescr.html

[...]
  In non-incremental mode, we make a decision about whether to garbage
  collect whenever an allocation would otherwise have failed with the
  current heap size. If the total amount of allocation since the last
  collection is less than the heap size divided by GC_free_space_divisor,
  we try to expand the heap. Otherwise, we initiate a garbage
  collection. This ensures that the amount of garbage collection work per
  allocated byte remains constant.
  
  The above is in fact an oversimplification of the real heap expansion
  and GC triggering heuristic, which adjusts slightly for root size and
  certain kinds of fragmentation. In particular:
  
  * Programs with a large root set size and little live heap memory will
    expand the heap to amortize the cost of scanning the roots.
  
  * Versions 5.x of the collector actually collect more frequently in
    nonincremental mode. The large block allocator usually refuses to
    split large heap blocks once the garbage collection threshold is
    reached. This often has the effect of collecting well before the heap
    fills up, thus reducing fragmentation and working set size at the
    expense of GC time. Versions 6.x choose an intermediate strategy
    depending on how much large object allocation has taken place in the
    past. (If the collector is configured to unmap unused pages, versions
    6.x use the 5.x strategy.)
  
  * In calculating the amount of allocation since the last collection we
    give partial credit for objects we expect to be explicitly
    deallocated. Even if all objects are explicitly managed, it is often
    desirable to collect on rare occasion, since that is our only
    mechanism for coalescing completely empty chunks.
  
  It has been suggested that this should be adjusted so that we favor
  expansion if the resulting heap still fits into physical memory. In many
  cases, that would no doubt help. But it is tricky to do this in a way
  that remains robust if multiple application are contending for a single
  pool of physical memory.

[...]

  In incremental mode, the heap is always expanded when we encounter
  insufficient space for an allocation. Garbage collection is triggered
  whenever we notice that more than GC_heap_size/2 *
  GC_free_space_divisor bytes of allocation have taken place. After
  GC_full_freq minor collections a major collection is started.