Re: GUILE_MAX_HEAP_SIZE

[Han-Wen Nienhuys <hanwen@xs4all.nl>]

Han-Wen Nienhuys escreveu:
> Ludovic Courtès escreveu:
>> Hi Han-Wen,
>>
>> Han-Wen Nienhuys <hanwen@xs4all.nl> writes:
>>
>>> I've implemented an env var GUILE_MAX_HEAP_SIZE (- some lilypond users
>>> were complaining that running lily on large sets of files keeps
>>> growing the heap forever, leading to trashing).
>> Of course, the correct fix would be to help the GC be more reasonable,
>> as it's currently somewhat broken:
>>
>>   http://thread.gmane.org/gmane.lisp.guile.devel/6699/focus=6832
> 
>> Also, it gives me the impression that we've definitely lost control over
>> the GC code, and we're adding yet another configuration variable to work
>> around that.  :-(
> 
> Hi,
> 
> I did a cleanup round over the GC code - see the dev/hanwen branch on
> sv.gnu.org ; It seems to work ok, there is a just one annoyance: there
> is a commented out assert that works most of the time, but once in a
> while is off by a small amount in either direction.  Comments/insights
> appreciated.
> 
> Unfortunately, I am not the patient type, and also not aflush with
> free time, so the changes come as one big bunch, with layout,
> refactoring and some bugfixes all lumped together.
> 
> (I intend to squash into a single commit before pushing to master).
> 
> 
> 
> Garbage collection cleanup.
> 
> * Remove data that might be out of date; remove
>   scm_i_adjust_min_yield().  We don't store min_yields, since they
>   are only accurate at one point in time (when the sweep finishes). 
>   We decide the min yield at that point from min_yield_fraction and
>   freelist->collected / freelist->swept
> 
> * Introduce scm_i_gc_heap_size_delta() replacing
>   scm_i_gc_grow_heap_p().
>   
> * Remove foo_1 fields containing penultimate results.
> 
> * After GC, count mark bit vector to discover number of live
>   objects. This simplifies hairy updates.
> 
> * Many formatting and layout cleanups.
> 
> * Fix in scm_i_sweep_card(): return the length of free_list returned,
>   rather than number of deleted objects.
> 
> * For mtrigger GCs: do not also run a full sweep after the gc() call, as
>   this is inconsistent with lazy sweeping.
> 
> * Remove scm_i_make_initial_segment().
> 
> * Use calloc in scm_i_make_empty_heap_segment() to save on init's.
> 
> * New function scm_i_sweep_for_freelist() which sweeps, with proper
>   statistic variable updates.
> 
> * New segments are conceptually blocks with 100% reclaimable cells.
> 
> * Remove some useless constants/comments: SCM_HEAP_SIZE,
>   SCM_INIT_HEAP_SIZE, SCM_EXPHEAP, SCM_HEAP_SEG_SIZE
> 
> * Do not increment scm_cells_allocated() from the
>   scm_[double]cell(). This would be a race condition.
> 
> * Move some deprecation checks in separate functions to not distract
>   from main code flow.

Note that your 'stress' test runs 10k iterations in about 11mb constant 
RSS (13mb virtual)- this is consistent with its mem requirements; 
it uses 1.1M  cells which is ~8.8M of memory.

-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen