MMCI 1.1.1 available

MMCI 1.1.1 available

[Thien-Thi Nguyen]

release notes:

  maintenance release:
  tools like birds, files like small worms.
  M-x v3!

  thi

README excerpt:

  This directory contains MMCI (multi-method check-in), a program
  that abstracts the "cvs add" procedure to also handle RCS files
  and Emacs-style backup files.

NEWS excerpt:

  - 1.1.1 | 2007-08-24
  
    - License now GPLv3+
  
    - Maintenance uses GNU Autoconf 2.60

tarball, prettified source, small buffer-thumbnails, etc, in dir:

  http://www.gnuvola.org/software/mmci/


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: What is best way to limit memory alloc?

[Roland Orre]

I found out that I had already solved the memory allocation
problem in one way a few years ago.
With the help of a small routine gc-heap-size, which accesses 
scm_i_master_freelist.heap_size
scm_i_master_freelist2.heap_size
I did:
(define gc-heap1 (gc-heap-size 1))
(define gc-heap2 (gc-heap-size 2))
(let loop
   ....
   (gc-heap-size 1 gc-heap1)
   (gc-heap-size 2 gc-heap2)
    (loop ...))
By not allowing the heap size to increase.
Then it worked, the memory just increased to
7.1 GB instead of 8.8 GB and it was then stable there during
the whole run. Is there any other better way to solve this?
I would like to be able to say something like:
don't allocate any memory above nn GB otherwise abort,
as a swapping guile process is just dead anyway.
The job now took just 19 minutes to finish, the other one
which is still swapping, has been running since Monday
(52 hours) and is just half ready...

/Roland

On Wed, 2007-08-29 at 08:13 +0200, Roland Orre wrote:
> What is the best way to limit the memory allocation in guile?
> 
> I have a process that has allocated 6GB when the first step
> is finished, this is much, but is fine as the machine has 8 GB.
> 
> Then when performing the last step, just involving reading
> from a file, doing some string split for each line, adding
> some data and writing it back to the file line per line, the
> memory suddenly increases to 8.8 GB and then the job starts
> swapping ridicously and stops of course.
> 
> It seems as the memory allocator in guile doesn't care about
> the physical memory limits and it seems as it prefers to
> allocate big chunks of more memory instead of performing gc
> despite very little extra memory should be needed.
> 
> I'm still running 1.7 as I haven't got the time and energy
> to change the array implementation yet.
> 
> I assume I have to change something in gc.c, the simplest I
> would like to say is, don't allocate more memory than this!
> 
> /Roland Orre
> 
> 



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: What is best way to limit memory alloc?

[Ludovic Courtès]

Hi,

Roland Orre <roland.orre@neurologic.se> writes:

> What is the best way to limit the memory allocation in guile?

First, make sure Guile's GC knows about every bit of memory that you
allocate.  This means that C code run from your Guile program must use
`scm_gc_malloc ()' et al.

If you don't do this, Guile's GC will think that less memory is used
than what is actually used, so it will GC less often than needed.

Also, make sure to compile Guile with `-O2', otherwise it is more likely
that references in the stack to Scheme objects will be visible much
longer than needed.

Similarly, make sure your Scheme code doesn't unduly keeps references to
objects that are no longer needed.

> Then when performing the last step, just involving reading
> from a file, doing some string split for each line, adding
> some data and writing it back to the file line per line, the
> memory suddenly increases to 8.8 GB and then the job starts
> swapping ridicously and stops of course.

Of course, there could also be a bug.  :-)

If you get convinced that this is the case, you can try to reproduce it
with a simple test case (typically: run the suspicious code in an
endless loop and see what happens).

> I'm still running 1.7 as I haven't got the time and energy
> to change the array implementation yet.

The array API hasn't changed between 1.7.2 and 1.8.x, so you could
hopefully upgrade with little troubles.

Thanks,
Ludovic.


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: What is best way to limit memory alloc?

[Ludovic Courtès]

Roland Orre <roland.orre@neurologic.se> writes:

> I found out that I had already solved the memory allocation
> problem in one way a few years ago.
> With the help of a small routine gc-heap-size, which accesses 
> scm_i_master_freelist.heap_size
> scm_i_master_freelist2.heap_size
> I did:
> (define gc-heap1 (gc-heap-size 1))
> (define gc-heap2 (gc-heap-size 2))
> (let loop
>    ....
>    (gc-heap-size 1 gc-heap1)
>    (gc-heap-size 2 gc-heap2)
>     (loop ...))
> By not allowing the heap size to increase.

That seems brutal. ;-)  What does `gc-heap-size' do exactly?

Another way would have been to fiddle with the `GUILE_MIN_YIELD_1' and
`GUILE_MIN_YIELD_2' environment variables.  These variables tell the GC
when it should grow the heap for the first and second freelist,
respectively.  More precisely, the GC does:

  if (number-of-cells-collected-recently < GUILE_MIN_YIELD_X)
  then
    allocate-new-heap
  else
    run-a-collection

(This takes place in `scm_i_gc_grow_heap_p ()' and `scm_gc_for_newcell ()'.)

The default value for `GUILE_MIN_YIELD_{1,2}' is 40, which means that if
the last GC run did not yield more than 40 cells, then more heap is
allocated.

If you set it to some _higher_ value, then the GC should be more
conservative and less memory-hungry, at the cost of being slower.

Thanks,
Ludovic.


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: What is best way to limit memory alloc?

[Kevin Ryde]

Roland Orre <roland.orre@neurologic.se> writes:
>
>> What is the best way to limit the memory allocation in guile?

Perhaps setrlimit would be the most reliable overall.

>> I'm still running 1.7 as I haven't got the time and energy
>> to change the array implementation yet.

I struck a bug lately in 1.8 where the collected cell counts are somehow
botched, leading to it wrongly thinking new heap is needed again and
again.

>> It seems as the memory allocator in guile doesn't care about
>> the physical memory limits and it seems as it prefers to
>> allocate big chunks of more memory instead of performing gc
>> despite very little extra memory should be needed.

It's not supposed to.  The rule is supposed to be to increase the heap
to make 40% of it free, after gc figures what's free and what's not, or
something like that.  In practice it means blocks each 1.6x (or so)
bigger than the one before getting allocated.  (As seen in `gc-stats'.)


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user