[bug #34140] gc not run correctly when allocating threads

[bug #34140] gc not run correctly when allocating threads

[Stefan Israelsson Tampe]

URL:
  <http://savannah.gnu.org/bugs/?34140>

                 Summary: gc not run correctly when allocating threads
                 Project: Guile
            Submitted by: tampe
            Submitted on: Fri 26 Aug 2011 08:49:53 AM GMT
                Category: None
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any

    _______________________________________________________

Details:

On my machine this crashes after about 4000 iterations due to the oom killer.
The reason is mainly that stack space is not reclaimed.

(define (f n)
  (let ((t (call-with-new-thread (lambda () 'ok))))
    (join-thread t))
  (if (= (modulo n 30) 0) 
     (begin
        (pk n) 
        (pk (gc-stats)) 
        (sleep 1)))
  (f (+ n 1)))

  (f 0)

On the other hand this code works perfectly (a explicit call to gc is used)
(define (f n)

  (let ((t (call-with-new-thread (lambda () 'ok))))
    (join-thread t))
  (if (= (modulo n 10) 0) (pk (gc-stats)))
  (if (= (modulo n 300) 0) (begin (pk n)  (gc) (sleep 1)))
  (f (+ n 1)))

  (f 0)

It would be nice if the spawning of threads checked to gc stats
to run the gc appropriatly if there is not so much space left to allow for a
new thread.

/Stefan




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?34140>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

[bug #34140] gc not run correctly when allocating threads

[Stefan Israelsson Tampe]

Follow-up Comment #1, bug #34140 (project guile):

When running the script below with explicit gc, one notice an avarage leak of
350 bytes per thread allocation. The origin of this
is kind of strange because many times the gc-stats are identical from run to
run indicating that the leak is not systematic. 

/Stefan

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?34140>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/

Re: [bug #34140] gc not run correctly when allocating threads

[Andy Wingo]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

Hi,

On Fri 26 Aug 2011 10:49, Stefan Israelsson Tampe <INVALID.NOREPLY@gnu.org> writes:

> On my machine this crashes after about 4000 iterations due to the oom killer.
> The reason is mainly that stack space is not reclaimed.
>
> (define (f n)
>   (let ((t (call-with-new-thread (lambda () 'ok))))
>     (join-thread t))
>   (if (= (modulo n 30) 0) 
>      (begin
>         (pk n) 
>         (pk (gc-stats)) 
>         (sleep 1)))
>   (f (+ n 1)))
>
>   (f 0)

Why is this, I wonder?  (Also, how much memory do you have, and what is
your architecture?  On my system each thread will allocate 512 KB by
default for a stack; if these thread stacks are not being collected,
4000 of them is 2 GB, which could hit the OOM on many systems.)

Anyway I could not reproduce it with the attached C program.  But the C
program isn't quite like what we do with threads, either; I suppose we
could try approximating the C program to the Scheme program.  We should
not have to introduce calls to `gc'; if the collector isn't collecting,
that's a problem!

Andy


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gcc -o test test.c `pkg-config --cflags --libs bdw-gc` -lpthread -Wall --]
[-- Type: text/x-csrc, Size: 955 bytes --]

#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <malloc.h>

#define GC_THREADS 1
#define GC_REDIRECT_TO_LOCAL 1
#define GC_NO_THREAD_REDIRECTS 1

#include <gc/gc.h>

#define ALLOC_SIZE (64 * 1024 * sizeof(void*))

static void
fail (char *reason)
{
  perror (reason);
  exit (1);
}

static void*
thr_func (void* arg)
{
  void *mem = GC_malloc (ALLOC_SIZE);
  if (!mem)
    fail ("malloc failed");

  memset (mem, 1, ALLOC_SIZE);

  return NULL;
}

int main (int argc, char *argv[])
{
  GC_INIT ();
  
  int i, err, limit;
  void *res = NULL;

  limit = (argc == 2 ? atoi (argv[1]) : 10000);
  
  for (i = 0; i < limit; i++)
    {
      pthread_t thr;

      printf ("Spawning %d\n", i);

      if ((err = GC_pthread_create (&thr, NULL, thr_func, NULL)))
        fail ("pthread_create");

      if ((err = GC_pthread_join (thr, &res)))
        fail ("pthread_join");
    }

  printf ("Success.\n");  

  return 0;
}

[-- Attachment #3: Type: text/plain, Size: 26 bytes --]


-- 
http://wingolog.org/

Re: [bug #34140] gc not run correctly when allocating threads

[Andy Wingo]

On Sun 04 Sep 2011 18:36, Stefan Israelsson Tampe <stefan.itampe@gmail.com> writes:

> Andy! test.c is a misstake from my side. It should not be attatched. It is the
> scheme programs without explicit gc that causes trouble.

I wrote test.c, at least the one that I attached.  I was trying to write
an equivalent C program that would trigger the bug, but obviously
didn't.  I can confirm your bug report regarding the Scheme program.

Andy
-- 
http://wingolog.org/