bug: defining things in different threads

bug: defining things in different threads

[Linas Vepstas]

Hi,

I am finding that things defined in one thread are not always
visible in another.  This seems to be due to some threads
having a different current-module than others.  I think this is a bug.
The example code below shows the issue.  Help, comments
appreciated.

--linas

/**
 * Guile threading bug/unexpected behaviour.
 *
 * Two posix threads are created.  The first thread runs, and defines
 * something. The second thread tries to access the thing defined by
 * the first. But, due to some "unexpected" threading behaviour, the
 * things defined in the first thread are not visible to the first.
 * I'm pretty convinced this is a bug, and am hunting it down now.
 *
 * The printed output of this program, for me, under guile1.8.5, is
 *
 * HEllo, thread one is running
 * Goodbye, thread one is exiting
 * HEllo, thread two is running
 * ERROR: Unbound variable: x
 * Goodbye, thread two is exiting
 * Main is exiting
 *
 * To get a better idea of what's going on, it seems to have something
 * to do with the current module. By turning on print debugging, the
 * output, for me, becomes:
 *
 * HEllo, thread one is running
 * thread one had output:
 * the root module is #<module (guile) f73f9e80>
 * the current module is #<directory (guile-user) f73fc600>
 *
 * Goodbye, thread one is exiting
 * HEllo, thread two is running
 * thread two had output:
 * the root module is #<module (guile) f73f9e80>
 * the current module is #<module (guile) f73f9e80>
 *
 * ERROR: Unbound variable: x
 * Goodbye, thread two is exiting
 * Main is exiting
 */


#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <libguile.h>

void prtdbg(const char * id)
{
// #define SHOW_STUFF 1
#ifdef SHOW_STUFF
	SCM outport = scm_open_output_string();
	scm_set_current_output_port(outport);

	scm_c_eval_string ("(display \"the root module is \")\n");
	scm_c_eval_string ("(display the-root-module)\n");
	scm_c_eval_string ("(newline)\n");
	scm_c_eval_string ("(display \"the current module is \")\n");
	scm_c_eval_string ("(display (current-module))\n");
	scm_c_eval_string ("(newline)\n");

   SCM out = scm_get_output_string(outport);
   char * str = scm_to_locale_string(out);
   // scm_truncate_file(outport, scm_from_uint16(0));

   printf("%s had output:\n%s\n", id, str);
#endif
}

void * scm_one (void *p)
{
	prtdbg("thread one");
	scm_c_eval_string ("(define x \"asddf\")\n");
}

void * scm_two (void *p)
{
	prtdbg("thread two");
	scm_c_eval_string ("(display x)\n");
}

static void * thread_one (void * arg)
{
	printf("HEllo, thread one is running\n");

	scm_with_guile(scm_one, NULL);

	printf("Goodbye, thread one is exiting\n");
	return NULL;
}

static void * thread_two (void * arg)
{
	printf("HEllo, thread two is running\n");
	scm_with_guile(scm_two, NULL);
	printf("Goodbye, thread two is exiting\n");
	return NULL;
}

main()
{
	int rc;
	pthread_attr_t attr;
	pthread_t t1, t2;

	pthread_attr_init(&attr);

	rc = pthread_create(&t1, &attr, thread_one, NULL);
	if(rc)
	{
		fprintf(stderr, "Fatal error: can't create thread one\n");
		exit(1);
	}
	sleep(1);
	rc = pthread_create(&t2, &attr, thread_two, NULL);
	if(rc)
	{
		fprintf(stderr, "Fatal error: can't create thread two\n");
		exit(1);
	}

	sleep(5);
	printf("Main is exiting\n");
}

Re: bug: defining things in different threads

[Ludovic Courtès]

Hi Linas,

"Linas Vepstas" <linasvepstas@gmail.com> writes:

> void * scm_one (void *p)
> {
> 	prtdbg("thread one");
> 	scm_c_eval_string ("(define x \"asddf\")\n");
> }
>
> void * scm_two (void *p)
> {
> 	prtdbg("thread two");
> 	scm_c_eval_string ("(display x)\n");
> }

AFAICS, there's not proper synchronization between these two threads
(e.g., a barrier), so it could be that thread 1 gets to define `x' after
thread 2 attempts to access it.  Sure, there's a "sleep (1);", but a
barrier would make sure we're not seeing such a case.

Can you add proper synchronization and see if the problem is still here?

Thanks,
Ludo'.

Re: bug: defining things in different threads

[Linas Vepstas]

2008/11/12 Ludovic Courtès <ludo@gnu.org>:
> Hi Linas,
>
> "Linas Vepstas" <linasvepstas@gmail.com> writes:
>
>> void * scm_one (void *p)
>> {
>>       prtdbg("thread one");
>>       scm_c_eval_string ("(define x \"asddf\")\n");
>> }
>>
>> void * scm_two (void *p)
>> {
>>       prtdbg("thread two");
>>       scm_c_eval_string ("(display x)\n");
>> }
>
> AFAICS, there's not proper synchronization between these two threads
> (e.g., a barrier), so it could be that thread 1 gets to define `x' after
> thread 2 attempts to access it.  Sure, there's a "sleep (1);", but a
> barrier would make sure we're not seeing such a case.
>
> Can you add proper synchronization and see if the problem is still here?

Ludo,

I am sorry I mailed out such a lame example, but surely
you must be joking when you make such a request! The
answer is, of course, obvious; the sleep is a defacto
thread yield.  Program correctness is easily verified
simply by looking at the tty while the program is running.

You may replace the first sleep by pthread_join(t1, NULL);
and the second one by pthread_join(t2, NULL); to get
your desired posixly-correct synchronization.

Yes, of course, the problem remains.

--linas

Re: bug: defining things in different threads

[Neil Jerram]

2008/11/12 Linas Vepstas <linasvepstas@gmail.com>:

> Yes, of course, the problem remains.

Explanation:

Thread 1 is the first thread that does any Guile stuff, so it loads
boot-9.scm, which means that it ends up in the (guile-user) module.

Thread 2 hasn't done any (set-current-module ...), so it stays at the
default, which is (guile).

Solution: add (define-module (guile-user)) to the code that you
execute in thread 2.

Regards,
     Neil

Re: bug: defining things in different threads

[Linas Vepstas]

Hi Neil,

Thanks for the reply,

2008/11/15 Neil Jerram <neiljerram@googlemail.com>:
> 2008/11/12 Linas Vepstas <linasvepstas@gmail.com>:
>
>> Yes, of course, the problem remains.
>
> Explanation:
>
> Thread 1 is the first thread that does any Guile stuff, so it loads
> boot-9.scm, which means that it ends up in the (guile-user) module.
>
> Thread 2 hasn't done any (set-current-module ...), so it stays at the
> default, which is (guile).
>
> Solution: add (define-module (guile-user)) to the code that you
> execute in thread 2.

Well, originally, my code loaded a smob interface
(defined a bunch of smobs)  in thread 1, which I then
found to be undefined in thread 2 (and all subsequent
threads).  Would your fix solve this?

Under the "principle of least surprise", I certainly was
surprised that not all threads behaved the same; I got
bitten by this, and wasted a day debugging this, and
designing a  work-around for this "feature".  I would
like to suggest  that libguile itself should be performing
this step, rather than having the developer get caught
off -guard and scratching their head.

--linas

Re: bug: defining things in different threads

[Neil Jerram]

2008/11/15 Linas Vepstas <linasvepstas@gmail.com>:
>
> Well, originally, my code loaded a smob interface
> (defined a bunch of smobs)  in thread 1, which I then
> found to be undefined in thread 2 (and all subsequent
> threads).  Would your fix solve this?

I guess yes if the problem was trying to reference the definitions
from the wrong module; no if the problem was define not being
threadsafe.  (And I'm guessing it was the latter.)

> Under the "principle of least surprise", I certainly was
> surprised that not all threads behaved the same; I got
> bitten by this, and wasted a day debugging this, and
> designing a  work-around for this "feature".

Do you mean you think fluids should not exist, or that
(current-module) should not be a fluid?

(On the other hand, if you mean just that define should be threadsafe,
I definitely agree with that!)

       Neil

Re: bug: defining things in different threads

[Linas Vepstas]

2008/11/19 Neil Jerram <neiljerram@googlemail.com>:
> 2008/11/15 Linas Vepstas <linasvepstas@gmail.com>:
>>
>> Well, originally, my code loaded a smob interface
>> (defined a bunch of smobs)  in thread 1, which I then
>> found to be undefined in thread 2 (and all subsequent
>> threads).  Would your fix solve this?
>
> I guess yes if the problem was trying to reference the definitions
> from the wrong module; no if the problem was define not being
> threadsafe.  (And I'm guessing it was the latter.)

No, the former:  as you point out in the previous email, the
first thread to run is in the (guile-user) module. Typically,
the first thread is where application-defined smobs would
be set up and initialized.   At some point, additional threads
are created, and go into guile mode.  None of these threads
perform an explicit  (set-current-module (guile-user)), and
so the application smobs are undefined in those threads.

Thus, it would seem that symbols defined at the "global
scope" of the first thread aren't perceived as global in the
others.

The problem can be thought of in two ways:
A) the guile docs do not mention that new threads should
perform a (set-current-module ...) and so app developers
won't ... and will discover this behaviour as a surprise.

B) I think documenting this is "wrong"; it puts un-necessary
additional burden on the app developer (it can be non trivial
to  perform all the required initializations in exactly the right
places).  Rather, the semantics that I want to have happen
is that something defined (at global scope) in one thread
should be visible in all threads.

>> Under the "principle of least surprise", I certainly was
>> surprised that not all threads behaved the same; I got
>> bitten by this, and wasted a day debugging this, and
>> designing a  work-around for this "feature".
>
> Do you mean you think fluids should not exist, or that
> (current-module) should not be a fluid?

No.

I don't understand why the (guile-user) module of the first
thread is scoped differently than the (guile) module of all
of the other threads. I think that what I want is for all
threads default to the same module on startup.

> (On the other hand, if you mean just that define should be threadsafe,
> I definitely agree with that!)

Well, yes.

--linas

Re: bug: defining things in different threads

[Linas Vepstas]

2008/11/15 Neil Jerram <neiljerram@googlemail.com>:
> 2008/11/12 Linas Vepstas <linasvepstas@gmail.com>:
>
>> Yes, of course, the problem remains.
>
> Explanation:
>
> Thread 1 is the first thread that does any Guile stuff, so it loads
> boot-9.scm, which means that it ends up in the (guile-user) module.
>
> Thread 2 hasn't done any (set-current-module ...), so it stays at the
> default, which is (guile).
>
> Solution: add (define-module (guile-user)) to the code that you
> execute in thread 2.

This won't actually work, for some reason or another.
Once I am in the second, third, etc. thread, then I cannot
set the current module to be guile-user.  So, for example:

scm-interp> (current-module)
#<module (guile) f5e93e80>
scm-interp> (define-module (guile-user))
#<directory (guile-user) f5e96600>
scm-interp>  (current-module)
#<module (guile) f5e93e80>
scm-interp> (set-current-module (define-module (guile-user)))
#<directory (guile-user) f5e96600>
scm-interp>  (current-module)
#<module (guile) f5e93e80>

Here, scm-interp is my shell, which just calls
scm_c_eval() from C code.

What's worse, I seem to not be able to use modules
when I'm in this mode.  I can reproduce this behaviour
from the standard command line client. So for example,
list-index is defined in boot-9, but I want the one in srfi-1.
They don't work alike; the boot-9 one complains about
something when fed an example from srfi-1:


linas@backlot: /usr/share/guile/1.8/srfi $ guile
guile>  (list-index even? '(3 1 4 1 5 9))
ERROR: Wrong type (expecting pair): #<primitive-procedure even?>
ABORT: (wrong-type-arg)
guile>  (use-modules (srfi srfi-1))
guile> (list-index even? '(3 1 4 1 5 9))
2
guile>  (current-module)
#<directory (guile-user) f7c48600>
guile>  (define-module (guile))
#<module (guile) f7c45e80>
guile> (current-module)
#<module (guile) f7c45e80>
guile> (list-index even? '(3 1 4 1 5 9))
ERROR: Wrong type (expecting pair): #<primitive-procedure even?>
ABORT: (wrong-type-arg)
guile>  (use-modules (srfi srfi-1))
guile> (list-index even? '(3 1 4 1 5 9))
ERROR: Wrong type (expecting pair): #<primitive-procedure even?>
ABORT: (wrong-type-arg)
guile>

So apparently, it seems that I can't do a (use-modules)
and have it stick, when I'm not in guile-user. What's worse,
when I'm in any thread other than the first thread, I cannot
get to guile-user.  Which means that, no matter what,
I cannot use srfi-1; there seems to be no workaround for this.

--linas

Re: bug: defining things in different threads

[Linas Vepstas]

The saga continues...

2008/11/20 Linas Vepstas <linasvepstas@gmail.com>:
> 2008/11/15 Neil Jerram <neiljerram@googlemail.com>:
>> 2008/11/12 Linas Vepstas <linasvepstas@gmail.com>:
>>
>>> Yes, of course, the problem remains.
>>
>> Explanation:
>>
>> Thread 1 is the first thread that does any Guile stuff, so it loads
>> boot-9.scm, which means that it ends up in the (guile-user) module.
>>
>> Thread 2 hasn't done any (set-current-module ...), so it stays at the
>> default, which is (guile).
>>
>> Solution: add (define-module (guile-user)) to the code that you
>> execute in thread 2.
>
> This won't actually work, for some reason or another.
> Once I am in the second, third, etc. thread, then I cannot
> set the current module to be guile-user.  So, for example:
>
> scm-interp> (current-module)
> #<module (guile) f5e93e80>
> scm-interp> (define-module (guile-user))
> #<directory (guile-user) f5e96600>
> scm-interp>  (current-module)
> #<module (guile) f5e93e80>
> scm-interp> (set-current-module (define-module (guile-user)))
> #<directory (guile-user) f5e96600>
> scm-interp>  (current-module)
> #<module (guile) f5e93e80>
>
> Here, scm-interp is my shell, which just calls
> scm_c_eval() from C code.

This is because the C eval uses the dynwind to get into
a certain module.  So if one *sets* the module, the
dynwind mechanism promptly unsets it as one returns
after the evaluation.

--linas