“Too many root sets” when calling compile frequently

“Too many root sets” when calling compile frequently

[Jean Abou Samra]

Hi,

Calling the Guile compiler often causes this BDWGC error: “Too
many root sets”.

scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do 
((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
scheme@(guile-user)> (use-modules (system base compile))
scheme@(guile-user)> (repeat 10000 (compile 5))
Too many root sets
Abandon (core dumped)

Any idea what is going on here? Should I report it as a bug?
Is there a workaround?

Thanks,
Jean

Re: “Too many root sets” when calling compile frequently

[Jean Abou Samra]

Le 19/08/2022 à 00:18, Jean Abou Samra a écrit :
> Calling the Guile compiler often causes this BDWGC error: “Too
> many root sets”.
>
> scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do 
> ((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
> scheme@(guile-user)> (use-modules (system base compile))
> scheme@(guile-user)> (repeat 10000 (compile 5))
> Too many root sets
> Abandon (core dumped)
>
> Any idea what is going on here? Should I report it as a bug?
> Is there a workaround?


Interestingly:

scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do 
((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
scheme@(guile-user)> (use-modules (system base compile))
scheme@(guile-user)> (repeat 10000 (compile 5 #:to 'bytecode))
scheme@(guile-user)> (use-modules (system vm loader))
scheme@(guile-user)> (repeat 10000 (load-thunk-from-memory (compile 5 
#:to 'bytecode)))
Too many root sets
Abandon (core dumped)


So the problem lies in the VM loading infrastructure. (This is
as far as I can investigate for now.)

Re: “Too many root sets” when calling compile frequently

[Jean Abou Samra]

Le 19/08/2022 à 00:33, Jean Abou Samra a écrit :
> Le 19/08/2022 à 00:18, Jean Abou Samra a écrit :
>> Calling the Guile compiler often causes this BDWGC error: “Too
>> many root sets”.
>>
>> scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) 
>> (do ((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
>> scheme@(guile-user)> (use-modules (system base compile))
>> scheme@(guile-user)> (repeat 10000 (compile 5))
>> Too many root sets
>> Abandon (core dumped)
>>
>> Any idea what is going on here? Should I report it as a bug?
>> Is there a workaround?
>
>
> Interestingly:
>
> scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do 
> ((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
> scheme@(guile-user)> (use-modules (system base compile))
> scheme@(guile-user)> (repeat 10000 (compile 5 #:to 'bytecode))
> scheme@(guile-user)> (use-modules (system vm loader))
> scheme@(guile-user)> (repeat 10000 (load-thunk-from-memory (compile 5 
> #:to 'bytecode)))
> Too many root sets
> Abandon (core dumped)
>
>
> So the problem lies in the VM loading infrastructure. (This is
> as far as I can investigate for now.)



I tried this code:

(use-modules (system vm loader))

(define-syntax-rule (repeat n-expr expr expr* ...)
   (let ((n n-expr))
     (do ((i 0 (1+ i)))
       ((eqv? i n))
       expr expr* ...)))

(let ((code (compile 5 #:to 'bytecode)))
   (repeat 10000
     (load-thunk-from-memory code)
     (display (length (all-mapped-elf-images))) (newline)))


For me, the output ends with

8174
8175
8176
8177
8178
8179
8180
8181
Too many root sets
Abandon (core dumped)


Now, BDWGC defines MAX_ROOT_SETS in include/private/gc_priv.h as

/* Root sets.  Logically private to mark_rts.c.  But we don't want the  */
/* tables scanned, so we put them here.                                 */
/* MAX_ROOT_SETS is the maximum number of ranges that can be    */
/* registered as static roots.                                  */
# ifdef LARGE_CONFIG
#   define MAX_ROOT_SETS 8192
# elif !defined(SMALL_CONFIG)
#   define MAX_ROOT_SETS 2048
# else
#   define MAX_ROOT_SETS 512
# endif


I am using Fedora, where BDWGC is compiled with --enable-large-config. When
the loader ingests a VM code chunk, it does (loader.c)


   if (gc_root)
     GC_add_roots (gc_root, gc_root + gc_root_size);


So each load is really adding a root until this threshold of 8192
is crossed.

I have no idea if it is possible to fix and/or work around this
other than by not calling compile often like this.

(Yes, I know, I should be using eval for one-off code evaluation,
but it discards source locations, which is what brought me here
in the first place.)

Jean

Re: “Too many root sets” when calling compile frequently

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 904 bytes --]


On 19-08-2022 00:18, Jean Abou Samra wrote:
> Hi,
>
> Calling the Guile compiler often causes this BDWGC error: “Too
> many root sets”.
>
> scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do 
> ((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
> scheme@(guile-user)> (use-modules (system base compile))
> scheme@(guile-user)> (repeat 10000 (compile 5))
> Too many root sets
> Abandon (core dumped)
>
> Any idea what is going on here? Should I report it as a bug?
> Is there a workaround?
>
> Thanks,
> Jean

IIRC, Guile used to support garbage collection of compiled code, but 
that support has been removed.

I cannot find that in the Git history or NEWS, so maybe that's incorrect.

If that is correct, maybe with sufficient tests and care, support for 
unloading compiled code can be restored, removing the call to GC_add_roots.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: “Too many root sets” when calling compile frequently

[Jean Abou Samra]

> Le 19 août 2022 à 12:19, Maxime Devos <maximedevos@telenet.be> a écrit :
> 
> 
>> On 19-08-2022 00:18, Jean Abou Samra wrote:
>> Hi,
>> 
>> Calling the Guile compiler often causes this BDWGC error: “Too
>> many root sets”.
>> 
>> scheme@(guile-user)> (define-syntax-rule (repeat n expr expr* ...) (do ((i 0 (1+ i))) ((eqv? i n)) expr expr* ...))
>> scheme@(guile-user)> (use-modules (system base compile))
>> scheme@(guile-user)> (repeat 10000 (compile 5))
>> Too many root sets
>> Abandon (core dumped)
>> 
>> Any idea what is going on here? Should I report it as a bug?
>> Is there a workaround?
>> 
>> Thanks,
>> Jean
> 
> IIRC, Guile used to support garbage collection of compiled code, but that support has been removed.
> 
> I cannot find that in the Git history or NEWS, so maybe that's incorrect.
> 
> If that is correct, maybe with sufficient tests and care, support for unloading compiled code can be restored, removing the call to GC_add_roots.


Thanks for your reply. I didn’t dig into the history yet, but I might do it later. For now, I have a question. What is the actual purpose of the GC_add_roots call? Is it just to give eternal protection to the objects pointed to by pointers in the memory section? In that case, wouldn’t it also work to create a bytevector from the root section and use scm_permanent_object on it? Or is this a subtlety where Guile registers its ELF bytecode as if it were a dynamic library (also ELF)?

It would be great if Guile supported garbage collection of loaded bytecode, but for my use case in LilyPond, it is actually not strictly  necessary. It should suffice not to error out on the nth call.

Jean

Re: “Too many root sets” when calling compile frequently

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 644 bytes --]


On 20-08-2022 13:08, Jean Abou Samra wrote:
> Thanks for your reply. I didn’t dig into the history yet, but I might do it later. For now, I have a question. What is the actual purpose of the GC_add_roots call? Is it just to give eternal protection to the objects pointed to by pointers in the memory section? In that case, wouldn’t it also work to create a bytevector from the root section and use scm_permanent_object on it? Or is this a subtlety where Guile registers its ELF bytecode as if it were a dynamic library (also ELF)?

My answer is 'Possibly' to all of that, I don't know the loader code at all.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]