crashes with Fibers

crashes with Fibers

[Clément Lassieur]

Hi,

I'm encountering a few crashes with Fibers that happen when I call
CALL-WITH-NEW-THREAD in a Fiber.  TEST4 crashes every time.  TEST5 never
outputs, but it doesn't crash every time.

TEST1 is exactly like TEST5 except that I replace CALL-WITH-NEW-THREAD
with SPAWN-FIBER.

Is it a mistake from me or a Guile bug?  If it is a bug, do you know if
there are workarounds?

Thank you,
Clément

    (use-modules (fibers channels)
                 (fibers))

    ;; good
    (define (test1)
      (run-fibers
       (lambda ()
         (spawn-fiber
          (lambda ()
            (let ((channel (make-channel)))
              (spawn-fiber
               (lambda ()
                 (put-message channel "hello world")))
              (format #t "~a~%" (get-message channel))))))
       #:drain? #t))
    ⊣ hello world

    ;; good
    (define (test2)
      (let ((channel (make-channel)))
        (call-with-new-thread
         (lambda ()
           (put-message channel "hello world")))
        (format #t "~a~%" (get-message channel))))
    ⊣ hello world
    ⇒ #t

    ;; good
    (define (test3)
      (run-fibers
       (lambda ()
         (let ((channel (make-channel)))
           (call-with-new-thread
            (lambda ()
              (put-message channel "hello world")))
           (format #t "~a~%" (get-message channel))))
       #:drain? #t))
    ⊣ hello world
    ⇒ #t

    ;; bad
    (define (test4)
      (run-fibers
       (lambda ()
         (spawn-fiber
          (lambda ()
            (let ((channel (make-channel)))
              (call-with-new-thread
               (lambda ()
                 (put-message channel "hello world")))))))
       #:drain? #t))
    ⊣ scheme@(guile-user)> In /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
          402:6  1 (suspend-current-fiber _)
      In unknown file:
                 0 (scm-error misc-error #f "~A" ("Attempt to suspend fiber within continuation barrier") #f)
      ERROR: In procedure scm-error:
      Attempt to suspend fiber within continuation barrier

    ;; bad
    (define (test5)
      (run-fibers
       (lambda ()
         (spawn-fiber
          (lambda ()
            (let ((channel (make-channel)))
              (call-with-new-thread
               (lambda ()
                 (put-message channel "hello world")))
              (format #t "~a~%" (get-message channel))))))
       #:drain? #t))
    ⊣ scheme@(guile-user)> In /home/clement/.guix-profile/share/guile/site/2.2/fibers/operations.scm:
          188:5  3 (perform-operation #<<base-op> wrap-fn: #f try-fn: #<procedure try-fn ()> block-fn: #<procedure block-fn (put-flag put-sched resume-put)>>)
      In /home/clement/.guix-profile/share/guile/site/2.2/fibers/channels.scm:
          88:26  2 (try-fn)
      In /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
          219:6  1 (schedule-fiber! _ _)
      In unknown file:
                 0 (scm-error misc-error #f "~A" ("epoll instance is dead") #f)
      ERROR: In procedure scm-error:
      epoll instance is dead

Re: crashes with Fibers

[Amirouche Boubekki]

On 2018-06-29 11:20, Clément Lassieur wrote:
> Hi,
> 
> I'm encountering a few crashes with Fibers that happen when I call
> CALL-WITH-NEW-THREAD in a Fiber.  TEST4 crashes every time.  TEST5 
> never
> outputs, but it doesn't crash every time.
> 
> TEST1 is exactly like TEST5 except that I replace CALL-WITH-NEW-THREAD
> with SPAWN-FIBER.
> 
> Is it a mistake from me or a Guile bug?  If it is a bug, do you know if
> there are workarounds?

Sorry, I did not read the code. What are you trying to achieve?


> 
> Thank you,
> Clément
> 
>     (use-modules (fibers channels)
>                  (fibers))
> 
>     ;; good
>     (define (test1)
>       (run-fibers
>        (lambda ()
>          (spawn-fiber
>           (lambda ()
>             (let ((channel (make-channel)))
>               (spawn-fiber
>                (lambda ()
>                  (put-message channel "hello world")))
>               (format #t "~a~%" (get-message channel))))))
>        #:drain? #t))
>     ⊣ hello world
> 
>     ;; good
>     (define (test2)
>       (let ((channel (make-channel)))
>         (call-with-new-thread
>          (lambda ()
>            (put-message channel "hello world")))
>         (format #t "~a~%" (get-message channel))))
>     ⊣ hello world
>     ⇒ #t
> 
>     ;; good
>     (define (test3)
>       (run-fibers
>        (lambda ()
>          (let ((channel (make-channel)))
>            (call-with-new-thread
>             (lambda ()
>               (put-message channel "hello world")))
>            (format #t "~a~%" (get-message channel))))
>        #:drain? #t))
>     ⊣ hello world
>     ⇒ #t
> 
>     ;; bad
>     (define (test4)
>       (run-fibers
>        (lambda ()
>          (spawn-fiber
>           (lambda ()
>             (letain' guile, an opinionated choic ((channel 
> (make-channel)))
>               (call-with-new-thread
>                (lambda ()
>                  (put-message channel "hello world")))))))
>        #:drain? #t))
>     ⊣ scheme@(guile-user)> In
> /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
>           402:6  1 (suspend-current-fiber _)
>       In unknown file:
>                  0 (scm-error misc-error #f "~A" ("Attempt to suspend
> fiber within continuation barrier") #f)
>       ERROR: In procedure scm-error:
>       Attempt to suspend fiber within continuation barrier
> 
>     ;; bad
>     (define (test5)
>       (run-fibers
>        (lambda ()
>          (spawn-fiber
>           (lambda ()
>             (let ((channel (make-channel)))
>               (call-with-new-thread
>                (lambda ()
>                  (put-message channel "hello world")))
>               (format #t "~a~%" (get-message channel))))))
>        #:drain? #t))
>     ⊣ scheme@(guile-user)> In
> /home/clement/.guix-profile/share/guile/site/2.2/fibers/operations.scm:
>           188:5  3 (perform-operation #<<base-op> wrap-fn: #f try-fn:
> #<procedure try-fn ()> block-fn: #<procedure block-fn (put-flag
> put-sched resume-put)>>)
>       In 
> /home/clement/.guix-profile/share/guile/site/2.2/fibers/channels.scm:
>           88:26  2 (try-fn)
>       In 
> /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
>           219:6  1 (schedule-fiber! _ _)
>       In unknown file:
>                  0 (scm-error misc-error #f "~A" ("epoll instance is 
> dead") #f)
>       ERROR: In procedure scm-error:
>       epoll instance is dead

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: crashes with Fibers

[Clément Lassieur]

Amirouche Boubekki <amirouche@hypermove.net> writes:

> Sorry, I did not read the code. What are you trying to achieve?

Within a fiber, I need to spawn several Guile-Git clones in parallel.
Since they block the Fibers scheduler, they need to be in separate
threads.

Re: crashes with Fibers

[Amirouche Boubekki]

On 2018-07-01 15:09, Clément Lassieur wrote:
> Amirouche Boubekki <amirouche@hypermove.net> writes:
> 
>> Sorry, I did not read the code. What are you trying to achieve?
> 
> Within a fiber, I need to spawn several Guile-Git clones in parallel.
> Since they block the Fibers scheduler, they need to be in separate
> threads.

I tried that in the past it was working, but don't remember correctly
how I did.

epoll is dead seems to indicate that there is no fiber scheduler running
from the thread your are calling put-message. This might be a design 
decision
or not. My understanding is that your code should work as intended.

Anyway, try to spawn the thread and/or create the channel before you
run fibers. I can't try that myself because of my slow connection which
takes ages to install guile-fibers.

Something like:

(define (test6)
   (let ((channel (make-channel)))
     (call-with-new-thread
      (lambda ()
        (put-message channel "hello world")))
     (run-fibers
      (lambda ()
        (spawn-fiber
         (lambda ()
           (format #t "~a~%" (get-message channel)))))
     #:drain? #t)))

Re: crashes with Fibers

[Amirouche Boubekki]

On 2018-07-02 00:32, Amirouche Boubekki wrote:
> On 2018-07-01 15:09, Clément Lassieur wrote:
>> Amirouche Boubekki <amirouche@hypermove.net> writes:
>> 
>>> Sorry, I did not read the code. What are you trying to achieve?
>> 
>> Within a fiber, I need to spawn several Guile-Git clones in parallel.
>> Since they block the Fibers scheduler, they need to be in separate
>> threads.
> 
> I tried that in the past it was working, but don't remember correctly
> how I did.

> epoll is dead seems to indicate that there is no fiber scheduler 
> running
> from the thread your are calling put-message. This might be a design 
> decision
> or not. My understanding is that your code should work as intended.

You test5 code should definitly work. Otherwise, it requires to 
pre-allocate
as many thread as you need before you know you will them. May be we can 
look
at it backward and say, that allocating a pool of threads in advance is 
more
interesting performance wise.

> Anyway, try to spawn the thread and/or create the channel before you
> run fibers. I can't try that myself because of my slow connection which
> takes ages to install guile-fibers.
> 
> Something like:

(use-modules (fibers))
(use-modules (fibers channels))

> (define (test6)
>   (let ((channel (make-channel)))
>     (call-with-new-thread
>      (lambda ()
>        (put-message channel "hello world")))
>     (run-fibers
>      (lambda ()
>        (spawn-fiber
>         (lambda ()
>           (format #t "~a~%" (get-message channel)))))
>     #:drain? #t)))

It works on my side.

Re: crashes with Fibers

[Amirouche Boubekki]

On 2018-07-02 00:41, Amirouche Boubekki wrote:
> On 2018-07-02 00:32, Amirouche Boubekki wrote:
>> On 2018-07-01 15:09, Clément Lassieur wrote:
>>> Amirouche Boubekki <amirouche@hypermove.net> writes:
>>> 
>>>> Sorry, I did not read the code. What are you trying to achieve?
>>> 
>>> Within a fiber, I need to spawn several Guile-Git clones in parallel.
>>> Since they block the Fibers scheduler, they need to be in separate
>>> threads.
>> 
>> I tried that in the past it was working, but don't remember correctly
>> how I did.
> 
>> epoll is dead seems to indicate that there is no fiber scheduler 
>> running
>> from the thread your are calling put-message. This might be a design 
>> decision
>> or not. My understanding is that your code should work as intended.
> 
> You test5 code should definitly work. Otherwise, it requires to 
> pre-allocate
> as many thread as you need before you know you will them. May be we can 
> look
> at it backward and say, that allocating a pool of threads in advance is 
> more
> interesting performance wise.
> 
>> Anyway, try to spawn the thread and/or create the channel before you
>> run fibers. I can't try that myself because of my slow connection 
>> which
>> takes ages to install guile-fibers.
>> 
>> Something like:
> 
> (use-modules (fibers))
> (use-modules (fibers channels))
> 
>> (define (test6)
>>   (let ((channel (make-channel)))
>>     (call-with-new-thread
>>      (lambda ()
>>        (put-message channel "hello world")))
>>     (run-fibers
>>      (lambda ()
>>        (spawn-fiber
>>         (lambda ()
>>           (format #t "~a~%" (get-message channel)))))
>>     #:drain? #t)))
> 
> It works on my side.

FWIW, the following code doesn't crash, but apparently does nothing:

(define (test7)
   (let ((channel (make-channel)))
     (run-fibers
      (lambda ()
        (spawn-fiber
         (lambda ()
           (call-with-new-thread
            (lambda ()
              (put-message channel "hello world")))
           (format #t "~a~%" (get-message channel)))))
     #:drain? #t)))

Re: crashes with Fibers

[Clément Lassieur]

Hi Amirouche,

Amirouche Boubekki <amirouche@hypermove.net> writes:

>> You test5 code should definitly work. Otherwise, it requires to pre-allocate
>> as many thread as you need before you know you will them. May be we can look
>> at it backward and say, that allocating a pool of threads in advance is more
>> interesting performance wise.

But the cost of allocating a thread pool is very small compared to the
cost of a doing git clone, which may take a minute or so.

>>> Anyway, try to spawn the thread and/or create the channel before you
>>> run fibers. I can't try that myself because of my slow connection which
>>> takes ages to install guile-fibers.

I can't do that because in the program I'm working on (Cuirass), the
first calls to RUN-FIBERS and SPAWN-FIBER happen very early[1].  They
are kind of the entry point of the program.  At that time I don't know
how many repositories I will need to fetch simultaneously.  Thus test3
and test6 don't meet my needs.

[1]: https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/bin/cuirass.in#n144

>>> Something like:
>>
>> (use-modules (fibers))
>> (use-modules (fibers channels))
>>
>>> (define (test6)
>>>   (let ((channel (make-channel)))
>>>     (call-with-new-thread
>>>      (lambda ()
>>>        (put-message channel "hello world")))
>>>     (run-fibers
>>>      (lambda ()
>>>        (spawn-fiber
>>>         (lambda ()
>>>           (format #t "~a~%" (get-message channel)))))
>>>     #:drain? #t)))
>>
>> It works on my side.
>
> FWIW, the following code doesn't crash, but apparently does nothing:
>
> (define (test7)
>   (let ((channel (make-channel)))
>     (run-fibers
>      (lambda ()
>        (spawn-fiber
>         (lambda ()
>           (call-with-new-thread
>            (lambda ()
>              (put-message channel "hello world")))
>           (format #t "~a~%" (get-message channel)))))
>     #:drain? #t)))

This crashes the same way test5 does, you need to run it more :-)

Re: crashes with Fibers

[Ludovic Courtès]

Hello Clément,

Clément Lassieur <clement@lassieur.org> skribis:

>     ;; bad
>     (define (test4)
>       (run-fibers
>        (lambda ()
>          (spawn-fiber
>           (lambda ()
>             (let ((channel (make-channel)))
>               (call-with-new-thread
>                (lambda ()
>                  (put-message channel "hello world")))))))
>        #:drain? #t))
>     ⊣ scheme@(guile-user)> In /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
>           402:6  1 (suspend-current-fiber _)
>       In unknown file:
>                  0 (scm-error misc-error #f "~A" ("Attempt to suspend fiber within continuation barrier") #f)
>       ERROR: In procedure scm-error:
>       Attempt to suspend fiber within continuation barrier

I think the problem here is that the new thread inherit the dynamic
environment of the spawning thread.  Thus, ‘put-message’, called in that
new thread, thinks it’s running within a Fiber, but it’s not.

Because of that, ‘put-message’ tries to suspend itself, but it cannot:
‘call-with-new-thread’ is written in C, so it’s a “continuation barrier”
(meaning that it’s a continuation that cannot be captured and resumed
later.)

So I think if you really want that, you can perhaps do something like
(untested):

  (call-with-new-thread
    (lambda ()
      (parameterize ((current-fiber #f))
        (put-message channel "hello world"))))

HTH!
Ludo’.

Re: crashes with Fibers

[Clément Lassieur]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Clément,
>
> Clément Lassieur <clement@lassieur.org> skribis:
>
>>     ;; bad
>>     (define (test4)
>>       (run-fibers
>>        (lambda ()
>>          (spawn-fiber
>>           (lambda ()
>>             (let ((channel (make-channel)))
>>               (call-with-new-thread
>>                (lambda ()
>>                  (put-message channel "hello world")))))))
>>        #:drain? #t))
>>     ⊣ scheme@(guile-user)> In /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
>>           402:6  1 (suspend-current-fiber _)
>>       In unknown file:
>>                  0 (scm-error misc-error #f "~A" ("Attempt to suspend fiber within continuation barrier") #f)
>>       ERROR: In procedure scm-error:
>>       Attempt to suspend fiber within continuation barrier
>
> I think the problem here is that the new thread inherit the dynamic
> environment of the spawning thread.  Thus, ‘put-message’, called in that
> new thread, thinks it’s running within a Fiber, but it’s not.
>
> Because of that, ‘put-message’ tries to suspend itself, but it cannot:
> ‘call-with-new-thread’ is written in C, so it’s a “continuation barrier”
> (meaning that it’s a continuation that cannot be captured and resumed
> later.)
>
> So I think if you really want that, you can perhaps do something like
> (untested):
>
>   (call-with-new-thread
>     (lambda ()
>       (parameterize ((current-fiber #f))
>         (put-message channel "hello world"))))

It works, but only with (@@ (fibers internal) current-fiber) because the
parameter isn't exported.

Thank you Ludo!

Re: crashes with Fibers

[Clément Lassieur]

Clément Lassieur <clement@lassieur.org> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello Clément,
>>
>> Clément Lassieur <clement@lassieur.org> skribis:
>>
>>>     ;; bad
>>>     (define (test4)
>>>       (run-fibers
>>>        (lambda ()
>>>          (spawn-fiber
>>>           (lambda ()
>>>             (let ((channel (make-channel)))
>>>               (call-with-new-thread
>>>                (lambda ()
>>>                  (put-message channel "hello world")))))))
>>>        #:drain? #t))
>>>     ⊣ scheme@(guile-user)> In /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
>>>           402:6  1 (suspend-current-fiber _)
>>>       In unknown file:
>>>                  0 (scm-error misc-error #f "~A" ("Attempt to suspend fiber within continuation barrier") #f)
>>>       ERROR: In procedure scm-error:
>>>       Attempt to suspend fiber within continuation barrier
>>
>> I think the problem here is that the new thread inherit the dynamic
>> environment of the spawning thread.  Thus, ‘put-message’, called in that
>> new thread, thinks it’s running within a Fiber, but it’s not.
>>
>> Because of that, ‘put-message’ tries to suspend itself, but it cannot:
>> ‘call-with-new-thread’ is written in C, so it’s a “continuation barrier”
>> (meaning that it’s a continuation that cannot be captured and resumed
>> later.)
>>
>> So I think if you really want that, you can perhaps do something like
>> (untested):
>>
>>   (call-with-new-thread
>>     (lambda ()
>>       (parameterize ((current-fiber #f))
>>         (put-message channel "hello world"))))
>
> It works, but only with (@@ (fibers internal) current-fiber) because the
> parameter isn't exported.
>
> Thank you Ludo!

Hi Andy,

I'm adding you to this conversation to notify you about this Fibers
issue.  (I don't have a Github account to report the bug there.)

Thanks,
Clément

Re: crashes with Fibers

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

https://github.com/wingo/fibers/issues/21

Le mar. 17 juil. 2018 à 20:17, Clément Lassieur <clement@lassieur.org> a
écrit :

> Clément Lassieur <clement@lassieur.org> writes:
>
> > Ludovic Courtès <ludo@gnu.org> writes:
> >
> >> Hello Clément,
> >>
> >> Clément Lassieur <clement@lassieur.org> skribis:
> >>
> >>>     ;; bad
> >>>     (define (test4)
> >>>       (run-fibers
> >>>        (lambda ()
> >>>          (spawn-fiber
> >>>           (lambda ()
> >>>             (let ((channel (make-channel)))
> >>>               (call-with-new-thread
> >>>                (lambda ()
> >>>                  (put-message channel "hello world")))))))
> >>>        #:drain? #t))
> >>>     ⊣ scheme@(guile-user)> In
> /home/clement/.guix-profile/share/guile/site/2.2/fibers/internal.scm:
> >>>           402:6  1 (suspend-current-fiber _)
> >>>       In unknown file:
> >>>                  0 (scm-error misc-error #f "~A" ("Attempt to suspend
> fiber within continuation barrier") #f)
> >>>       ERROR: In procedure scm-error:
> >>>       Attempt to suspend fiber within continuation barrier
> >>
> >> I think the problem here is that the new thread inherit the dynamic
> >> environment of the spawning thread.  Thus, ‘put-message’, called in that
> >> new thread, thinks it’s running within a Fiber, but it’s not.
> >>
> >> Because of that, ‘put-message’ tries to suspend itself, but it cannot:
> >> ‘call-with-new-thread’ is written in C, so it’s a “continuation barrier”
> >> (meaning that it’s a continuation that cannot be captured and resumed
> >> later.)
> >>
> >> So I think if you really want that, you can perhaps do something like
> >> (untested):
> >>
> >>   (call-with-new-thread
> >>     (lambda ()
> >>       (parameterize ((current-fiber #f))
> >>         (put-message channel "hello world"))))
> >
> > It works, but only with (@@ (fibers internal) current-fiber) because the
> > parameter isn't exported.
> >
> > Thank you Ludo!
>
> Hi Andy,
>
> I'm adding you to this conversation to notify you about this Fibers
> issue.  (I don't have a Github account to report the bug there.)
>
> Thanks,
> Clément
>
>

[-- Attachment #2: Type: text/html, Size: 3269 bytes --]