(n-for-each-par-map 16 store download (reverse (iota (max-id))))) crash

(n-for-each-par-map 16 store download (reverse (iota (max-id))))) crash

[Amirouche Boubekki]

I have a program that try to download hackernews locally.

What it does is simple, it fetch the max identifier and
http-get each json value starting with the most recent
item. I use n-for-each-par-map with 16 threads I have
8 cores.

Here is the full program:

(define-module (hn))

(use-modules (srfi srfi-1))
(use-modules (ice-9 receive))
(use-modules (ice-9 threads))
(use-modules (ice-9 iconv))
(use-modules (web client))
(use-modules (json))


(define (max-id)
   (receive (response body) (http-get 
"https://hacker-news.firebaseio.com/v0/maxitem.json")
     (string->number (bytevector->string body "utf-8"))))

(define (download uid)
   (catch #t
     (lambda ()
       (let* ((uid (1+ uid))
              (url "https://hacker-news.firebaseio.com/v0/item/~a.json")
              (url (format #f url uid)))
         (cons uid
               (json-string->scm
                (call-with-values (lambda () (http-get url))
                  (lambda (response body)
                    (bytevector->string body "utf-8")))))))
     (lambda _ '())))

(define (store pair)
   (if (null? pair)
       (format #t "X\n")
       (let ((port (open-file "hn.scm" "a")))
         (format #t "~a\n" (car pair))
         (write (cdr pair) port)
         (close port))))

(define (dump)
   (n-for-each-par-map 16 store download (reverse (iota (max-id)))))

(dump)

It also requires json module from 
https://raw.githubusercontent.com/a-guile-mind/Culturia/master/src/json.scm

How can I debug this?

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: (n-for-each-par-map 16 store download (reverse (iota (max-id))))) crash

[Mark H Weaver]

Hi,

Amirouche Boubekki <amirouche@hypermove.net> writes:

> I have a program that try to download hackernews locally.
>
> What it does is simple, it fetch the max identifier and
> http-get each json value starting with the most recent
> item. I use n-for-each-par-map with 16 threads I have
> 8 cores.
>
> Here is the full program:

[...]

> (define (store pair)
>   (if (null? pair)
>       (format #t "X\n")
>       (let ((port (open-file "hn.scm" "a")))
>         (format #t "~a\n" (car pair))

These calls to (format #t ...), which write to a port that is shared by
multiple threads, should be performed while holding a mutex to prevent
concurrent writes to the same port.

I/O operations in Guile do not include built-in thread synchronization,
at least not in the fast path cases.  However, an effort was made to
avoid _crashes_ on common architectures in the event of concurrent use
of the same port.  Our hope was that the worst that would typically
happen is garbled I/O.  Perhaps we failed to realize that hope.

> How can I debug this?

It would be helpful to see a GDB backtrace from a crash in this program.
Does it help to protect the shared port with a mutex?

     Thanks,
       Mark

Re: (n-for-each-par-map 16 store download (reverse (iota (max-id))))) crash

[Amirouche Boubekki]

Hi Mark,

Thanks for your support and time.

On 2018-07-05 20:30, Mark H Weaver wrote:
> Hi,
> 
> Amirouche Boubekki <amirouche@hypermove.net> writes:
> 
>> I have a program that try to download hackernews locally.
>> 
>> What it does is simple, it fetch the max identifier and
>> http-get each json value starting with the most recent
>> item. I use n-for-each-par-map with 16 threads I have
>> 8 cores.
>> 
>> Here is the full program:
> 
> [...]
> 
>> (define (store pair)
>>   (if (null? pair)
>>       (format #t "X\n")
>>       (let ((port (open-file "hn.scm" "a")))
>>         (format #t "~a\n" (car pair))
> 
> These calls to (format #t ...), which write to a port that is shared by
> multiple threads, should be performed while holding a mutex to prevent
> concurrent writes to the same port.

Ok, but it's called by:

   (n-for-each-par-map n sproc pproc lst1 lst2)

and the manual says about SPROC that :

   The sproc calls are made serially,
   in list element order, one at a time.

In my case SPROC is 'store', so my understanding is that
it should require no lock.

> 
> I/O operations in Guile do not include built-in thread synchronization,
> at least not in the fast path cases.  However, an effort was made to
> avoid _crashes_ on common architectures in the event of concurrent use
> of the same port.  Our hope was that the worst that would typically
> happen is garbled I/O.  Perhaps we failed to realize that hope.

I saw that behavior while use fibers.

>> How can I debug this?
> 
> It would be helpful to see a GDB backtrace from a crash in this 
> program.

I will retry without mutex and only gdb to reproduce the crash
and have a trace.

> Does it help to protect the shared port with a mutex?

I ran the program with gdb and a lock, it downloads more suff,
but it's also much slower.

>      Thanks,
>        Mark

Thanks again!

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: (n-for-each-par-map 16 store download (reverse (iota (max-id))))) crash

[Mark H Weaver]

Hi,

Amirouche Boubekki <amirouche@hypermove.net> writes:

> On 2018-07-05 20:30, Mark H Weaver wrote:
>> Amirouche Boubekki <amirouche@hypermove.net> writes:
>>
>>> (define (store pair)
>>>   (if (null? pair)
>>>       (format #t "X\n")
>>>       (let ((port (open-file "hn.scm" "a")))
>>>         (format #t "~a\n" (car pair))
>>
>> These calls to (format #t ...), which write to a port that is shared by
>> multiple threads, should be performed while holding a mutex to prevent
>> concurrent writes to the same port.
>
> Ok, but it's called by:
>
>   (n-for-each-par-map n sproc pproc lst1 lst2)
>
> and the manual says about SPROC that :
>
>   The sproc calls are made serially,
>   in list element order, one at a time.
>
> In my case SPROC is 'store', so my understanding is that
> it should require no lock.

You're right, my mistake.  I'm not sure what's going on here.  It would
be good to send a message to bug-guile@gnu.org to file a bug report, and
to continue the discussion there.

    Thanks!
      Mark