bug#24357: pure guile program leaks memory

bug#24357: pure guile program leaks memory

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Using guile 2.1.3, I have a program that:

- reads urls from a text file
- download the urls using curl command via popen
- output the result to stdout

Also, it relies on n-for-each-par-map for ice-9 threads.

IMO, the most suspicious is the definition of the `curl' proc:


   (define (curl url)
      (let* ((port (open-input-pipe (format #f "curl -is \"~a\"" url)))
             (response (read-string port)))
        (close-pipe port)
        response))


I fail to see how this can leak.

To reproduce the bug you need a giant list of preferably different urls.
Such a list is available at http://hyperdev.fr/data/hn/hn.urls.txt.xz

You can run the test program with the following command:

    cat hn.urls.txt | guile urls-step00-fetch.scm > /dev/null

This is not the only program I tried on guile-2.1.3 that leaks but
it's the easiest to reproduce.

I will try to reproduce the bug on 2.0.12.

[-- Attachment #2: urls-step00-fetch.scm --]
[-- Type: text/plain, Size: 652 bytes --]

(use-modules (ice-9 threads))
(use-modules (ice-9 rdelim))
(use-modules (ice-9 popen))
(use-modules (ice-9 rdelim))


;;; wrapping curl command

(define (curl url)
  (let* ((port (open-input-pipe (format #f "curl -is \"~a\"" url)))
         (response (read-string port)))
    (close-pipe port)
    response))


(define (maybe-curl url)
  (catch #t
    (lambda () (cons url (curl url)))
    (lambda _ '())))


(define urls (let loop ((line (read-line))
                        (out '()))
               (if (eof-object? line)
                   out
                   (loop (read-line) (cons line out)))))

(n-for-each-par-map 16 write maybe-curl urls)

bug#24357: pure guile program leaks memory

[Andy Wingo]

On Sat 03 Sep 2016 09:20, Amirouche Boubekki <amirouche@hypermove.net> writes:

> Using guile 2.1.3, I have a program that:
>
> - reads urls from a text file
> - download the urls using curl command via popen
> - output the result to stdout
>
> Also, it relies on n-for-each-par-map for ice-9 threads.

Can you reduce it please?  For example, remove the use of threads.

Andy

bug#24357: pure guile program leaks memory

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On 2016-09-03 11:54, Andy Wingo wrote:
> On Sat 03 Sep 2016 09:20, Amirouche Boubekki <amirouche@hypermove.net> 
> writes:
> 
>> Using guile 2.1.3, I have a program that:
>> 
>> - reads urls from a text file
>> - download the urls using curl command via popen
>> - output the result to stdout
>> 
>> Also, it relies on n-for-each-par-map for ice-9 threads.
> 
> Can you reduce it please?  For example, remove the use of threads.
> 

Ok. I removed threads and only download the same url over and over 
again.

Here is the error I get on stdout:

    (23) Failed writing body

The program is:

(use-modules (ice-9 popen))


;;; wrapping curl command

(define (curl url)
   (let* ((port (open-input-pipe (format #f "curl -is \"~a\"" url)))
          (response (read-string port)))
     (close-pipe port)
     response))


(define (maybe-curl url)
   (catch #t
     (lambda ()
       (display "." (current-error-port))
       (write (cons url (curl url))))
     (lambda _ '())))

(define urls (map (lambda _ "http://hyperdev.fr/") (iota 1000)))

(display "started")

(for-each maybe-curl urls)

[-- Attachment #2: guile-bug-24357.scm --]
[-- Type: text/plain, Size: 489 bytes --]

(use-modules (ice-9 popen))


;;; wrapping curl command

(define (curl url)
  (let* ((port (open-input-pipe (format #f "curl -is \"~a\"" url)))
         (response (read-string port)))
    (close-pipe port)
    response))


(define (maybe-curl url)
  (catch #t
    (lambda ()
      (display "." (current-error-port))
      (write (cons url (curl url))))
    (lambda _ '())))

(define urls (map (lambda _ "http://hyperdev.fr/") (iota 1000)))

(display "started")

(for-each maybe-curl urls)

bug#24357: pure guile program leaks memory

[Amirouche Boubekki]

On 2016-09-03 20:49, Amirouche Boubekki wrote:
> On 2016-09-03 11:54, Andy Wingo wrote:
>> On Sat 03 Sep 2016 09:20, Amirouche Boubekki <amirouche@hypermove.net> 
>> writes:
>> 
>>> Using guile 2.1.3, I have a program that:
>>> 
>>> - reads urls from a text file
>>> - download the urls using curl command via popen
>>> - output the result to stdout
>>> 
>>> Also, it relies on n-for-each-par-map for ice-9 threads.
>> 
>> Can you reduce it please?  For example, remove the use of threads.
>> 
> 
> Ok. I removed threads and only download the same url over and over 
> again.
> 
> Here is the error I get on stdout:
> 
>    (23) Failed writing body
> 

Internet says this a curl issue. Please close this bug.

bug#24357: pure guile program leaks memory

[Andy Wingo]

On Mon 12 Sep 2016 19:47, Amirouche Boubekki <amirouche@hypermove.net> writes:

> On 2016-09-03 20:49, Amirouche Boubekki wrote:
>> On 2016-09-03 11:54, Andy Wingo wrote:
>>> On Sat 03 Sep 2016 09:20, Amirouche Boubekki
>>> <amirouche@hypermove.net> writes:
>>>
>>>> Using guile 2.1.3, I have a program that:
>>>>
>>>> - reads urls from a text file
>>>> - download the urls using curl command via popen
>>>> - output the result to stdout
>>>>
>>>> Also, it relies on n-for-each-par-map for ice-9 threads.
>>>
>>> Can you reduce it please?  For example, remove the use of threads.
>>>
>>
>> Ok. I removed threads and only download the same url over and over
>> again.
>>
>> Here is the error I get on stdout:
>>
>>    (23) Failed writing body
>
> Internet says this a curl issue. Please close this bug.

No problem.  For reference, in the future just add "-done" after the bug
number in the mail, as I did above.  Cheers :)

Andy