bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB

bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB

[Anand Mohanadoss]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]

Hi,

I had sent the following to the user forum and did not receive any
comments.  I am reposting it in the bug forum with the hope that one of the
experts may be able to comment...

Thanks,
Anand

---------- Forwarded message ----------
From: Anand Mohanadoss <anand108@gmail.com>
Date: Wed, Feb 25, 2015 at 9:35 PM
Subject: Memory leak from seek/ftell with files larger than 2GB
To: guile-user@gnu.org


Hi,

We are seeing an issue with seek and ftell leaking memory with files larger
than 2GB.

We are using 2.0.11 guile built as a 32-bit application with large file
support enabled (guile was built using gcc 4.4.0 for Linux with flags
_FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE).  The
issue also appears to happen with guile 2.2.

The memory leaks start only after the offset exceeds maximum positive value
for a 32-bit signed integer. ftell and seek do work as expected (given how
lseek should work with large file support).

Appended is a program that illustrates the problem.  The first seek simply
skips the part of the file where you won't see a memory leak. If you
comment out ftell and the second seek lines and un-comment the lines that
follow them, there is no memory leak.

Is this a bug in guile or should we be doing things differently?  If this
is a known issue, is there a recommended work around?

Thanks,
Anand

(define MAX_SIGNED_INT 2147483647)
(define BYTES_TO_READ 10)

(define file "/tmp/test.pcap")  ;sample file greater than 2.5GB

(define (traverse file)
 (let* ((port (open-input-file file #:binary #t))
        (file-sz (stat:size (stat port)))
        (ua (make-bytevector BYTES_TO_READ 0))
        (cur-offset 0))
   (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
   (while (< (ftell port) (- file-sz BYTES_TO_READ))
   ;(while (< cur-offset (- file-sz BYTES_TO_READ))
       (seek  port BYTES_TO_READ SEEK_CUR)
       ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
       (set! cur-offset (+ BYTES_TO_READ cur-offset)))
   (close-port port)))

(traverse file)

[-- Attachment #2: Type: text/html, Size: 2757 bytes --]

bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB

[Andy Wingo]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]

Very strange bug!  I can reproduce it with this file:


[-- Attachment #2: run as "guile foo.scm" --]
[-- Type: text/plain, Size: 578 bytes --]

(use-modules (rnrs bytevectors)
             (ice-9 binary-ports))

(define MAX_SIGNED_INT 2147483647)
(define BYTES_TO_READ 10)

(define (traverse port)
  (let* ((file-sz (stat:size (stat port)))
         (ua (make-bytevector BYTES_TO_READ 0))
         (cur-offset 0))
    (let lp ((cur-offset (seek port (- MAX_SIGNED_INT 1000) SEEK_CUR)))
      (when (< cur-offset (- file-sz BYTES_TO_READ))
        (lp (seek port BYTES_TO_READ SEEK_CUR))))
    (close-port port))) 

(define port (mkstemp! (string-copy "/tmp/big-file-XXXXXX")))
(truncate-file port #e2.5e9)
(traverse port)

[-- Attachment #3: Type: text/plain, Size: 2163 bytes --]


I wonder what it could be!

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108@gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand108@gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user@gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files
> larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with
> flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
>
> The memory leaks start only after the offset exceeds maximum positive
> value for a 32-bit signed integer. ftell and seek do work as expected
> (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory leak.
> If you comment out ftell and the second seek lines and un-comment the
> lines that follow them, there is no memory leak. 
>
> Is this a bug in guile or should we be doing things differently? If
> this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port))) 
>
> (traverse file)

bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB

[Andy Wingo]

Hi,

Thank you very much for this one!  Turns out we had an incredibly
embarrassing bug in which we forgot to attach finalizers for bignums
created by scm_from_{uint64,int64} on 32-bit platforms.  Fixed in master
and stable-2.0.

Cheers,

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108@gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand108@gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user@gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files
> larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with
> flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
>
> The memory leaks start only after the offset exceeds maximum positive
> value for a 32-bit signed integer. ftell and seek do work as expected
> (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory leak.
> If you comment out ftell and the second seek lines and un-comment the
> lines that follow them, there is no memory leak. 
>
> Is this a bug in guile or should we be doing things differently? If
> this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port))) 
>
> (traverse file)

bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB

[Anand Mohanadoss]

[-- Attachment #1: Type: text/plain, Size: 2626 bytes --]

Hi Andy,

Thanks a lot for the fix!

Anand

On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo <wingo@pobox.com> wrote:

> Hi,
>
> Thank you very much for this one!  Turns out we had an incredibly
> embarrassing bug in which we forgot to attach finalizers for bignums
> created by scm_from_{uint64,int64} on 32-bit platforms.  Fixed in master
> and stable-2.0.
>
> Cheers,
>
> Andy
>
> On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108@gmail.com> writes:
>
> > Hi,
> >
> > I had sent the following to the user forum and did not receive any
> > comments. I am reposting it in the bug forum with the hope that one of
> > the experts may be able to comment...
> >
> > Thanks,
> > Anand
> >
> > ---------- Forwarded message ----------
> > From: Anand Mohanadoss <anand108@gmail.com>
> > Date: Wed, Feb 25, 2015 at 9:35 PM
> > Subject: Memory leak from seek/ftell with files larger than 2GB
> > To: guile-user@gnu.org
> >
> > Hi,
> >
> > We are seeing an issue with seek and ftell leaking memory with files
> > larger than 2GB.
> >
> > We are using 2.0.11 guile built as a 32-bit application with large
> > file support enabled (guile was built using gcc 4.4.0 for Linux with
> > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
> >
> > The memory leaks start only after the offset exceeds maximum positive
> > value for a 32-bit signed integer. ftell and seek do work as expected
> > (given how lseek should work with large file support).
> >
> > Appended is a program that illustrates the problem. The first seek
> > simply skips the part of the file where you won't see a memory leak.
> > If you comment out ftell and the second seek lines and un-comment the
> > lines that follow them, there is no memory leak.
> >
> > Is this a bug in guile or should we be doing things differently? If
> > this is a known issue, is there a recommended work around?
> >
> > Thanks,
> > Anand
> >
> > (define MAX_SIGNED_INT 2147483647)
> > (define BYTES_TO_READ 10)
> >
> > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
> >
> > (define (traverse file)
> > (let* ((port (open-input-file file #:binary #t))
> > (file-sz (stat:size (stat port)))
> > (ua (make-bytevector BYTES_TO_READ 0))
> > (cur-offset 0))
> > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> > (while (< (ftell port) (- file-sz BYTES_TO_READ))
> > ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> > (seek port BYTES_TO_READ SEEK_CUR)
> > ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> > (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> > (close-port port)))
> >
> > (traverse file)
>

[-- Attachment #2: Type: text/html, Size: 3505 bytes --]