bug#12095: Protecting pointer on bytevector with guardian does not protect memory

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Patrick Bernaud]

[-- Attachment #1: message body and .signature --]
[-- Type: text/plain, Size: 658 bytes --]


The memory from a bytevector of which a pointer is taken (with
'bytevector->pointer') can be overwritten even if that pointer has
been put inside a guardian. 'make-c-struct' from (system foreign) is
using 'bytevector->pointer'.

With the test script attached:

$ guile -v | head -1
guile (GNU Guile) 2.0.6.8-cc26b9-dirty
$ guile --no-auto-compile -s test.scm
#<pointer 0x974648c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(110 103 45 108 101 110 103 116 104 0)  <<<< memory overwrite with "ng-length\0" from module
#<pointer 0x974648c>
$

With auto compilation turned on, it looks like the problem can not be
reproduced.

-- 
Patrick Bernaud


[-- Attachment #2: test.scm --]
[-- Type: application/octet-stream, Size: 526 bytes --]

(use-modules (system foreign) (rnrs bytevectors))
(define my-guardian (make-guardian))
(define len 10)
(define x (bytevector->pointer (make-bytevector len 1)))
(define a (pointer-address x))
(display x)(newline)
(my-guardian x)
;(my-guardian (pointer->bytevector x len))
(set! x #f)
(write (pointer->bytevector (make-pointer a) len))(newline)
(gc)
(write (pointer->bytevector (make-pointer a) len))(newline)
(use-modules (htmlprag))
(write (pointer->bytevector (make-pointer a) len))(newline)
(display (my-guardian))(newline)

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Ian Price]

Patrick Bernaud <patrickb@chez.com> writes:

> With auto compilation turned on, it looks like the problem can not be
> reproduced.

I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
there was one caveat, as I didn't have html prag, I substituted for
various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
my own (pfds queues).

Is this bug somehow particular to htmlprag, or can you confirm it with others?

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Daniel Hartwig]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

On 7 August 2012 00:37, Ian Price <ianprice90@googlemail.com> wrote:
> Patrick Bernaud <patrickb@chez.com> writes:
>
>> With auto compilation turned on, it looks like the problem can not be
>> reproduced.

That is to say, when using --fresh-auto-compile.

>
> I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
> 2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
> there was one caveat, as I didn't have html prag, I substituted for
> various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
> my own (pfds queues).
>
> Is this bug somehow particular to htmlprag, or can you confirm it with others?

On x86 Debian sid I reproduced the bug by loading (web server) with
the original test.scm from the mailing list.

The bug report copy differs from the original by replacing
parse-c-struct with pointer->bytevector.

I attach test1.scm, with a loop to load many modules one at a time.
Curiously, when using pointer->bytevector the contents change (for me)
only when freshly compiling:

$ guile --fresh-auto-compile test1.scm
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /home/daniel/Downloads/test1.scm
;;; compiled /home/daniel/.cache/guile/ccache/2.0-LE-4-2.0/home/daniel/Downloads/test1.scm.go
#<pointer 0xa1cdb7c>
#vu8(1 1 1 1 1 1 1 1 1 1)
Contents differ after (web http)
#vu8(110 116 104 101 115 105 122 101 0 1)
#<pointer 0xa1cdb7c>
$ guile test1.scm
#<pointer 0x82c754c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#<pointer 0x82c754c>

With the original (using parse-c-struct) it was the other way around.

[-- Attachment #2: test1.scm --]
[-- Type: application/octet-stream, Size: 1037 bytes --]

(use-modules (system foreign) (rnrs bytevectors))
(define my-guardian (make-guardian))
(define len 10)
(define x (bytevector->pointer (make-bytevector len 1)))
(define a (pointer-address x))
(display x)(newline)
(my-guardian x)
;(my-guardian (pointer->bytevector x len))
(set! x #f)

(define (dump-struct)
  (write (pointer->bytevector (make-pointer a) len))(newline))

(dump-struct)

(let lp ((s1 (with-output-to-string dump-struct))
         (m '((web uri) (web http) (web request) (web response)
              (web client) (web server) (sxml apply-templates)
              (sxml fold) (sxml simple) (sxml ssax)
              (ice-9 popen) (ice-9 getopt-long) (srfi srfi-42))))
  (gc)
  (primitive-eval `(use-modules ,(car m)))
  (let ((s2 (with-output-to-string dump-struct)))
    (if (not (string=? s1 s2))
        (begin
          (display "Contents differ after ")
          (display (car m))
          (newline)
          (display s2))
        (if (pair? (cdr m))
            (lp s2 (cdr m))))))

(display (my-guardian))(newline)

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Ludovic Courtès]

Hi,

Daniel Hartwig <mandyke@gmail.com> skribis:

> (define x (bytevector->pointer (make-bytevector len 1)))
> (define a (pointer-address x))
> (display x)(newline)
> (my-guardian x)
> ;(my-guardian (pointer->bytevector x len))
> (set! x #f)
>
> (define (dump-struct)
>   (write (pointer->bytevector (make-pointer a) len))(newline))

This is expected to fail: ‘bytevector->pointer’ creates a weak reference
from the returned pointer object to the given bytevector.  So when the
pointer object is reclaimed, the bytevector can be reclaimed too, hence
the problem you’re observing.  (And no, guardians don’t protect objects
from garbage collection.)

To put it differently, memory management is left to the user.  The weak
reference I mention is a convenience for simple cases, but for “real
world” situations, one has to take all steps necessary to ensure that
the lifetime of C objects and that of their Scheme counterparts is in sync.

Hope this helps,
Ludo’.

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Daniel Hartwig]

On 7 October 2012 05:41, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi,
>
> Daniel Hartwig <mandyke@gmail.com> skribis:
>
>> (define x (bytevector->pointer (make-bytevector len 1)))
>> (define a (pointer-address x))
>> (display x)(newline)
>> (my-guardian x)
>> ;(my-guardian (pointer->bytevector x len))
>> (set! x #f)
>>
>> (define (dump-struct)
>>   (write (pointer->bytevector (make-pointer a) len))(newline))
>
> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
> from the returned pointer object to the given bytevector.  So when the
> pointer object is reclaimed, the bytevector can be reclaimed too, hence
> the problem you’re observing.  (And no, guardians don’t protect objects
> from garbage collection.)

If I understand correctly, there is never any non-weak reference to
the bv above and so it can be collected at any time.

It's a bit of an “of course!” moment to realise that the pointer is
only a weak reference.

Thanks

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Ludovic Courtès]

Hi,

Daniel Hartwig <mandyke@gmail.com> skribis:

> On 7 October 2012 05:41, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi,
>>
>> Daniel Hartwig <mandyke@gmail.com> skribis:
>>
>>> (define x (bytevector->pointer (make-bytevector len 1)))
>>> (define a (pointer-address x))
>>> (display x)(newline)
>>> (my-guardian x)
>>> ;(my-guardian (pointer->bytevector x len))
>>> (set! x #f)
>>>
>>> (define (dump-struct)
>>>   (write (pointer->bytevector (make-pointer a) len))(newline))
>>
>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>> from the returned pointer object to the given bytevector.  So when the
>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>> the problem you’re observing.  (And no, guardians don’t protect objects
>> from garbage collection.)
>
> If I understand correctly, there is never any non-weak reference to
> the bv above and so it can be collected at any time.

There’s a weak reference from the pointer object to the bytevector.

Once that pointer object has been collected (as in the example above),
the bytevector can be collected anytime.

Thanks,
Ludo’.

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Daniel Hartwig]

On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>> from the returned pointer object to the given bytevector.  So when the
>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>> the problem you’re observing.  (And no, guardians don’t protect objects
>>> from garbage collection.)
>>
>> If I understand correctly, there is never any non-weak reference to
>> the bv above and so it can be collected at any time.
>
> There’s a weak reference from the pointer object to the bytevector.
>
> Once that pointer object has been collected (as in the example above),
> the bytevector can be collected anytime.

Right.  But then the pointer is being collected even though it remains
inside the guardian, in the example it is never extracted from there.

I'm not sure I follow. :-/

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Ludovic Courtès]

Hi,

Daniel Hartwig <mandyke@gmail.com> skribis:

> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>>> from the returned pointer object to the given bytevector.  So when the
>>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>>> the problem you’re observing.  (And no, guardians don’t protect objects
>>>> from garbage collection.)
>>>
>>> If I understand correctly, there is never any non-weak reference to
>>> the bv above and so it can be collected at any time.
>>
>> There’s a weak reference from the pointer object to the bytevector.
>>
>> Once that pointer object has been collected (as in the example above),
>> the bytevector can be collected anytime.
>
> Right.  But then the pointer is being collected even though it remains
> inside the guardian, in the example it is never extracted from there.

Well, when the object reaches the guardian’s zombie list, that’s because
it’s been finalized, so any weak references from that object can also be
nullified.

Anyway, guardians are not a mechanism to protect objects from being
GC’d.  To prevent the bytevector from being GC’d, you should either keep
the pointer object or the bytevector itself in non-GC’d memory, such as
a global variable or hash table.

How does it help?  Should we close the bug?  :-)

Thanks,
Ludo’.

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Daniel Hartwig]

On 8 October 2012 21:44, Ludovic Courtès <ludo@gnu.org> wrote:
>> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>> Right.  But then the pointer is being collected even though it remains
>> inside the guardian, in the example it is never extracted from there.
>
> Well, when the object reaches the guardian’s zombie list, that’s because
> it’s been finalized, so any weak references from that object can also be
> nullified.

Ah.  So I thought that being in the zombie list prevented any
finalization, thus when the guardian returns an object it is still
fully functional and only after the reference is lost again does it
really get finalized.

I will have to reread the doc for make-guardian where it talks about
the weak links, perhaps with a healthy amount of source-code
inspection ;-)

Admittedly the test case here is quite contrived, and I agreed with
your next remark in the original (pre-bug-report) thread.

> Anyway, guardians are not a mechanism to protect objects from being
> GC’d.  To prevent the bytevector from being GC’d, you should either keep
> the pointer object or the bytevector itself in non-GC’d memory, such as
> a global variable or hash table.

Yes.  The intended use was obviously troubled, but it still seemed odd
about the guardian's lack of protection.

> How does it help?  Should we close the bug?  :-)

Sure.  At least you seem convinced and you has actually hacked on it :-)

Regards

bug#12095: Protecting pointer on bytevector with guardian does not protect memory

[Ludovic Courtès]

Daniel Hartwig <mandyke@gmail.com> skribis:

> On 8 October 2012 21:44, Ludovic Courtès <ludo@gnu.org> wrote:
>>> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Right.  But then the pointer is being collected even though it remains
>>> inside the guardian, in the example it is never extracted from there.
>>
>> Well, when the object reaches the guardian’s zombie list, that’s because
>> it’s been finalized, so any weak references from that object can also be
>> nullified.
>
> Ah.  So I thought that being in the zombie list prevented any
> finalization, thus when the guardian returns an object it is still
> fully functional and only after the reference is lost again does it
> really get finalized.

Well, the object is still usable when the guardian returns it, because
it’s been kept alive by the finalizer (‘finalize_guarded’ in
guardians.c).

Now, whether weak references from the object are subject to
“nullification” by the GC is clearly a gray area, but I’m not surprised
that it is.

>> How does it help?  Should we close the bug?  :-)
>
> Sure.  At least you seem convinced and you has actually hacked on it :-)

Good.  :-)

So closing it now, but feel free to reopen if you think something’s wrong.

Thanks,
Ludo’.