* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
@ 2012-07-30 16:01 Patrick Bernaud
2012-08-06 16:37 ` Ian Price
0 siblings, 1 reply; 10+ messages in thread
From: Patrick Bernaud @ 2012-07-30 16:01 UTC (permalink / raw
To: 12095
[-- Attachment #1: message body and .signature --]
[-- Type: text/plain, Size: 658 bytes --]
The memory from a bytevector of which a pointer is taken (with
'bytevector->pointer') can be overwritten even if that pointer has
been put inside a guardian. 'make-c-struct' from (system foreign) is
using 'bytevector->pointer'.
With the test script attached:
$ guile -v | head -1
guile (GNU Guile) 2.0.6.8-cc26b9-dirty
$ guile --no-auto-compile -s test.scm
#<pointer 0x974648c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(110 103 45 108 101 110 103 116 104 0) <<<< memory overwrite with "ng-length\0" from module
#<pointer 0x974648c>
$
With auto compilation turned on, it looks like the problem can not be
reproduced.
--
Patrick Bernaud
[-- Attachment #2: test.scm --]
[-- Type: application/octet-stream, Size: 526 bytes --]
(use-modules (system foreign) (rnrs bytevectors))
(define my-guardian (make-guardian))
(define len 10)
(define x (bytevector->pointer (make-bytevector len 1)))
(define a (pointer-address x))
(display x)(newline)
(my-guardian x)
;(my-guardian (pointer->bytevector x len))
(set! x #f)
(write (pointer->bytevector (make-pointer a) len))(newline)
(gc)
(write (pointer->bytevector (make-pointer a) len))(newline)
(use-modules (htmlprag))
(write (pointer->bytevector (make-pointer a) len))(newline)
(display (my-guardian))(newline)
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-07-30 16:01 bug#12095: Protecting pointer on bytevector with guardian does not protect memory Patrick Bernaud
@ 2012-08-06 16:37 ` Ian Price
2012-08-07 3:32 ` Daniel Hartwig
0 siblings, 1 reply; 10+ messages in thread
From: Ian Price @ 2012-08-06 16:37 UTC (permalink / raw
To: Patrick Bernaud; +Cc: 12095
Patrick Bernaud <patrickb@chez.com> writes:
> With auto compilation turned on, it looks like the problem can not be
> reproduced.
I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
there was one caveat, as I didn't have html prag, I substituted for
various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
my own (pfds queues).
Is this bug somehow particular to htmlprag, or can you confirm it with others?
--
Ian Price
"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-08-06 16:37 ` Ian Price
@ 2012-08-07 3:32 ` Daniel Hartwig
2012-10-06 21:41 ` Ludovic Courtès
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Hartwig @ 2012-08-07 3:32 UTC (permalink / raw
To: Ian Price; +Cc: 12095
[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]
On 7 August 2012 00:37, Ian Price <ianprice90@googlemail.com> wrote:
> Patrick Bernaud <patrickb@chez.com> writes:
>
>> With auto compilation turned on, it looks like the problem can not be
>> reproduced.
That is to say, when using --fresh-auto-compile.
>
> I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
> 2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
> there was one caveat, as I didn't have html prag, I substituted for
> various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
> my own (pfds queues).
>
> Is this bug somehow particular to htmlprag, or can you confirm it with others?
On x86 Debian sid I reproduced the bug by loading (web server) with
the original test.scm from the mailing list.
The bug report copy differs from the original by replacing
parse-c-struct with pointer->bytevector.
I attach test1.scm, with a loop to load many modules one at a time.
Curiously, when using pointer->bytevector the contents change (for me)
only when freshly compiling:
$ guile --fresh-auto-compile test1.scm
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;; or pass the --no-auto-compile argument to disable.
;;; compiling /home/daniel/Downloads/test1.scm
;;; compiled /home/daniel/.cache/guile/ccache/2.0-LE-4-2.0/home/daniel/Downloads/test1.scm.go
#<pointer 0xa1cdb7c>
#vu8(1 1 1 1 1 1 1 1 1 1)
Contents differ after (web http)
#vu8(110 116 104 101 115 105 122 101 0 1)
#<pointer 0xa1cdb7c>
$ guile test1.scm
#<pointer 0x82c754c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#<pointer 0x82c754c>
With the original (using parse-c-struct) it was the other way around.
[-- Attachment #2: test1.scm --]
[-- Type: application/octet-stream, Size: 1037 bytes --]
(use-modules (system foreign) (rnrs bytevectors))
(define my-guardian (make-guardian))
(define len 10)
(define x (bytevector->pointer (make-bytevector len 1)))
(define a (pointer-address x))
(display x)(newline)
(my-guardian x)
;(my-guardian (pointer->bytevector x len))
(set! x #f)
(define (dump-struct)
(write (pointer->bytevector (make-pointer a) len))(newline))
(dump-struct)
(let lp ((s1 (with-output-to-string dump-struct))
(m '((web uri) (web http) (web request) (web response)
(web client) (web server) (sxml apply-templates)
(sxml fold) (sxml simple) (sxml ssax)
(ice-9 popen) (ice-9 getopt-long) (srfi srfi-42))))
(gc)
(primitive-eval `(use-modules ,(car m)))
(let ((s2 (with-output-to-string dump-struct)))
(if (not (string=? s1 s2))
(begin
(display "Contents differ after ")
(display (car m))
(newline)
(display s2))
(if (pair? (cdr m))
(lp s2 (cdr m))))))
(display (my-guardian))(newline)
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-08-07 3:32 ` Daniel Hartwig
@ 2012-10-06 21:41 ` Ludovic Courtès
2012-10-07 2:56 ` Daniel Hartwig
0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2012-10-06 21:41 UTC (permalink / raw
To: Daniel Hartwig; +Cc: 12095, Ian Price
Hi,
Daniel Hartwig <mandyke@gmail.com> skribis:
> (define x (bytevector->pointer (make-bytevector len 1)))
> (define a (pointer-address x))
> (display x)(newline)
> (my-guardian x)
> ;(my-guardian (pointer->bytevector x len))
> (set! x #f)
>
> (define (dump-struct)
> (write (pointer->bytevector (make-pointer a) len))(newline))
This is expected to fail: ‘bytevector->pointer’ creates a weak reference
from the returned pointer object to the given bytevector. So when the
pointer object is reclaimed, the bytevector can be reclaimed too, hence
the problem you’re observing. (And no, guardians don’t protect objects
from garbage collection.)
To put it differently, memory management is left to the user. The weak
reference I mention is a convenience for simple cases, but for “real
world” situations, one has to take all steps necessary to ensure that
the lifetime of C objects and that of their Scheme counterparts is in sync.
Hope this helps,
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-06 21:41 ` Ludovic Courtès
@ 2012-10-07 2:56 ` Daniel Hartwig
2012-10-07 20:38 ` Ludovic Courtès
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Hartwig @ 2012-10-07 2:56 UTC (permalink / raw
To: Ludovic Courtès; +Cc: 12095, Ian Price
On 7 October 2012 05:41, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi,
>
> Daniel Hartwig <mandyke@gmail.com> skribis:
>
>> (define x (bytevector->pointer (make-bytevector len 1)))
>> (define a (pointer-address x))
>> (display x)(newline)
>> (my-guardian x)
>> ;(my-guardian (pointer->bytevector x len))
>> (set! x #f)
>>
>> (define (dump-struct)
>> (write (pointer->bytevector (make-pointer a) len))(newline))
>
> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
> from the returned pointer object to the given bytevector. So when the
> pointer object is reclaimed, the bytevector can be reclaimed too, hence
> the problem you’re observing. (And no, guardians don’t protect objects
> from garbage collection.)
If I understand correctly, there is never any non-weak reference to
the bv above and so it can be collected at any time.
It's a bit of an “of course!” moment to realise that the pointer is
only a weak reference.
Thanks
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-07 2:56 ` Daniel Hartwig
@ 2012-10-07 20:38 ` Ludovic Courtès
2012-10-07 23:49 ` Daniel Hartwig
0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2012-10-07 20:38 UTC (permalink / raw
To: Daniel Hartwig; +Cc: 12095, Ian Price
Hi,
Daniel Hartwig <mandyke@gmail.com> skribis:
> On 7 October 2012 05:41, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi,
>>
>> Daniel Hartwig <mandyke@gmail.com> skribis:
>>
>>> (define x (bytevector->pointer (make-bytevector len 1)))
>>> (define a (pointer-address x))
>>> (display x)(newline)
>>> (my-guardian x)
>>> ;(my-guardian (pointer->bytevector x len))
>>> (set! x #f)
>>>
>>> (define (dump-struct)
>>> (write (pointer->bytevector (make-pointer a) len))(newline))
>>
>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>> from the returned pointer object to the given bytevector. So when the
>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>> the problem you’re observing. (And no, guardians don’t protect objects
>> from garbage collection.)
>
> If I understand correctly, there is never any non-weak reference to
> the bv above and so it can be collected at any time.
There’s a weak reference from the pointer object to the bytevector.
Once that pointer object has been collected (as in the example above),
the bytevector can be collected anytime.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-07 20:38 ` Ludovic Courtès
@ 2012-10-07 23:49 ` Daniel Hartwig
2012-10-08 13:44 ` Ludovic Courtès
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Hartwig @ 2012-10-07 23:49 UTC (permalink / raw
To: Ludovic Courtès; +Cc: 12095, Ian Price
On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>> from the returned pointer object to the given bytevector. So when the
>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>> the problem you’re observing. (And no, guardians don’t protect objects
>>> from garbage collection.)
>>
>> If I understand correctly, there is never any non-weak reference to
>> the bv above and so it can be collected at any time.
>
> There’s a weak reference from the pointer object to the bytevector.
>
> Once that pointer object has been collected (as in the example above),
> the bytevector can be collected anytime.
Right. But then the pointer is being collected even though it remains
inside the guardian, in the example it is never extracted from there.
I'm not sure I follow. :-/
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-07 23:49 ` Daniel Hartwig
@ 2012-10-08 13:44 ` Ludovic Courtès
2012-10-08 14:10 ` Daniel Hartwig
0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2012-10-08 13:44 UTC (permalink / raw
To: Daniel Hartwig; +Cc: 12095, Ian Price
Hi,
Daniel Hartwig <mandyke@gmail.com> skribis:
> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>>> from the returned pointer object to the given bytevector. So when the
>>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>>> the problem you’re observing. (And no, guardians don’t protect objects
>>>> from garbage collection.)
>>>
>>> If I understand correctly, there is never any non-weak reference to
>>> the bv above and so it can be collected at any time.
>>
>> There’s a weak reference from the pointer object to the bytevector.
>>
>> Once that pointer object has been collected (as in the example above),
>> the bytevector can be collected anytime.
>
> Right. But then the pointer is being collected even though it remains
> inside the guardian, in the example it is never extracted from there.
Well, when the object reaches the guardian’s zombie list, that’s because
it’s been finalized, so any weak references from that object can also be
nullified.
Anyway, guardians are not a mechanism to protect objects from being
GC’d. To prevent the bytevector from being GC’d, you should either keep
the pointer object or the bytevector itself in non-GC’d memory, such as
a global variable or hash table.
How does it help? Should we close the bug? :-)
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-08 13:44 ` Ludovic Courtès
@ 2012-10-08 14:10 ` Daniel Hartwig
2012-10-08 15:42 ` Ludovic Courtès
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Hartwig @ 2012-10-08 14:10 UTC (permalink / raw
To: Ludovic Courtès; +Cc: 12095, Ian Price
On 8 October 2012 21:44, Ludovic Courtès <ludo@gnu.org> wrote:
>> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>> Right. But then the pointer is being collected even though it remains
>> inside the guardian, in the example it is never extracted from there.
>
> Well, when the object reaches the guardian’s zombie list, that’s because
> it’s been finalized, so any weak references from that object can also be
> nullified.
Ah. So I thought that being in the zombie list prevented any
finalization, thus when the guardian returns an object it is still
fully functional and only after the reference is lost again does it
really get finalized.
I will have to reread the doc for make-guardian where it talks about
the weak links, perhaps with a healthy amount of source-code
inspection ;-)
Admittedly the test case here is quite contrived, and I agreed with
your next remark in the original (pre-bug-report) thread.
> Anyway, guardians are not a mechanism to protect objects from being
> GC’d. To prevent the bytevector from being GC’d, you should either keep
> the pointer object or the bytevector itself in non-GC’d memory, such as
> a global variable or hash table.
Yes. The intended use was obviously troubled, but it still seemed odd
about the guardian's lack of protection.
> How does it help? Should we close the bug? :-)
Sure. At least you seem convinced and you has actually hacked on it :-)
Regards
^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#12095: Protecting pointer on bytevector with guardian does not protect memory
2012-10-08 14:10 ` Daniel Hartwig
@ 2012-10-08 15:42 ` Ludovic Courtès
0 siblings, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2012-10-08 15:42 UTC (permalink / raw
To: Daniel Hartwig; +Cc: 12095-done, Ian Price
Daniel Hartwig <mandyke@gmail.com> skribis:
> On 8 October 2012 21:44, Ludovic Courtès <ludo@gnu.org> wrote:
>>> On 8 October 2012 04:38, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Right. But then the pointer is being collected even though it remains
>>> inside the guardian, in the example it is never extracted from there.
>>
>> Well, when the object reaches the guardian’s zombie list, that’s because
>> it’s been finalized, so any weak references from that object can also be
>> nullified.
>
> Ah. So I thought that being in the zombie list prevented any
> finalization, thus when the guardian returns an object it is still
> fully functional and only after the reference is lost again does it
> really get finalized.
Well, the object is still usable when the guardian returns it, because
it’s been kept alive by the finalizer (‘finalize_guarded’ in
guardians.c).
Now, whether weak references from the object are subject to
“nullification” by the GC is clearly a gray area, but I’m not surprised
that it is.
>> How does it help? Should we close the bug? :-)
>
> Sure. At least you seem convinced and you has actually hacked on it :-)
Good. :-)
So closing it now, but feel free to reopen if you think something’s wrong.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2012-10-08 15:42 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-07-30 16:01 bug#12095: Protecting pointer on bytevector with guardian does not protect memory Patrick Bernaud
2012-08-06 16:37 ` Ian Price
2012-08-07 3:32 ` Daniel Hartwig
2012-10-06 21:41 ` Ludovic Courtès
2012-10-07 2:56 ` Daniel Hartwig
2012-10-07 20:38 ` Ludovic Courtès
2012-10-07 23:49 ` Daniel Hartwig
2012-10-08 13:44 ` Ludovic Courtès
2012-10-08 14:10 ` Daniel Hartwig
2012-10-08 15:42 ` Ludovic Courtès
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).