crash in guile-sqlite3

crash in guile-sqlite3

[Joonas Sarajärvi]

Hello,

Some time ago, I noticed that there is a nice-looking libsqlite3
wrapper for Guile available at [1]. I am now trying to use it, but the
short example program I wrote seems to cause guile to crash in certain
places where sqlite-finalize is called.

I tried to run guile in gdb to get a backtrace from the crash. The
segfault seems to be triggered inside libsqlite3 code, but I could not
really get much useful information from it. I am thus writing here to
ask if someone else has more insight on why this is happening and if
it could be resolved.

The test program that demostrates this behavior is available in [2]
and it is also attached to this message. On a 32-bit Exherbo
installation with guile-2.0.5 it reproducibly crashes into "Illegal
instruction" message, while in a 64-bit Fedora 16 installation that
has guile-2.0.2 added it produces a Segmentation fault. The program is
fairly short, which I hope might make it easier to track the problem
down.

Is guile-sqlite3 actually in such a shape that it might be useful for
curious testers?

Thanks,
-- Joonas Sarajärvi

References:
[1] http://gitorious.org/guile-sqlite3
[2] http://paste.pocoo.org/show/587676/

Re: crash in guile-sqlite3

[rixed]

> Is guile-sqlite3 actually in such a shape that it might be useful for
> curious testers?

I use it daily and it's working allright.
I'd have a closer look to the libffi you used to compile guile
if I were you.

Re: crash in guile-sqlite3

[Noah Lavine]

> I use it daily and it's working allright.
> I'd have a closer look to the libffi you used to compile guile
> if I were you.

What version of Guile do you use it with?

Just curious if it's different than 2.0.2 or 2.0.5.

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

Hi, Joonas;

2012/4/26 Joonas Sarajärvi <muep@iki.fi>
>
> I tried to run guile in gdb to get a backtrace from the crash. The
> segfault seems to be triggered inside libsqlite3 code, but I could not
> really get much useful information from it. I am thus writing here to
> ask if someone else has more insight on why this is happening and if
> it could be resolved.
>

Same crash on Guile 2.0.5. It looks like string encoding problem. Suppose
name be an integer not a string like 10, it would not crash. sqlite-bind
of Guile-SQLite3 will deliver this 10 to sqlite3_bind_int64 of libsqlite3.
If a string, it will use sqlite3_bind_text of libsqlite3. Before
delivering, it will encoding this string to utf-8 string pointer because
sqlite3_bind_text need to know string pointer and it's length. I think you
or Andy may look around that.

[-- Attachment #2: Type: text/html, Size: 1382 bytes --]

Re: crash in guile-sqlite3

[rixed]

-[ Thu, Apr 26, 2012 at 05:04:23PM -0400, Noah Lavine ]----
> > I use it daily and it's working allright.
> > I'd have a closer look to the libffi you used to compile guile
> > if I were you.
> 
> What version of Guile do you use it with?

2.0.2.224-b1a52-dirty compiled with libffi 5.0.10

Re: crash in guile-sqlite3

[David Pirotte]

Hi,

> > I tried to run guile in gdb to get a backtrace from the crash. The
> > segfault seems to be triggered inside libsqlite3 code, but I could not
> > ...

> Same crash on Guile 2.0.5. It looks like string encoding problem. Suppose
> name be an integer not a string like 10, it would not crash. sqlite-bind
> ...

Best would be to send us an small, complete and concise example of you consider
reproducible, could you d that ?

David

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

Hi, David;

2012/4/28 David Pirotte <david@altosw.be>
>
> Best would be to send us an small, complete and concise example of you
> consider
> reproducible, could you d that ?
>

I mean a example code of Joonas Sarajärvi,
http://paste.pocoo.org/show/587676/ , produces a Segmentation fault on
Guile 2.0.5 too. I though what if a integer. In the integer case, there is
no Segmentation fault. Here is a diff of my test code from Joonas's:

--- ex0.scm.old 2012-04-28 04:41:23.545825725 +0900
+++ ex0.scm 2012-04-28 04:51:47.065231945 +0900
@@ -16,16 +16,16 @@
   (sqlite-finalize stmt))

 (display "creating\n")
-(let* ((sql "CREATE TABLE foos(dbid INTEGER PRIMARY KEY, name TEXT);")
+(let* ((sql "CREATE TABLE foos(dbid INTEGER PRIMARY KEY, name INTEGER);")
        (stmt (sqlite-prepare db sql)))
   (sqlite-step stmt)
   (sqlite-finalize stmt))

 ;; Insert some rows of data.
 (display "inserting\n")
-(let ((name "myfoo")
+(let ((name 10)
       (stmt (sqlite-prepare db "INSERT INTO foos(name) VALUES(?);")))
-  (display (string-append "for \"" name "\":\n"))
+  (display (string-append "for " (number->string name) ":\n"))
   (display "binding\n")
   (sqlite-bind stmt 1 name)
   (display "stepping\n")
@@ -34,7 +34,7 @@
   ;; This finalize call in particular seems to break things
   (sqlite-finalize stmt)
   ;; This seems to be never reached.
-  (display "finalized\n")))
+  (display "finalized\n"))

 (display "closing\n")
 (sqlite-close db)
\ No newline at end of file

The differences are only string to integer but Joonas's code produces a
Segmentation fault. Why? So, I read the sqlite3.scm, Andy's Guile-SQLite3.
And then, I realized sqlite-bind will call different functions; bind-int64
and bind-text. Joonas asked the insight on why this is happening. It would
be best if I be able to solve that for Joonas.

[-- Attachment #2: Type: text/html, Size: 3646 bytes --]

Re: crash in guile-sqlite3

[David Pirotte]

Hi,
 
> I mean a example code of Joonas Sarajärvi,
> http://paste.pocoo.org/show/587676/ , produces a Segmentation fault on
> Guile 2.0.5 too. I though what if a integer. In the integer case, there is
> no Segmentation fault. Here is a diff of my test code from Joonas's:

the paste seems unavalaible, could you send the code within the email, it is rather
small anyway... 

d.

Re: crash in guile-sqlite3

[David Pirotte]

hello,

> ...
>        (stmt (sqlite-prepare db "INSERT INTO foos(name) VALUES(?);")))

i'd rather bind the statment in scheme _anyway_ if i was you.

david

;; --

(use-modules (sqlite3))

(define db (sqlite-open "ex0.db"))

(define (sqlite/command db command)
  (let ((stmt (sqlite-prepare db command)))
    (sqlite-step stmt)
    (sqlite-finalize stmt)
    (if #f #f)))

(define (my-insert new-name)
  (sqlite/command db (format #f "INSERT INTO foos(name) VALUES('~A')" new-name)))

(sqlite/command db "DROP TABLE IF EXISTS foos;")
(sqlite/command db "CREATE TABLE foos(dbid INTEGER PRIMARY KEY, name TEXT);")

(for-each (lambda (name)
	    (my-insert name))
    '("foo-1" "foo-2" "foo-3"))

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 2497 bytes --]

Hi, Joonas;

Yesterday, David said simple solution of yours. I think I found the case
of segfault.

########## cut here ##########
--- sqlite3.scm.old 2012-01-24 00:22:10.000000000 +0900
+++ sqlite3.scm 2012-04-29 10:37:37.999722092 +0900
@@ -284,11 +284,11 @@
   (let ((bind-blob (pointer->procedure
                     int
                     (dynamic-func "sqlite3_bind_blob" libsqlite3)
-                    (list '* int '* int '*)))
+                    (list '* int '* int int)))
         (bind-text (pointer->procedure
                     int
                     (dynamic-func "sqlite3_bind_text" libsqlite3)
-                    (list '* int '* int '*)))
+                    (list '* int '* int int)))
         (bind-int64 (pointer->procedure
                      int
                      (dynamic-func "sqlite3_bind_int64" libsqlite3)
@@ -301,8 +301,7 @@
                     int
                     (dynamic-func "sqlite3_bind_null" libsqlite3)
                     (list '* int)))
-        (sqlite-transient (bytevector->pointer
-                           (make-bytevector (sizeof '*) #xff))))
+        (sqlite-transient -1))
     (lambda (stmt key val)
       (assert-live-stmt! stmt)
       (let ((idx (key->index stmt key))
########## cut here ##########

2012/4/27 Sunjoong Lee <sunjoong@gmail.com>
>
> Same crash on Guile 2.0.5. It looks like string encoding problem. Suppose
> name be an integer not a string like 10, it would not crash. sqlite-bind
> of Guile-SQLite3 will deliver this 10 to sqlite3_bind_int64 of libsqlite3.
> If a string, it will use sqlite3_bind_text of libsqlite3. Before
> delivering, it will encoding this string to utf-8 string pointer because
> sqlite3_bind_text need to know string pointer and it's length. I think you
> or Andy may look around that.
>

 In the http://www.sqlite.org/c3ref/c_static.html ,
#define SQLITE_TRANSIENT   ((sqlite3_destructor_type)-1)

I.e, SQLITE_TRANSIENT is actually -1, not some position of
memory. sqlite-bind of Guile-SQLite3 had treated it as a pointer, an
address of memory position.

Oh, another one; your test code ex0.scm has a bug.

--- ex0.scm.old 2012-04-29 10:57:09.717398060 +0900
+++ ex0.scm 2012-04-29 10:39:20.051261614 +0900
@@ -34,7 +34,7 @@
   ;; This finalize call in particular seems to break things
   (sqlite-finalize stmt)
   ;; This seems to be never reached.
-  (display "finalized\n")))
+  (display "finalized\n"))

 (display "closing\n")
 (sqlite-close db)
\ No newline at end of file

[-- Attachment #2: Type: text/html, Size: 4028 bytes --]

Re: crash in guile-sqlite3

[Andy Wingo]

On Sun 29 Apr 2012 03:58, Sunjoong Lee <sunjoong@gmail.com> writes:

> Yesterday, David said simple solution of yours. I think I found the case of segfault.

Nice!

Would you mind sending a pull request on gitorious?  Preferably we could
add a test case too.

Cheers,

Andy
-- 
http://wingolog.org/

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

2012/4/30 Andy Wingo <wingo@pobox.com>
>
> Would you mind sending a pull request on gitorious?  Preferably we could
> add a test case too.
>

 I apologize for my poor english; I'm willing to "send a pull request on
gitorious" but don't know what means that sentence.

Joonas had reported that Segmentation fault crash will be occur
in guile-sqlite3 when you use sqlite-bind with a string; I think he would
suspect it  as sqlite-finalize's problem.

I realized sqlite-bind treats sqlite-transient as a pointer and it would
occur same Segmentation fault when you with bytevector; only numerical
value will work well. That's because sqlite3_bind_blob
and sqlite3_bind_text of libsqlite3 expect it's last argument as a function
pointer or just uint64_t type of 0x00000000 or 0xffffffff; you can use C
function if sqlite-transient be it but you should use 0x00000000 or
0xffffffff if not so, and the meaning of SQLITE_TRANSIENT is it's a
0xffffffff.

Joonas, would you mind sending a pull request on gitorious?

[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: crash in guile-sqlite3

[Andy Wingo]

On Mon 30 Apr 2012 12:32, Sunjoong Lee <sunjoong@gmail.com> writes:

> 2012/4/30 Andy Wingo <wingo@pobox.com>
>
>     Would you mind sending a pull request on gitorious?  Preferably we could
>     add a test case too.
>
>  I apologize for my poor english; I'm willing to "send a pull request on gitorious" but don't know what means that sentence.

No problem, and no need to apologize :)

What it means is to go to:

  https://gitorious.org/guile-sqlite3

Create an account and log in.  Then go back to:

  https://gitorious.org/guile-sqlite3

And click "Clone this repository".  That will give you a Git repository
for guile-sqlite3, on gitorious.

Then, you do the "git clone ssh://....", make your change, "git commit"
it locally, then "git push" to the origin.

When that is done, there is a button in the web interface to "request a
pull", meaning, request that dzu (or me) merges your changes to the
"main repository".

There are other ways to do this, but if you are willing to try this one,
it is the best.

Let me know if you have questions, and thanks for hacking Guile!

Andy
-- 
http://wingolog.org/

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Hi, Andy;

Where is a "request a pull" button?

I tried somethings; one of them makes 3e2add75. I don't know what's exactly
work but https://gitorious.org/guile-sqlite3 display "sunjoong committed
3e2add75 on April 30, 2012 13:23". Now, the time to "pull request."

2012/4/30 Andy Wingo <wingo@pobox.com>
>
> What it means is to go to:
>
>  https://gitorious.org/guile-sqlite3
>
> Create an account and log in.  Then go back to:
>
>  https://gitorious.org/guile-sqlite3
>
> And click "Clone this repository".  That will give you a Git repository
> for guile-sqlite3, on gitorious.
>
> Then, you do the "git clone ssh://....", make your change, "git commit"
> it locally, then "git push" to the origin.
>
> When that is done, there is a button in the web interface to "request a
> pull", meaning, request that dzu (or me) merges your changes to the
> "main repository".
>

Thanks.

[-- Attachment #2: Type: text/html, Size: 1566 bytes --]

Re: crash in guile-sqlite3

[Joonas Sarajärvi]

2012/4/29 Sunjoong Lee <sunjoong@gmail.com>:
> Hi, Joonas;
>
> Yesterday, David said simple solution of yours. I think I found the case
> of segfault.
>

Hello,

Sorry for having been so inactive in this thread, despite starting it.
Other life activities pretty much took the whole weekend.

However, thank you for taking a look at the problem. The change you
placed at git://gitorious.org/~sunjoong/guile-sqlite3/sunjoongs-guile-sqlite3.git
seems to completely resolve the crash for me.

-Joonas

Re: crash in guile-sqlite3

[Joonas Sarajärvi]

2012/5/1 Joonas Sarajärvi <muep@iki.fi>:

> However, thank you for taking a look at the problem. The change you
> placed at git://gitorious.org/~sunjoong/guile-sqlite3/sunjoongs-guile-sqlite3.git
> seems to completely resolve the crash for me.
>
> -Joonas

Replying to myself, I now noticed that the change will not work in a
64-bit computer. My impression is that this is due to the fact that
the 5th parameter taken by sqlite3_bind_text function is really a
pointer and not an int. On amd64 computers, int are 4 bytes wide while
pointers are 8 bytes.

I also think that the older way of passing SQLITE_TRANSIENT as a
pointer should work. However, there was a small bug in the code that
constructed the pointer. I made a yet-another clone of the main
guile-sqlite3 repository and pushed a change that I think fixes this.
The change I did is also here:

diff --git a/sqlite3.scm b/sqlite3.scm
index fcc1fdf..5047e71 100644
--- a/sqlite3.scm
+++ b/sqlite3.scm
@@ -301,8 +301,9 @@
                     int
                     (dynamic-func "sqlite3_bind_null" libsqlite3)
                     (list '* int)))
-        (sqlite-transient (bytevector->pointer
-                           (make-bytevector (sizeof '*) #xff))))
+        (sqlite-transient (dereference-pointer
+                           (bytevector->pointer
+                            (make-bytevector (sizeof '*) #xff)))))
     (lambda (stmt key val)
       (assert-live-stmt! stmt)
       (let ((idx (key->index stmt key))


The test case by Sunjoong Lee is also cherry-picked to the git branch
I placed on gitorious. Hopefully it was acceptable to include it.
Should I make a merge request of this?

-Joonas

Re: crash in guile-sqlite3

[Daniel Krueger]

Hi,

I just wanted to note that it should work just to use (make-pointer #xff).

- Daniel

On Tue, May 1, 2012 at 5:01 PM, Joonas Sarajärvi <muep@iki.fi> wrote:
> 2012/5/1 Joonas Sarajärvi <muep@iki.fi>:
>
>> However, thank you for taking a look at the problem. The change you
>> placed at git://gitorious.org/~sunjoong/guile-sqlite3/sunjoongs-guile-sqlite3.git
>> seems to completely resolve the crash for me.
>>
>> -Joonas
>
> Replying to myself, I now noticed that the change will not work in a
> 64-bit computer. My impression is that this is due to the fact that
> the 5th parameter taken by sqlite3_bind_text function is really a
> pointer and not an int. On amd64 computers, int are 4 bytes wide while
> pointers are 8 bytes.
>
> I also think that the older way of passing SQLITE_TRANSIENT as a
> pointer should work. However, there was a small bug in the code that
> constructed the pointer. I made a yet-another clone of the main
> guile-sqlite3 repository and pushed a change that I think fixes this.
> The change I did is also here:
>
> diff --git a/sqlite3.scm b/sqlite3.scm
> index fcc1fdf..5047e71 100644
> --- a/sqlite3.scm
> +++ b/sqlite3.scm
> @@ -301,8 +301,9 @@
>                     int
>                     (dynamic-func "sqlite3_bind_null" libsqlite3)
>                     (list '* int)))
> -        (sqlite-transient (bytevector->pointer
> -                           (make-bytevector (sizeof '*) #xff))))
> +        (sqlite-transient (dereference-pointer
> +                           (bytevector->pointer
> +                            (make-bytevector (sizeof '*) #xff)))))
>     (lambda (stmt key val)
>       (assert-live-stmt! stmt)
>       (let ((idx (key->index stmt key))
>
>
> The test case by Sunjoong Lee is also cherry-picked to the git branch
> I placed on gitorious. Hopefully it was acceptable to include it.
> Should I make a merge request of this?
>
> -Joonas
>

Re: crash in guile-sqlite3

[Daniel Krueger]

Oh, I'm sorry, I missed the fact that all bits in the pointer would be
set. Correct would be this:

(make-pointer (bit-extract (lognot 0) 0 (* 8 (sizeof '*))))

So, okay, this may not be a better solution :D

- Daniel

On Tue, May 1, 2012 at 5:05 PM, Daniel Krueger <keenbug@googlemail.com> wrote:
> Hi,
>
> I just wanted to note that it should work just to use (make-pointer #xff).
>
> - Daniel
>
> On Tue, May 1, 2012 at 5:01 PM, Joonas Sarajärvi <muep@iki.fi> wrote:
>> 2012/5/1 Joonas Sarajärvi <muep@iki.fi>:
>>
>>> However, thank you for taking a look at the problem. The change you
>>> placed at git://gitorious.org/~sunjoong/guile-sqlite3/sunjoongs-guile-sqlite3.git
>>> seems to completely resolve the crash for me.
>>>
>>> -Joonas
>>
>> Replying to myself, I now noticed that the change will not work in a
>> 64-bit computer. My impression is that this is due to the fact that
>> the 5th parameter taken by sqlite3_bind_text function is really a
>> pointer and not an int. On amd64 computers, int are 4 bytes wide while
>> pointers are 8 bytes.
>>
>> I also think that the older way of passing SQLITE_TRANSIENT as a
>> pointer should work. However, there was a small bug in the code that
>> constructed the pointer. I made a yet-another clone of the main
>> guile-sqlite3 repository and pushed a change that I think fixes this.
>> The change I did is also here:
>>
>> diff --git a/sqlite3.scm b/sqlite3.scm
>> index fcc1fdf..5047e71 100644
>> --- a/sqlite3.scm
>> +++ b/sqlite3.scm
>> @@ -301,8 +301,9 @@
>>                     int
>>                     (dynamic-func "sqlite3_bind_null" libsqlite3)
>>                     (list '* int)))
>> -        (sqlite-transient (bytevector->pointer
>> -                           (make-bytevector (sizeof '*) #xff))))
>> +        (sqlite-transient (dereference-pointer
>> +                           (bytevector->pointer
>> +                            (make-bytevector (sizeof '*) #xff)))))
>>     (lambda (stmt key val)
>>       (assert-live-stmt! stmt)
>>       (let ((idx (key->index stmt key))
>>
>>
>> The test case by Sunjoong Lee is also cherry-picked to the git branch
>> I placed on gitorious. Hopefully it was acceptable to include it.
>> Should I make a merge request of this?
>>
>> -Joonas
>>

Re: crash in guile-sqlite3

[Sunjoong Lee]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

Hi, Daniel and Joonas;

2012/5/1 Daniel Krueger <keenbug@googlemail.com>
>
> (make-pointer (bit-extract (lognot 0) 0 (* 8 (sizeof '*))))
>

Good and thank you. I made a new patch based on that.

> On Tue, May 1, 2012 at 5:01 PM, Joonas Sarajärvi <address@hidden> wrote:
> >> Replying to myself, I now noticed that the change will not work in a
> >> 64-bit computer. My impression is that this is due to the fact that
> >> the 5th parameter taken by sqlite3_bind_text function is really a
> >> pointer and not an int. On amd64 computers, int are 4 bytes wide while
> >> pointers are 8 bytes.
>

Sorry for my introducing a new bug. In the most platform, pointers are 8
byte unsigned integer. I had thought you would forget making a merge
request on gitorious and made a new patch.

Joonas, test my new one on your 64-bit computer and/or make a merge request
of yours unless it work.

[-- Attachment #2: Type: text/html, Size: 1663 bytes --]

Re: crash in guile-sqlite3

[Joonas Sarajärvi]

Hello,

2012/5/1 Sunjoong Lee <sunjoong@gmail.com>:

> Joonas, test my new one on your 64-bit computer and/or make a merge request
> of yours unless it work.
>

I tried it now, and the version in commit
c55adf43a8c0e59706f40e65a07d457d60e55a83 seems to work fine in
computers of different sizes of pointer widths.

-Joonas