sqlite update causes failure of python-sqlalchemy

sqlite update causes failure of python-sqlalchemy

[Pjotr Prins]

Patch b24765139c8940541b23f84592d3580d53f71d71

 (define-public sqlite
   (package
    (name "sqlite")
-   (version "3.8.11.1")
+   (version "3.10.0")
    (source (origin

is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
regressing to the original sqlite package. Since the python binding is
part of the interpreter, I suspect there may be more python modules
vulnerable. I updated python-sqlalchemy to latest and that makes no
difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.

What do we do? Revert on this sqlite patch for the new guix release?
Or add a second sqlite package and have that as a python dependency?

Or live with the fact that python-sqlalchemy is broken for now. Maybe
it is possible to override the sqlite bindings to an earlier version,
but even if that worked I would think it tricky.

Personally I think we should revert. But then so many packages depend
on sqlite. Tricky.

Pj.

Re: sqlite update causes failure of python-sqlalchemy

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> Patch b24765139c8940541b23f84592d3580d53f71d71
>
>  (define-public sqlite
>    (package
>     (name "sqlite")
> -   (version "3.8.11.1")
> +   (version "3.10.0")
>     (source (origin
>
> is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
> regressing to the original sqlite package. Since the python binding is
> part of the interpreter, I suspect there may be more python modules
> vulnerable. I updated python-sqlalchemy to latest and that makes no
> difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.
>
> What do we do? Revert on this sqlite patch for the new guix release?
> Or add a second sqlite package and have that as a python dependency?

I would do the latter, assuming that soon a new python-sqlalchemy
release would solve the problem.  WDYT?

This is probably OK since python-sqlalchemy is a leaf, and so we’re
unlikely to end up mixing two different SQLite versions.

Ludo’.

[PATCH] Add and use sqlite-legacy-for-python

[Christopher Allan Webber]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]

Ludovic Courtès writes:

> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>
>> Patch b24765139c8940541b23f84592d3580d53f71d71
>>
>>  (define-public sqlite
>>    (package
>>     (name "sqlite")
>> -   (version "3.8.11.1")
>> +   (version "3.10.0")
>>     (source (origin
>>
>> is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
>> regressing to the original sqlite package. Since the python binding is
>> part of the interpreter, I suspect there may be more python modules
>> vulnerable. I updated python-sqlalchemy to latest and that makes no
>> difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.
>>
>> What do we do? Revert on this sqlite patch for the new guix release?
>> Or add a second sqlite package and have that as a python dependency?
>
> I would do the latter, assuming that soon a new python-sqlalchemy
> release would solve the problem.  WDYT?
>
> This is probably OK since python-sqlalchemy is a leaf, and so we’re
> unlikely to end up mixing two different SQLite versions.
>
> Ludo’.

Will sqlalchemy really remain a leaf node?  I hope not, since I'm
working on packaging MediaGoblin now :)

Regardless, I agree that the second approach seems to be the right one.
I've built a modified package, sqlite-legacy-for-python, and put it to
use.  I built it and confirmed it builds fine and that the tests pass,
and with it, the tests pass in python-sqlalchemy too.

Ok to push?


[-- Attachment #2: 0001-gnu-Add-sqlite-legacy-for-python-and-use-it-in-pytho.patch --]
[-- Type: text/x-patch, Size: 3333 bytes --]

From 20b9e3ac19b77b33f88d45e33ad9ceafacbdb1bd Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Fri, 12 Feb 2016 16:09:52 -0800
Subject: [PATCH] gnu: Add sqlite-legacy-for-python and use it in python's
 definition.

* gnu/packages/database.scm (sqlite-legacy-for-python): New variable.
* gnu/packages/python.scm (python): Use sqlite-legacy-for-python.
---
 gnu/packages/databases.scm | 30 ++++++++++++++++++++++++++++++
 gnu/packages/python.scm    |  3 ++-
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index b36f5d8..c00cbb4 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2015 Sou Bunnbu <iyzsong@gmail.com>
 ;;; Copyright © 2015 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Christopher Allan Webber <cwebber@dustycloud.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -388,6 +389,35 @@ widely deployed SQL database engine in the world.  The source code for SQLite
 is in the public domain.")
    (license public-domain)))
 
+;; Python packages (well, at least python-sqlalchemy) break with the latest
+;; release of sqlite, so we need to keep this around for legacy purposes...
+(define-public sqlite-legacy-for-python
+  (package
+    (inherit sqlite)
+    (version "3.8.11.1")
+    ;; Repeating a lot of the above definition... there's no way around it!
+    (source (origin
+              (method url-fetch)
+              (uri (let ((numeric-version
+                          (match (string-split version #\.)
+                            ((first-digit other-digits ...)
+                             (string-append first-digit
+                                            (string-pad-right
+                                             (string-concatenate
+                                              (map (cut string-pad <> 2 #\0)
+                                                   other-digits))
+                                             6 #\0))))))
+                     (list
+                      (string-append
+                       "https://fossies.org/linux/misc/sqlite-autoconf-"
+                       numeric-version ".tar.gz")
+                      (string-append
+                       "http://distfiles.gentoo.org/distfiles/"
+                       "/sqlite-autoconf-" numeric-version ".tar.gz"))))
+              (sha256
+               (base32
+                "1dnkl4qr1dgaprbyf3jddfiynkhxnin86qabni47wjlc0fnb16gv"))))))
+
 (define-public tdb
   (package
     (name "tdb")
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 3758eed..8d6f520 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -233,7 +233,8 @@
      `(("bzip2" ,bzip2)
        ("gdbm" ,gdbm)
        ("libffi" ,libffi)                         ; for ctypes
-       ("sqlite" ,sqlite)                         ; for sqlite extension
+       ;; Hopefully soon we'll have this back at vanilla sqlite
+       ("sqlite" ,sqlite-legacy-for-python)       ; for sqlite extension
        ("openssl" ,openssl)
        ("readline" ,readline)
        ("zlib" ,zlib)
-- 
2.6.3

Re: [PATCH] Add and use sqlite-legacy-for-python

[Thompson, David]

On Fri, Feb 12, 2016 at 7:13 PM, Christopher Allan Webber
<cwebber@dustycloud.org> wrote:
> Ludovic Courtès writes:
>
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>
>>> Patch b24765139c8940541b23f84592d3580d53f71d71
>>>
>>>  (define-public sqlite
>>>    (package
>>>     (name "sqlite")
>>> -   (version "3.8.11.1")
>>> +   (version "3.10.0")
>>>     (source (origin
>>>
>>> is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
>>> regressing to the original sqlite package. Since the python binding is
>>> part of the interpreter, I suspect there may be more python modules
>>> vulnerable. I updated python-sqlalchemy to latest and that makes no
>>> difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.
>>>
>>> What do we do? Revert on this sqlite patch for the new guix release?
>>> Or add a second sqlite package and have that as a python dependency?
>>
>> I would do the latter, assuming that soon a new python-sqlalchemy
>> release would solve the problem.  WDYT?
>>
>> This is probably OK since python-sqlalchemy is a leaf, and so we’re
>> unlikely to end up mixing two different SQLite versions.
>>
>> Ludo’.
>
> Will sqlalchemy really remain a leaf node?  I hope not, since I'm
> working on packaging MediaGoblin now :)

Yeah, sqlalchemy being a leaf node is accidental.  It's a library that
will be depended on by MediaGoblin and maybe other software.

> Regardless, I agree that the second approach seems to be the right one.
> I've built a modified package, sqlite-legacy-for-python, and put it to
> use.  I built it and confirmed it builds fine and that the tests pass,
> and with it, the tests pass in python-sqlalchemy too.

I'm concerned about this.  What exactly is being used here, a client
library?  If so, it means that we may have an issue when a python
application uses a library that wants to dynamically link against both
the normal sqlite library and this older version.  Maybe this is still
fine, but proceed with caution.

- Dave

Re: [PATCH] Add and use sqlite-legacy-for-python

[Christopher Allan Webber]

Thompson, David writes:

> On Fri, Feb 12, 2016 at 7:13 PM, Christopher Allan Webber
> <cwebber@dustycloud.org> wrote:
>> Ludovic Courtès writes:
>>
>>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>>
>>>> Patch b24765139c8940541b23f84592d3580d53f71d71
>>>>
>>>>  (define-public sqlite
>>>>    (package
>>>>     (name "sqlite")
>>>> -   (version "3.8.11.1")
>>>> +   (version "3.10.0")
>>>>     (source (origin
>>>>
>>>> is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
>>>> regressing to the original sqlite package. Since the python binding is
>>>> part of the interpreter, I suspect there may be more python modules
>>>> vulnerable. I updated python-sqlalchemy to latest and that makes no
>>>> difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.
>>>>
>>>> What do we do? Revert on this sqlite patch for the new guix release?
>>>> Or add a second sqlite package and have that as a python dependency?
>>>
>>> I would do the latter, assuming that soon a new python-sqlalchemy
>>> release would solve the problem.  WDYT?
>>>
>>> This is probably OK since python-sqlalchemy is a leaf, and so we’re
>>> unlikely to end up mixing two different SQLite versions.
>>>
>>> Ludo’.
>>
>> Will sqlalchemy really remain a leaf node?  I hope not, since I'm
>> working on packaging MediaGoblin now :)
>
> Yeah, sqlalchemy being a leaf node is accidental.  It's a library that
> will be depended on by MediaGoblin and maybe other software.
>
>> Regardless, I agree that the second approach seems to be the right one.
>> I've built a modified package, sqlite-legacy-for-python, and put it to
>> use.  I built it and confirmed it builds fine and that the tests pass,
>> and with it, the tests pass in python-sqlalchemy too.
>
> I'm concerned about this.  What exactly is being used here, a client
> library?  If so, it means that we may have an issue when a python
> application uses a library that wants to dynamically link against both
> the normal sqlite library and this older version.  Maybe this is still
> fine, but proceed with caution.
>
> - Dave

I share your concern, and admittedly don't understand myself what the
implications are here (despite producing a patch, egads!).  It's part of
the standard library though.  I would *think* that it would stay
pointing at the very specific version of sqlite.  Nonetheless, it does
seem unsettling, and unclear to me personally if something unexpected
could happen.

Maybe some "expert" could weigh in... :)

Re: [PATCH] Add and use sqlite-legacy-for-python

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Thompson, David writes:
>
>> On Fri, Feb 12, 2016 at 7:13 PM, Christopher Allan Webber
>> <cwebber@dustycloud.org> wrote:
>>> Ludovic Courtès writes:
>>>
>>>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>>>>
>>>>> Patch b24765139c8940541b23f84592d3580d53f71d71
>>>>>
>>>>>  (define-public sqlite
>>>>>    (package
>>>>>     (name "sqlite")
>>>>> -   (version "3.8.11.1")
>>>>> +   (version "3.10.0")
>>>>>     (source (origin
>>>>>
>>>>> is the cause of python(2|3)-sqlalchemy breaking. I confirmed that by
>>>>> regressing to the original sqlite package. Since the python binding is
>>>>> part of the interpreter, I suspect there may be more python modules
>>>>> vulnerable. I updated python-sqlalchemy to latest and that makes no
>>>>> difference. Its tests fail on sqlite 3.10.0 and pass on 3.8.11.1.
>>>>>
>>>>> What do we do? Revert on this sqlite patch for the new guix release?
>>>>> Or add a second sqlite package and have that as a python dependency?
>>>>
>>>> I would do the latter, assuming that soon a new python-sqlalchemy
>>>> release would solve the problem.  WDYT?
>>>>
>>>> This is probably OK since python-sqlalchemy is a leaf, and so we’re
>>>> unlikely to end up mixing two different SQLite versions.
>>>>
>>>> Ludo’.
>>>
>>> Will sqlalchemy really remain a leaf node?  I hope not, since I'm
>>> working on packaging MediaGoblin now :)
>>
>> Yeah, sqlalchemy being a leaf node is accidental.  It's a library that
>> will be depended on by MediaGoblin and maybe other software.
>>
>>> Regardless, I agree that the second approach seems to be the right one.
>>> I've built a modified package, sqlite-legacy-for-python, and put it to
>>> use.  I built it and confirmed it builds fine and that the tests pass,
>>> and with it, the tests pass in python-sqlalchemy too.
>>
>> I'm concerned about this.  What exactly is being used here, a client
>> library?  If so, it means that we may have an issue when a python
>> application uses a library that wants to dynamically link against both
>> the normal sqlite library and this older version.  Maybe this is still
>> fine, but proceed with caution.
>>
>> - Dave
>
> I share your concern, and admittedly don't understand myself what the
> implications are here (despite producing a patch, egads!).  It's part of
> the standard library though.  I would *think* that it would stay
> pointing at the very specific version of sqlite.  Nonetheless, it does
> seem unsettling, and unclear to me personally if something unexpected
> could happen.
>
> Maybe some "expert" could weigh in... :)

Mark Weaver suggested that I try with the latest sqlite release (3.10.2)
and see if upgrading would fix the problem.  It did not.

python-alembic is the only Python package I have left to do for handling
all of MediaGoblin's Python dependencies.  I would really appreciate it
if we could find a solution for this.

(I suppose bugs should also be filed upstream.  I can do that tomorrow,
I hope.)

 - Chris

Re: [PATCH] Add and use sqlite-legacy-for-python

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Mark Weaver suggested that I try with the latest sqlite release (3.10.2)
> and see if upgrading would fix the problem.  It did not.
>
> python-alembic is the only Python package I have left to do for handling
> all of MediaGoblin's Python dependencies.  I would really appreciate it
> if we could find a solution for this.
>
> (I suppose bugs should also be filed upstream.  I can do that tomorrow,
> I hope.)
>
>  - Chris

Building with a newer version still fails.

I filed a bug on the sqlalchemy tracker:

  https://bitbucket.org/zzzeek/sqlalchemy/issues/3649/tests-fail-with-python-built-with-newer

Re: [PATCH] Add and use sqlite-legacy-for-python

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Building with a newer version still fails.
>
> I filed a bug on the sqlalchemy tracker:
>
>   https://bitbucket.org/zzzeek/sqlalchemy/issues/3649/tests-fail-with-python-built-with-newer

Good news, see the above ticket!  There's a fix upstream.  I'm working
backporting as a patch for Guix.

Only python-sqlalchemy will need to be updated, which meanss this won't
require a ton of rebuilds.

Re: [PATCH] Add and use sqlite-legacy-for-python

[Pjotr Prins]

On Mon, Feb 15, 2016 at 12:13:36PM -0800, Christopher Allan Webber wrote:
> Christopher Allan Webber writes:
> 
> > Building with a newer version still fails.
> >
> > I filed a bug on the sqlalchemy tracker:
> >
> >   https://bitbucket.org/zzzeek/sqlalchemy/issues/3649/tests-fail-with-python-built-with-newer
> 
> Good news, see the above ticket!  There's a fix upstream.  I'm working
> backporting as a patch for Guix.
> 
> Only python-sqlalchemy will need to be updated, which meanss this won't
> require a ton of rebuilds.

That is good news indeed :)
-- 

Re: [PATCH] Add and use sqlite-legacy-for-python

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Christopher Allan Webber writes:
>
>> Building with a newer version still fails.
>>
>> I filed a bug on the sqlalchemy tracker:
>>
>>   https://bitbucket.org/zzzeek/sqlalchemy/issues/3649/tests-fail-with-python-built-with-newer
>
> Good news, see the above ticket!  There's a fix upstream.  I'm working
> backporting as a patch for Guix.

Excellent, sounds like the best possible outcome.  :-)

Ludo’.

Re: [PATCH] Add and use sqlite-legacy-for-python

[Andreas Enge]

Hello,


On Fri, Feb 12, 2016 at 04:13:52PM -0800, Christopher Allan Webber wrote:
> +(define-public sqlite-legacy-for-python
> +    (version "3.8.11.1")

in line with other cases where we package several versions, I would simply
call this sqlite-3.8.

Maybe it could be a simple "define" instead of "define-public", assuming
it would effectively be used only as in input inside the python module?
(Then it would need to be placed in the same module, of course.)

Andreas