python to support SQLite extensions

python to support SQLite extensions

[Jelle Licht]

Hello folks,

I want to enable support for loadable extensions in python's sqlite3
module. This should be enabled by passing
`--enable-loadable-sqlite-extensions' to `./configure'.

I've spend an embarassing amount of time on this, so perhaps someone out
there understands python better than I do and can elucidate me.

This does depend on our SQLite package needing support for loadable
extensions, but it seems it does.

I've tried the following custom package definition:

--8<---------------cut here---------------start------------->8---
(define-public python/sqlite-extensions
  (package
    (inherit python)
    (arguments
     (substitute-keyword-arguments (package-arguments python)
       ((#:configure-flags flags '())
        `(cons "--enable-loadable-sqlite-extensions" ,flags))))))
--8<---------------cut here---------------end--------------->8---

... which seems to 'work', according to the build log:
--8<---------------cut here---------------start------------->8---
[snip]
checking for --enable-loadable-sqlite-extensions... yes
[snip]
--8<---------------cut here---------------end--------------->8---

Sadly, even with my freshly built python3, I get the following:

--8<---------------cut here---------------start------------->8---
#> import sqlite3
#> conn=sqlite3.connect("/tmp/some-db.sqlite")
#> conn.enable_load_extension(True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension'
--8<---------------cut here---------------end--------------->8---

Does anybody perhaps know why my python is correctly receiving (and
subsequently ignoring) this configure flag? It seems that *something*
happens between the 'configure' script receiving the flag, and the file
in `Modules/_sqlite/connection.c' being built, it seems
SQLITE_OMIT_LOAD_EXTENSION is set to "1".

This (seemingly) happens in setup.py, lines 1432 and on:
--8<---------------cut here---------------start------------->8---
 if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"):
     sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))
--8<---------------cut here---------------end--------------->8---

Indeed, in *any* python3 I can find on my Guix System, I see:

--8<---------------cut here---------------start------------->8---
#> import sysconfig
#> sysconfig.get_config_var("CONFIG_ARGS")
'--with-system-ffi'
--8<---------------cut here---------------end--------------->8---

Something weird is going on!

I have no clue how to continue finding why this happens, but for my
personal problem I can just patch python's setup.py file.

Thanks for any pointers,
 - Jelle

NB, this configure option happens to be enabled by default in many other
distro's (among which Nix [1]), so it might actually make sense to also
set things up in such a way on our next core-updates cycle, once the
flag actually does something.

[1] https://github.com/NixOS/nixpkgs/pull/67472/files

Re: python to support SQLite extensions

[Felix Gruber]

Hi Jelle,

On 5/10/21 11:31 PM, Jelle Licht wrote:
> Does anybody perhaps know why my python is correctly receiving (and
> subsequently ignoring) this configure flag? It seems that *something*
> happens between the 'configure' script receiving the flag, and the file
> in `Modules/_sqlite/connection.c' being built, it seems
> SQLITE_OMIT_LOAD_EXTENSION is set to "1".
> 
> This (seemingly) happens in setup.py, lines 1432 and on:
> --8<---------------cut here---------------start------------->8---
>  if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"):
>      sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))
> --8<---------------cut here---------------end--------------->8---
> 
> Indeed, in *any* python3 I can find on my Guix System, I see:
> 
> --8<---------------cut here---------------start------------->8---
> #> import sysconfig
> #> sysconfig.get_config_var("CONFIG_ARGS")
> '--with-system-ffi'
> --8<---------------cut here---------------end--------------->8---
> 
> Something weird is going on!
> 
> I have no clue how to continue finding why this happens, but for my
> personal problem I can just patch python's setup.py file.

You're onto something with `sysconfig.get_config_var("CONFIG_ARGS")`:
In the definition of the python-2.7 package from which the python
package eventually derives from, a phase
`'do-not-record-configure-flags` is added before `'configure`. This
phase removes everything but '--with-system-ffi' from `CONFIG_ARGS`.

I've successfully enabled loadable sqlite extensions with the following
package definition.
-------8<----------------8<----------------8<----------------8<-------
(define python-with-loadable-sqlite-modules
  (package (inherit python)
    (arguments
     (substitute-keyword-arguments (package-arguments python)
       ((#:configure-flags cf)
        `(cons* "--enable-loadable-sqlite-extensions" ,cf))
       ((#:phases phases)
        `(modify-phases ,phases
           (delete 'do-not-record-configure-flags)))))))
------->8---------------->8---------------->8---------------->8-------

While this works, it might be better to modify the
`'do-not-record-configure-flags` phase to keep the
`--enable-loadable-sqlite-extensions` option instead of completely
removing this phase.

I think it might be a good idea to enable loadable sqlite extensions by
default in the python package. As you've already pointed out, other
distros have this option enabled which leads to surprises when trying to
load sqlite extensions in guix's python.


Hope that helps,
Felix

Re: python to support SQLite extensions

[Jelle Licht]

Hey Felix,

Felix Gruber <felgru@posteo.net> writes:

> Hi Jelle,
>
> On 5/10/21 11:31 PM, Jelle Licht wrote:
>> Does anybody perhaps know why my python is correctly receiving (and
>> subsequently ignoring) this configure flag? It seems that *something*
>> happens between the 'configure' script receiving the flag, and the file
>> in `Modules/_sqlite/connection.c' being built, it seems
>> SQLITE_OMIT_LOAD_EXTENSION is set to "1".
>> 
>> This (seemingly) happens in setup.py, lines 1432 and on:
>> --8<---------------cut here---------------start------------->8---
>>  if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"):
>>      sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Indeed, in *any* python3 I can find on my Guix System, I see:
>> 
>> --8<---------------cut here---------------start------------->8---
>> #> import sysconfig
>> #> sysconfig.get_config_var("CONFIG_ARGS")
>> '--with-system-ffi'
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Something weird is going on!
>> 
>> I have no clue how to continue finding why this happens, but for my
>> personal problem I can just patch python's setup.py file.
>
> You're onto something with `sysconfig.get_config_var("CONFIG_ARGS")`:
> In the definition of the python-2.7 package from which the python
> package eventually derives from, a phase
> `'do-not-record-configure-flags` is added before `'configure`. This
> phase removes everything but '--with-system-ffi' from `CONFIG_ARGS`.
>
> I've successfully enabled loadable sqlite extensions with the following
> package definition.
> -------8<----------------8<----------------8<----------------8<-------
> (define python-with-loadable-sqlite-modules
>   (package (inherit python)
>     (arguments
>      (substitute-keyword-arguments (package-arguments python)
>        ((#:configure-flags cf)
>         `(cons* "--enable-loadable-sqlite-extensions" ,cf))
>        ((#:phases phases)
>         `(modify-phases ,phases
>            (delete 'do-not-record-configure-flags)))))))
> ------->8---------------->8---------------->8---------------->8-------

This works well-enough for me atm, thanks for getting me to understand
the missing part here :).

> While this works, it might be better to modify the
> `'do-not-record-configure-flags` phase to keep the
> `--enable-loadable-sqlite-extensions` option instead of completely
> removing this phase.
>
> I think it might be a good idea to enable loadable sqlite extensions by
> default in the python package. As you've already pointed out, other
> distros have this option enabled which leads to surprises when trying to
> load sqlite extensions in guix's python.

Makes sense, but I am no python guru. I'll open a separate ticket for
this discussion, because there might be more flags that have similar
issues that we might want to support.

Thanks again,
 Jelle