Hello folks, I want to enable support for loadable extensions in python's sqlite3 module. This should be enabled by passing `--enable-loadable-sqlite-extensions' to `./configure'. I've spend an embarassing amount of time on this, so perhaps someone out there understands python better than I do and can elucidate me. This does depend on our SQLite package needing support for loadable extensions, but it seems it does. I've tried the following custom package definition: --8<---------------cut here---------------start------------->8--- (define-public python/sqlite-extensions (package (inherit python) (arguments (substitute-keyword-arguments (package-arguments python) ((#:configure-flags flags '()) `(cons "--enable-loadable-sqlite-extensions" ,flags)))))) --8<---------------cut here---------------end--------------->8--- ... which seems to 'work', according to the build log: --8<---------------cut here---------------start------------->8--- [snip] checking for --enable-loadable-sqlite-extensions... yes [snip] --8<---------------cut here---------------end--------------->8--- Sadly, even with my freshly built python3, I get the following: --8<---------------cut here---------------start------------->8--- #> import sqlite3 #> conn=sqlite3.connect("/tmp/some-db.sqlite") #> conn.enable_load_extension(True) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension' --8<---------------cut here---------------end--------------->8--- Does anybody perhaps know why my python is correctly receiving (and subsequently ignoring) this configure flag? It seems that *something* happens between the 'configure' script receiving the flag, and the file in `Modules/_sqlite/connection.c' being built, it seems SQLITE_OMIT_LOAD_EXTENSION is set to "1". This (seemingly) happens in setup.py, lines 1432 and on: --8<---------------cut here---------------start------------->8--- if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"): sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1")) --8<---------------cut here---------------end--------------->8--- Indeed, in *any* python3 I can find on my Guix System, I see: --8<---------------cut here---------------start------------->8--- #> import sysconfig #> sysconfig.get_config_var("CONFIG_ARGS") '--with-system-ffi' --8<---------------cut here---------------end--------------->8--- Something weird is going on! I have no clue how to continue finding why this happens, but for my personal problem I can just patch python's setup.py file. Thanks for any pointers, - Jelle NB, this configure option happens to be enabled by default in many other distro's (among which Nix [1]), so it might actually make sense to also set things up in such a way on our next core-updates cycle, once the flag actually does something. [1] https://github.com/NixOS/nixpkgs/pull/67472/files
Hi Jelle,
On 5/10/21 11:31 PM, Jelle Licht wrote:
> Does anybody perhaps know why my python is correctly receiving (and
> subsequently ignoring) this configure flag? It seems that *something*
> happens between the 'configure' script receiving the flag, and the file
> in `Modules/_sqlite/connection.c' being built, it seems
> SQLITE_OMIT_LOAD_EXTENSION is set to "1".
>
> This (seemingly) happens in setup.py, lines 1432 and on:
> --8<---------------cut here---------------start------------->8---
> if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"):
> sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))
> --8<---------------cut here---------------end--------------->8---
>
> Indeed, in *any* python3 I can find on my Guix System, I see:
>
> --8<---------------cut here---------------start------------->8---
> #> import sysconfig
> #> sysconfig.get_config_var("CONFIG_ARGS")
> '--with-system-ffi'
> --8<---------------cut here---------------end--------------->8---
>
> Something weird is going on!
>
> I have no clue how to continue finding why this happens, but for my
> personal problem I can just patch python's setup.py file.
You're onto something with `sysconfig.get_config_var("CONFIG_ARGS")`:
In the definition of the python-2.7 package from which the python
package eventually derives from, a phase
`'do-not-record-configure-flags` is added before `'configure`. This
phase removes everything but '--with-system-ffi' from `CONFIG_ARGS`.
I've successfully enabled loadable sqlite extensions with the following
package definition.
-------8<----------------8<----------------8<----------------8<-------
(define python-with-loadable-sqlite-modules
(package (inherit python)
(arguments
(substitute-keyword-arguments (package-arguments python)
((#:configure-flags cf)
`(cons* "--enable-loadable-sqlite-extensions" ,cf))
((#:phases phases)
`(modify-phases ,phases
(delete 'do-not-record-configure-flags)))))))
------->8---------------->8---------------->8---------------->8-------
While this works, it might be better to modify the
`'do-not-record-configure-flags` phase to keep the
`--enable-loadable-sqlite-extensions` option instead of completely
removing this phase.
I think it might be a good idea to enable loadable sqlite extensions by
default in the python package. As you've already pointed out, other
distros have this option enabled which leads to surprises when trying to
load sqlite extensions in guix's python.
Hope that helps,
Felix
Hey Felix, Felix Gruber <felgru@posteo.net> writes: > Hi Jelle, > > On 5/10/21 11:31 PM, Jelle Licht wrote: >> Does anybody perhaps know why my python is correctly receiving (and >> subsequently ignoring) this configure flag? It seems that *something* >> happens between the 'configure' script receiving the flag, and the file >> in `Modules/_sqlite/connection.c' being built, it seems >> SQLITE_OMIT_LOAD_EXTENSION is set to "1". >> >> This (seemingly) happens in setup.py, lines 1432 and on: >> --8<---------------cut here---------------start------------->8--- >> if '--enable-loadable-sqlite-extensions' not in sysconfig.get_config_var("CONFIG_ARGS"): >> sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1")) >> --8<---------------cut here---------------end--------------->8--- >> >> Indeed, in *any* python3 I can find on my Guix System, I see: >> >> --8<---------------cut here---------------start------------->8--- >> #> import sysconfig >> #> sysconfig.get_config_var("CONFIG_ARGS") >> '--with-system-ffi' >> --8<---------------cut here---------------end--------------->8--- >> >> Something weird is going on! >> >> I have no clue how to continue finding why this happens, but for my >> personal problem I can just patch python's setup.py file. > > You're onto something with `sysconfig.get_config_var("CONFIG_ARGS")`: > In the definition of the python-2.7 package from which the python > package eventually derives from, a phase > `'do-not-record-configure-flags` is added before `'configure`. This > phase removes everything but '--with-system-ffi' from `CONFIG_ARGS`. > > I've successfully enabled loadable sqlite extensions with the following > package definition. > -------8<----------------8<----------------8<----------------8<------- > (define python-with-loadable-sqlite-modules > (package (inherit python) > (arguments > (substitute-keyword-arguments (package-arguments python) > ((#:configure-flags cf) > `(cons* "--enable-loadable-sqlite-extensions" ,cf)) > ((#:phases phases) > `(modify-phases ,phases > (delete 'do-not-record-configure-flags))))))) > ------->8---------------->8---------------->8---------------->8------- This works well-enough for me atm, thanks for getting me to understand the missing part here :). > While this works, it might be better to modify the > `'do-not-record-configure-flags` phase to keep the > `--enable-loadable-sqlite-extensions` option instead of completely > removing this phase. > > I think it might be a good idea to enable loadable sqlite extensions by > default in the python package. As you've already pointed out, other > distros have this option enabled which leads to surprises when trying to > load sqlite extensions in guix's python. Makes sense, but I am no python guru. I'll open a separate ticket for this discussion, because there might be more flags that have similar issues that we might want to support. Thanks again, Jelle