configure checks for python3-notmuch2 module

configure checks for python3-notmuch2 module

[David Bremner]

It turns out we need pyconfig.h to build the (new) python bindings.  I
don't think this is adequately checked for by the configure script,
since I had a Fedora user report a build failure (during make install,
which I found odd) missing pyconfig.h

On Debian this is on libpython3.x-dev, which is not a dependency of the
python3-cffi package.

log follows.
----------------------------------------------------------------------

cd bindings/python-cffi && \
	python3 setup.py build --build-lib build/stage && \
	mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
running build
running build_py
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.9/notmuch2._capi.c'
creating build/temp.linux-x86_64-3.9
building 'notmuch2._capi' extension
creating build/temp.linux-x86_64-3.9/build
creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../lib -I/usr/include/python3.9 -c build/temp.linux-x86_64-3.9/notmuch2._capi.c -o bui
 ld/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/notmuch2._capi.o
build/temp.linux-x86_64-3.9/notmuch2._capi.c:50:14: fatal error: pyconfig.h: No such file or directory
   50 | #    include <pyconfig.h>
      |              ^~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Re: configure checks for python3-notmuch2 module

[Michael J Gruber]

David Bremner venit, vidit, dixit 2021-07-11 23:29:57:
> 
> It turns out we need pyconfig.h to build the (new) python bindings.  I
> don't think this is adequately checked for by the configure script,
> since I had a Fedora user report a build failure (during make install,
> which I found odd) missing pyconfig.h
> 
> On Debian this is on libpython3.x-dev, which is not a dependency of the
> python3-cffi package.

You need python3-cffi to run python code which uses cffi bindings -
think of it as the "lib" part. You need the devel package to generate a
python module using cffi (i.e. binding for a specific c library).

So, it makes sense for the lib package not to depend on the devel
package (which is python3-devel on Fedora).

I can't help with adapting configure to the header requirement, though.

Michael

> log follows.
> ----------------------------------------------------------------------
> 
> cd bindings/python-cffi && \
>         python3 setup.py build --build-lib build/stage && \
>         mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
> running build
> running build_py
> running build_ext
> generating cffi module 'build/temp.linux-x86_64-3.9/notmuch2._capi.c'
> creating build/temp.linux-x86_64-3.9
> building 'notmuch2._capi' extension
> creating build/temp.linux-x86_64-3.9/build
> creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
> gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../lib -I/usr/include/python3.9 -c build/temp.linux-x86_64-3.9/notmuch2._capi.c -o b
 ui
>  ld/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/notmuch2._capi.o
> build/temp.linux-x86_64-3.9/notmuch2._capi.c:50:14: fatal error: pyconfig.h: No such file or directory
>    50 | #    include <pyconfig.h>
>       |              ^~~~~~~~~~~~
> compilation terminated.
> error: command '/usr/bin/gcc' failed with exit code 1
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: configure checks for python3-notmuch2 module

[Tomi Ollila]

On Sun, Jul 11 2021, David Bremner wrote:

> It turns out we need pyconfig.h to build the (new) python bindings.  I
> don't think this is adequately checked for by the configure script,
> since I had a Fedora user report a build failure (during make install,
> which I found odd) missing pyconfig.h
>
> On Debian this is on libpython3.x-dev, which is not a dependency of the
> python3-cffi package.
>
> log follows.
> ----------------------------------------------------------------------
>
> cd bindings/python-cffi && \
> 	python3 setup.py build --build-lib build/stage && \
> 	mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
> running build
> running build_py
> running build_ext
> generating cffi module 'build/temp.linux-x86_64-3.9/notmuch2._capi.c'
> creating build/temp.linux-x86_64-3.9
> building 'notmuch2._capi' extension
> creating build/temp.linux-x86_64-3.9/build
> creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
> gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../lib -I/usr/include/python3.9 -c build/temp.linux-x86_64-3.9/notmuch2._capi.c -o b
 ui
>  ld/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/notmuch2._capi.o
> build/temp.linux-x86_64-3.9/notmuch2._capi.c:50:14: fatal error: pyconfig.h: No such file or directory
>    50 | #    include <pyconfig.h>
>       |              ^~~~~~~~~~~~
> compilation terminated.
> error: command '/usr/bin/gcc' failed with exit code 1

While me trying (Fedora 34 .. ooh no, 33 (but anyway ;D)):

$ ./configure
...
Checking for python3 cffi and setuptools...
No (will not install CFFI-based python bindings).
...

OK, why -- I do have cffi and setuptools python packages installed
..so:

$ script -c 'sh -x ./configure'

reveals (after trying w/ python 3.9 repl) that cffi.FFI().verify()
fails due to missing Python.h... let's hunt (again, I recall solving
this like a year ago or so..) ... apparently python3-devel needed.

$ sudo dnf install python3-devel
$ make distclean
$ ./configure --prefix=$PWD/TTT
$ make
$ make install

... hmm all worked.... pyconfig.h is part of python3-devel package

Have to check how all this behaves on Fedora 34 machine when I 
have access to one (currently away from such machines and from
power grid also (laptop battery power, which I have to preserve,
mobile devices gets charged by small solar panels so no problem
there ;D))

Tomi 

Re: configure checks for python3-notmuch2 module

[Tomi Ollila]

On Mon, Jul 12 2021, Michael J. Gruber wrote:

> David Bremner venit, vidit, dixit 2021-07-11 23:29:57:
>> 
>> It turns out we need pyconfig.h to build the (new) python bindings.  I
>> don't think this is adequately checked for by the configure script,
>> since I had a Fedora user report a build failure (during make install,
>> which I found odd) missing pyconfig.h
>> 
>> On Debian this is on libpython3.x-dev, which is not a dependency of the
>> python3-cffi package.
>
> You need python3-cffi to run python code which uses cffi bindings -
> think of it as the "lib" part. You need the devel package to generate a
> python module using cffi (i.e. binding for a specific c library).
>
> So, it makes sense for the lib package not to depend on the devel
> package (which is python3-devel on Fedora).
>
> I can't help with adapting configure to the header requirement, though.

In case of python3-devel missing in Fedora, the compilation should not pass
as far as shown in David's email (and in log below) as it is needed for 

Checking for python3 cffi and setuptools... 

configure part to complete with 'Yes'

But the message could be better. Perhaps

Checking for python3 dev(el), cffi and setuptools...

(although this may not be clear enough for debian users as the package
there is libpython3-dev, cannot say...)

For me, to figure out what was needed in Fedora 34 (ok, 33, but now tested
in upgraded fedora 34 host in running fedora:34 podman container) was to run
script -c 'sh -x ./configure'; less typescript; and then do internet search
how to install missing 'Python.h' include file...

Tomi

>
> Michael
>
>> log follows.
>> ----------------------------------------------------------------------
>> 
>> cd bindings/python-cffi && \
>>         python3 setup.py build --build-lib build/stage && \
>>         mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
>> running build
>> running build_py
>> running build_ext
>> generating cffi module 'build/temp.linux-x86_64-3.9/notmuch2._capi.c'
>> creating build/temp.linux-x86_64-3.9
>> building 'notmuch2._capi' extension
>> creating build/temp.linux-x86_64-3.9/build
>> creating build/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9
>> gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I../../lib -I/usr/include/python3.9 -c build/temp.linux-x86_64-3.9/notmuch2._capi.c -o 
 b
>  ui
>>  ld/temp.linux-x86_64-3.9/build/temp.linux-x86_64-3.9/notmuch2._capi.o
>> build/temp.linux-x86_64-3.9/notmuch2._capi.c:50:14: fatal error: pyconfig.h: No such file or directory
>>    50 | #    include <pyconfig.h>
>>       |              ^~~~~~~~~~~~
>> compilation terminated.
>> error: command '/usr/bin/gcc' failed with exit code 1

Re: configure checks for python3-notmuch2 module

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

>
> Have to check how all this behaves on Fedora 34 machine when I 
> have access to one (currently away from such machines and from
> power grid also (laptop battery power, which I have to preserve,
> mobile devices gets charged by small solar panels so no problem
> there ;D))

Since winter is coming, I hope you are back in civilization :P. Did you
get a chance to try this?

cheers,

d

Re: configure checks for python3-notmuch2 module

[Tomi Ollila]

On Wed, Oct 06 2021, David Bremner wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>>
>> Have to check how all this behaves on Fedora 34 machine when I 
>> have access to one (currently away from such machines and from
>> power grid also (laptop battery power, which I have to preserve,
>> mobile devices gets charged by small solar panels so no problem
>> there ;D))
>
> Since winter is coming, I hope you are back in civilization :P. Did you
> get a chance to try this?

Yes, come back -- due to climate change outside temperature is not a
problem, just that sun is so low it is hard to get any electricity ;/
-- now I have power and network -- civilization is harder to find...


So, on Fedora 34, without python3-devel installed

$ python3 -c 'import cffi,setuptools; cffi.FFI().verify()'                      
__pycache__/_cffi__xd915627fxd88346ae.c:2:10: fatal error: Python.h: No
such file or directory
    2 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

(as of now python3-devel-3.9.7-1.fc34 provides /usr/include/python3.9/Python.h)

(debian systems may work differently -- w/ strace the above command
line did not show any access to Python.h...)

>
> cheers,
>
> d

Tomi

Re: configure checks for python3-notmuch2 module

[Tomi Ollila]

On Wed, Oct 06 2021, Tomi Ollila wrote:

>
> So, on Fedora 34, without python3-devel installed
>
> $ python3 -c 'import cffi,setuptools; cffi.FFI().verify()'
> __pycache__/_cffi__xd915627fxd88346ae.c:2:10: fatal error: Python.h: No
> such file or directory
>     2 | #include <Python.h>
>       |          ^~~~~~~~~~
> compilation terminated.
>
> (as of now python3-devel-3.9.7-1.fc34 provides /usr/include/python3.9/Python.h)
>
> (debian systems may work differently -- w/ strace the above command
> line did not show any access to Python.h...)


The reason for the above was probably that something was cached; got
exactly the same problem when tried in debian:11 (podman) container 
w/o libpython3-dev (*) -- and after apt-get install libpython3-dev
the above cffi.FFI().verify() worked

Tomi


(*) executed the following on top of docker.io/library/debian:11
    and then ./configure in notmuch source dir...

  export DEBIAN_FRONTEND=noninteractive
  apt-get update
  apt-get install -y -q build-essential emacs-nox gdb git man \
          dtach libxapian-dev libgmime-3.0-dev libtalloc-dev \
          python3-sphinx python3-cffi python3-pytest \
          python3-setuptools gpgsm parallel
  apt-get -y autoremove
  apt-get -y clean
  rm -rf /var/lib/apt/lists/

>
>>
>> cheers,
>>
>> d
>
> Tomi

[PATCH] configure: check explicitely for python dev (include) files

[David Bremner]

As discussed at [1] we have received reports that the implicit check
using cffi.FFI().verify() is not reliable in all environments. Since
we already use pkg-config, and the python dev package should include a
.pc file [2], add an extra check using pkg-config.  On at least
Debian, we have to know which version of python dev files with are
looking for, so calculate that first.

[1]: id:87im1g35ey.fsf@tethera.netid:87im1g35ey.fsf@tethera.net,
[2]: checked on Debian and Fedora
---
 configure | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/configure b/configure
index 4262d122..fc9512f9 100755
--- a/configure
+++ b/configure
@@ -777,7 +777,23 @@ fi
 
 have_python3_cffi=0
 have_python3_pytest=0
+have_python3_dev=0
 if [ $have_python3 -eq 1 ]; then
+    printf "Checking for python3 version ..."
+    python3_version=$("$python" -c 'import sysconfig; print(sysconfig.get_python_version());')
+    printf "(%s)\n" $python3_version
+
+    printf "Checking for python $python3_version development files..."
+    if pkg-config --exists "python-$python3_version"; then
+	have_python3_dev=1
+	printf "Yes.\n"
+    else
+	have_python3_dev=0
+        printf "No (will not install CFFI-based python bindings).\n"
+    fi
+fi
+
+if [ $have_python3_dev -eq 1 ]; then
     printf "Checking for python3 cffi and setuptools... "
     if "$python" -c 'import cffi,setuptools; cffi.FFI().verify()' >/dev/null 2>&1; then
         printf "Yes.\n"
-- 
2.33.0

Re: [PATCH] configure: check explicitely for python dev (include) files

[Tomi Ollila]

On Thu, Oct 07 2021, David Bremner wrote:

> As discussed at [1] we have received reports that the implicit check
> using cffi.FFI().verify() is not reliable in all environments. Since
> we already use pkg-config, and the python dev package should include a
> .pc file [2], add an extra check using pkg-config.  On at least
> Debian, we have to know which version of python dev files with are
> looking for, so calculate that first.
>
> [1]: id:87im1g35ey.fsf@tethera.netid:87im1g35ey.fsf@tethera.net,
> [2]: checked on Debian and Fedora
> ---
>  configure | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/configure b/configure
> index 4262d122..fc9512f9 100755
> --- a/configure
> +++ b/configure
> @@ -777,7 +777,23 @@ fi
>  
>  have_python3_cffi=0
>  have_python3_pytest=0

Works and look good (but some style mismatches) -- but move the whole added
block before the 2  variables set abobe to zero -- those does not need to
be defined before the code below is executed and it is more consistent that
way... 

> +have_python3_dev=0
>  if [ $have_python3 -eq 1 ]; then
> +    printf "Checking for python3 version ..."
 ----------- extra space here -------------^
> +    python3_version=$("$python" -c 'import sysconfig; print(sysconfig.get_python_version());')
> +    printf "(%s)\n" $python3_version
> +
> +    printf "Checking for python $python3_version development files..."
> +    if pkg-config --exists "python-$python3_version"; then
> +	have_python3_dev=1
> +	printf "Yes.\n"
> +    else
> +	have_python3_dev=0
> +        printf "No (will not install CFFI-based python bindings).\n"
--- tabs vs spaces difference above (use only spaces above for local consistency) ---
> +    fi
> +fi

...i.e. the lines 'have_python3_cffi=0' and 'have_python3_pytest=0' to appear here

> +
> +if [ $have_python3_dev -eq 1 ]; then
>      printf "Checking for python3 cffi and setuptools... "
>      if "$python" -c 'import cffi,setuptools; cffi.FFI().verify()' >/dev/null 2>&1; then
>          printf "Yes.\n"
> -- 
> 2.33.0


Tomi

Re: [PATCH] configure: check explicitely for python dev (include) files

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

>> +	have_python3_dev=0
>> +        printf "No (will not install CFFI-based python bindings).\n"
> --- tabs vs spaces difference above (use only spaces above for local consistency) ---
>> +    fi
>> +fi

I hadn't realized configure was such a mess indentation wise; see
id:20211009200916.1121658-1-david@tethera.net for a proposed global
consistency fix.

>
> ...i.e. the lines 'have_python3_cffi=0' and 'have_python3_pytest=0' to appear here
>

Applied to master with your suggestions