[WIP PATCH 0/1] autodoc failed to import notmuch2

[WIP PATCH 0/1] autodoc failed to import notmuch2

[Michael J Gruber]

Hi there

During Fedora package builds I noticed that autodoc fails to import the
notmuch2 module:

```
WARNING: autodoc: failed to import module 'notmuch2'; the following
exception was raised:
No module named 'notmuch2'
```

You never notice this on your own box where notmuch2 is installed,
because then the build finds the previously installed module. But this
is wrong, of course, because autodoc is supposed to document tthe module
being build (in-source module).

There are actually several problems leading to this:

1. search path
The module residing in `bindings/python-cffi/notmuch2/__init__.py` needs
a search path `bindings/python-cffi/`, not `bindings/python-cffi/notmuch2`.

If we fix just that we get:
```
WARNING: autodoc: failed to import module 'notmuch2'; the following
exception was raised:
cannot import name '_capi' from partially initialized module 'notmuch2'
(most likely due to a circular import)
(/builddir/build/BUILD/notmuch-0.34/doc/../bindings/python-cffi/notmuch2/__init__.py)
```

2. search path, again
The module is built in `bindings/python-cffi/build/stage`, that is where
`_capi` is to be found, and PATCH 1/1 fixes the path in `doc/conf.py`.

3. build order
The patch from 2. actually helps only if we build in the following
order (make targets):
notmuch notmuch-shared ruby-bindings python-cffi-bindings
all

In particular, the bindings need to be built before the docs

I can do this using two make calls, but a proper solution would specify
the dependencies in the makefile.

4. library search path

When you do 1. through 3. then autodoc almost manages to import
notmuch2: it finds the proper module, _capi is found, but then
libnotmuch.so is not found because LD_LIBRARY_PATH would need to be
amended to find the freshly build library.

Now, maybe I should simply ignore the autodoc warning. Or I should
change my packaging to build twice (make && make install && make clean
&& make && make install). At least I wanted to raise attention to this.

This problem (if it is one) is not specific to 0.34, it was the same
at least with 0.33 but - being just a warning - was unnoticed.

Cheers
Michael

Michael J Gruber (1):
  autodoc: fix search path for notmuch2 module

 doc/conf.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.33.0.1146.g7ac282475c

[WIP PATCH 1/1] autodoc: fix search path for notmuch2 module

[Michael J Gruber]

a05da455 ("doc: add new python bindings to main documentation tree.", 2020-07-11)
added the path "bindings/python-cffi/notmuch2" to the search path for
autodoc. But the notmuch2 module cannot be imported from there - the dir
"is" the module, and it resides in "bindings/python-cffi" and in
"bindings/python-cffi/build/stage", with only the latter location having
the compiled capi part.

This goes unnoticed as long as you have notmuch2 in your standard search
path, but it means autodocs finds the wrong module, not the one one we
are building the doc against.

Fix the path so that the build tree module comes first.

Signed-off-by: Michael J Gruber <git@grubix.eu>
---
 doc/conf.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/conf.py b/doc/conf.py
index 1fbd102b..c7fd8f5a 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -25,7 +25,7 @@ for pathdir in ['.', '..']:
             version=infile.read().replace('\n','')
 
 # for autodoc
-sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi', 'notmuch2'))
+sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi', 'build', 'stage'))
 
 # read generated config
 for pathdir in ['.', '..']:
-- 
2.33.0.1146.g7ac282475c

Re: [WIP PATCH 0/1] autodoc failed to import notmuch2

[David Bremner]

Michael J Gruber <git@grubix.eu> writes:

> Hi there
>
> During Fedora package builds I noticed that autodoc fails to import the
> notmuch2 module:
>
> ```
> WARNING: autodoc: failed to import module 'notmuch2'; the following
> exception was raised:
> No module named 'notmuch2'
> ```
>
> You never notice this on your own box where notmuch2 is installed,
> because then the build finds the previously installed module. But this
> is wrong, of course, because autodoc is supposed to document tthe module
> being build (in-source module).
>

I also see this error when doing builds in a clean environment for
Debian. I guess I'm currently not using the docs from those builds, so I
don't know how bad the damage is.

Re: [WIP PATCH 0/1] autodoc failed to import notmuch2

[Michael J Gruber]

David Bremner venit, vidit, dixit 2021-10-21 14:20:11:
> Michael J Gruber <git@grubix.eu> writes:
> 
> > Hi there
> >
> > During Fedora package builds I noticed that autodoc fails to import the
> > notmuch2 module:
> >
> > ```
> > WARNING: autodoc: failed to import module 'notmuch2'; the following
> > exception was raised:
> > No module named 'notmuch2'
> > ```
> >
> > You never notice this on your own box where notmuch2 is installed,
> > because then the build finds the previously installed module. But this
> > is wrong, of course, because autodoc is supposed to document tthe module
> > being build (in-source module).
> >
> 
> I also see this error when doing builds in a clean environment for
> Debian. I guess I'm currently not using the docs from those builds, so I
> don't know how bad the damage is.

In that case: Which docs do you use, and which docs do you want us
(packagers) to use? The usual fedora policy is to (re)build docs if
the doc source is shipped.

Disclaimer: I have no clue about autodoc ;)

Michael

Re: [WIP PATCH 0/1] autodoc failed to import notmuch2

[David Bremner]

Michael J Gruber <git@grubix.eu> writes:

>
> In that case: Which docs do you use, and which docs do you want us
> (packagers) to use? The usual fedora policy is to (re)build docs if
> the doc source is shipped.
>
> Disclaimer: I have no clue about autodoc ;)

I agree with this in general. Currently Debian does not ship the docs. I
also don't know anything about autodoc, so curently I'm just deploying
locally built docs on notmuchmail.org.

d

[PATCH] doc/python-cffi: import from built bindings, not installed module

[David Bremner]

Previously the python-cffi bindings either failed to build, or built
for the wrong module by using the installed module.

The fix requires correction the module path, building the bindings
before docs, and helping python find the built libnotmuch.

Based on patch / discussion from Micheal Gruber [1]

[1]: id:cover.1634808719.git.git@grubix.eu
---
 doc/Makefile.local | 12 ++++++++----
 doc/conf.py        |  2 +-
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index f476d1da..730ad4fb 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -4,7 +4,7 @@ dir := doc
 
 # You can set these variables from the command line.
 SPHINXOPTS    := -q
-SPHINXBUILD   = sphinx-build
+SPHINXBUILD   = env LD_LIBRARY_PATH=${NOTMUCH_BUILDDIR}/lib sphinx-build
 DOCBUILDDIR      := $(dir)/_build
 
 # Internal variables.
@@ -35,7 +35,7 @@ endif
 
 INFO_INFO_FILES := $(INFO_TEXI_FILES:.texi=.info)
 
-.PHONY: sphinx-html sphinx-texinfo sphinx-info
+.PHONY: sphinx-html sphinx-texinfo sphinx-info doc-prereqs
 
 .PHONY: install-man build-man apidocs install-apidocs
 
@@ -46,15 +46,19 @@ ifeq ($(WITH_EMACS),1)
 $(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.texi.stamp : docstring.stamp
 endif
 
+ifeq ($(HAVE_PYTHON3_CFFI),1)
+doc-prereqs: python-cffi-bindings
+endif
+
 sphinx-html: $(DOCBUILDDIR)/.html.stamp
 
-$(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES)
+$(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES) doc-prereqs
 	$(SPHINXBUILD) -b html -d $(DOCBUILDDIR)/html_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
 	touch $@
 
 sphinx-texinfo: $(DOCBUILDDIR)/.texi.stamp
 
-$(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES)
+$(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES) doc-prereqs
 	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
diff --git a/doc/conf.py b/doc/conf.py
index 1fbd102b..c7fd8f5a 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -25,7 +25,7 @@ for pathdir in ['.', '..']:
             version=infile.read().replace('\n','')
 
 # for autodoc
-sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi', 'notmuch2'))
+sys.path.insert(0, os.path.join(location, '..', 'bindings', 'python-cffi', 'build', 'stage'))
 
 # read generated config
 for pathdir in ['.', '..']:
-- 
2.33.0

Re: [PATCH] doc/python-cffi: import from built bindings, not installed module

[David Bremner]

David Bremner <david@tethera.net> writes:

> Previously the python-cffi bindings either failed to build, or built
> for the wrong module by using the installed module.

applied to release and master