Fix order of -I and -L flags

Fix order of -I and -L flags

[Ryan Schmidt]

The notmuch build system puts -I and -L flags in the wrong order.

Specifically, -I flags the user might specify in the CPPFLAGS environment variable appear before the -I flags for the project's own directories, resulting in build failure if a previous version of notmuch (whose headers differ sufficiently from the new version) was already installed.

https://trac.macports.org/ticket/63274

Similarly, -L flags the user might specify in the LDFLAGS environment variable appear before the -L flags for the project's own directories, resulting in build failure if a previous version of notmuch (whose libraries differ sufficiently from the new version) was already installed.

https://trac.macports.org/ticket/63665

Re: Fix order of -I and -L flags

[David Bremner]

Ryan Schmidt <notmuch@ryandesign.com> writes:

> The notmuch build system puts -I and -L flags in the wrong order.
>
> Specifically, -I flags the user might specify in the CPPFLAGS
> environment variable appear before the -I flags for the project's own
> directories, resulting in build failure if a previous version of
> notmuch (whose headers differ sufficiently from the new version) was
> already installed.
>
> https://trac.macports.org/ticket/63274
>
> Similarly, -L flags the user might specify in the LDFLAGS environment
> variable appear before the -L flags for the project's own directories,
> resulting in build failure if a previous version of notmuch (whose
> libraries differ sufficiently from the new version) was already
> installed.
>
> https://trac.macports.org/ticket/63665

Although I don't consider GNU standards normative for notmuch, there is
some value in doing things a standard way. In particular the way notmuch
uses {C,CPP,LD,CXX}FLAGS follows e.g. [1].

I guess on the Linux / BSD side we expect the configure script to do the
heavy lifting so that manual setting of CPPFLAGS / LDFLAGS at build time
is not needed in general. So one question is why isn't this the case for
macports?

I think there is value in letting individual end-users use these
variables to override things (we just saw a case the other day where
that fixed someone's unique build problem). 

I'm open to ideas for how we can make things easier for macports without
taking away existing functionality for other users.

d

[1]: https://www.gnu.org/prep/standards/html_node/Command-Variables.html.

Re: Fix order of -I and -L flags

[Tomi Ollila]

On Fri, Dec 17 2021, David Bremner wrote:

> Ryan Schmidt <notmuch@ryandesign.com> writes:
>
>> The notmuch build system puts -I and -L flags in the wrong order.
>>
>> Specifically, -I flags the user might specify in the CPPFLAGS
>> environment variable appear before the -I flags for the project's own
>> directories, resulting in build failure if a previous version of
>> notmuch (whose headers differ sufficiently from the new version) was
>> already installed.
>>
>> https://trac.macports.org/ticket/63274
>>
>> Similarly, -L flags the user might specify in the LDFLAGS environment
>> variable appear before the -L flags for the project's own directories,
>> resulting in build failure if a previous version of notmuch (whose
>> libraries differ sufficiently from the new version) was already
>> installed.
>>
>> https://trac.macports.org/ticket/63665
>
> Although I don't consider GNU standards normative for notmuch, there is
> some value in doing things a standard way. In particular the way notmuch
> uses {C,CPP,LD,CXX}FLAGS follows e.g. [1].

Does it ?

I initially thought CFLAGS should be first so that user can modify
anything, but then I thought that CFLAGS should be last just so that
the "project internal" includes are taken first. 

2 things) (1) I was wrong with where user can modify anything: -I's, -L's
in c compiler options are used in order, but (OTOH) (probably) some other
options given later may override previously given option.

then (2) [1] seems to say that

"Put CFLAGS last in the compilation command, after other variables
 containing compiler options, so the user can use CFLAGS to override the
 others. "

^^ that would also say mean that the -I's and -L's given in ${CFLAGS}
would be effective after the -I's and -L' configured...

>
> I guess on the Linux / BSD side we expect the configure script to do the
> heavy lifting so that manual setting of CPPFLAGS / LDFLAGS at build time
> is not needed in general. So one question is why isn't this the case for
> macports?
>
> I think there is value in letting individual end-users use these
> variables to override things (we just saw a case the other day where
> that fixed someone's unique build problem). 

What was the case ?

> I'm open to ideas for how we can make things easier for macports without
> taking away existing functionality for other users.

Would putting CFLAGS last break someone's workflow? Did I understand 
correctly what [1] mean for use of CFLAGS ?

>
> d
>
> [1]: https://www.gnu.org/prep/standards/html_node/Command-Variables.html.


Tomi

Re: Fix order of -I and -L flags

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Fri, Dec 17 2021, David Bremner wrote:
>>
>> Although I don't consider GNU standards normative for notmuch, there is
>> some value in doing things a standard way. In particular the way notmuch
>> uses {C,CPP,LD,CXX}FLAGS follows e.g. [1].
>
> Does it ?
>
> I initially thought CFLAGS should be first so that user can modify
> anything, but then I thought that CFLAGS should be last just so that
> the "project internal" includes are taken first. 

> "Put CFLAGS last in the compilation command, after other variables
>  containing compiler options, so the user can use CFLAGS to override the
>  others. "

That's a good point. I was thinking about CPPFLAGS, because of Ryan's
original question(s).

>
> ^^ that would also say mean that the -I's and -L's given in ${CFLAGS}
> would be effective after the -I's and -L' configured...
>
>>
>> I guess on the Linux / BSD side we expect the configure script to do the
>> heavy lifting so that manual setting of CPPFLAGS / LDFLAGS at build time
>> is not needed in general. So one question is why isn't this the case for
>> macports?
>>
>> I think there is value in letting individual end-users use these
>> variables to override things (we just saw a case the other day where
>> that fixed someone's unique build problem). 
>
> What was the case ?
>

using LDFLAGS

      id:87lf0thhft.fsf@tethera.net

I have to admit I'm a bit fuzzy on how LDFLAGS work. I would imagine
that -L works left to right like -I, but I'm too lazy to check right
now.

>> I'm open to ideas for how we can make things easier for macports without
>> taking away existing functionality for other users.
>
> Would putting CFLAGS last break someone's workflow? Did I understand 
> correctly what [1] mean for use of CFLAGS ?
>

I think you're right, but I think it won't help Ryan.

>>
>> [1]: https://www.gnu.org/prep/standards/html_node/Command-Variables.html.
>
>
> Tomi

Re: Fix order of -I and -L flags

[Ryan Schmidt]

On Dec 20, 2021, at 16:58, David Bremner wrote:

> Tomi Ollila writes:
> 
>> On Fri, Dec 17 2021, David Bremner wrote:
>>> 
>>> Although I don't consider GNU standards normative for notmuch, there is
>>> some value in doing things a standard way. In particular the way notmuch
>>> uses {C,CPP,LD,CXX}FLAGS follows e.g. [1].
>> 
>> Does it ?
>> 
>> I initially thought CFLAGS should be first so that user can modify
>> anything, but then I thought that CFLAGS should be last just so that
>> the "project internal" includes are taken first. 
> 
>> "Put CFLAGS last in the compilation command, after other variables
>> containing compiler options, so the user can use CFLAGS to override the
>> others. "
> 
> That's a good point. I was thinking about CPPFLAGS, because of Ryan's
> original question(s).
> 
>> 
>> ^^ that would also say mean that the -I's and -L's given in ${CFLAGS}
>> would be effective after the -I's and -L' configured...
>> 
>>> 
>>> I guess on the Linux / BSD side we expect the configure script to do the
>>> heavy lifting so that manual setting of CPPFLAGS / LDFLAGS at build time
>>> is not needed in general. So one question is why isn't this the case for
>>> macports?
>>> 
>>> I think there is value in letting individual end-users use these
>>> variables to override things (we just saw a case the other day where
>>> that fixed someone's unique build problem). 
>> 
>> What was the case ?
>> 
> 
> using LDFLAGS
> 
>      id:87lf0thhft.fsf@tethera.net
> 
> I have to admit I'm a bit fuzzy on how LDFLAGS work. I would imagine
> that -L works left to right like -I, but I'm too lazy to check right
> now.
> 
>>> I'm open to ideas for how we can make things easier for macports without
>>> taking away existing functionality for other users.
>> 
>> Would putting CFLAGS last break someone's workflow? Did I understand 
>> correctly what [1] mean for use of CFLAGS ?
>> 
> 
> I think you're right, but I think it won't help Ryan.
> 
>>> 
>>> [1]: https://www.gnu.org/prep/standards/html_node/Command-Variables.html.
>> 
>> 
>> Tomi

Yes, following GNU standards is fine. I have not read them thoroughly but GNU standards surely say to behave the way I requested in my first message. MacPorts has no problem building most GNU projects that adhere to the standards.

Users (or package managers on their behalf) may specify things in CFLAGS, CXXFLAGS, CPPFLAGS, LDFLAGS that are required for their situation. Your build system must place its own flags of the equivalent type BEFORE anything in those user flags variables or else breakage can occur.

For example, a user (or for example MacPorts) may put -I/opt/local/include into CPPFLAGS to indicate that the build should search there for things it does not find in standard places. If you have any -I flags that your build specifies, for example to include files in your distribution (e.g. -Iinclude) you must put them BEFORE $(CPPFLAGS) otherwise user-specified directories will override your build directories which can result in build failure if for example an earlier version of notmuch is already installed.

Same goes for LDFLAGS flags. The user (or MacPorts) may specify -L/opt/local/lib in LDFLAGS. If you have any -L flags pointing to directories where you just built a library (e.g. -Llib), you must put those BEFORE user $(LDFLAGS) so that an already-installed notmuch library does not cause the build to fail.

Same goes for CFLAGS and CXXFLAGS. The user (or MacPorts) may specify -Os or some other optimization flag. If you want to set some optimization flag in your build, you must do so BEFORE user $(CFLAGS) or $(CXXFLAGS) so that the user can override your default optimization value.

Flags are processed in left to right order. For -I and -L flags, each flag specifies a directory in which to search for headers or libraries, respectively. If a file is not found in the first specified path, the second is tried, and so on. For -O optimization flags, the last specified flag takes effect.

[PATCH 1/3] build: move LDFLAGS after notmuch libraries.

[David Bremner]

In [1] Ryan Schmidt reported a problem on macports [2] with notmuch
finding an existing installed version of libnotmuch during the build
when the user specified LDFLAGS including the libnotmuch install
directory.

This change should prevent that.

LDFLAGS also occurs in FINAL_LIBNOTMUCH_LDFLAGS. The only built
library linked to that is util/libtnotmuch_util.a, and that passed as
explicit (relative) path, and is thus not affected by -L.

[1]: id:7851CAB5-4556-4931-A0A2-37003E56C927@ryandesign.com

[2]: The problem does not arise when libnotmuch is installed into a
"system" library path that the compiler/linker searches by default.
---
 Makefile.global | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.global b/Makefile.global
index fe79121d..e6b00815 100644
--- a/Makefile.global
+++ b/Makefile.global
@@ -52,7 +52,7 @@ PV_FILE=bindings/python/notmuch/version.py
 # Smash together user's values with our extra values
 FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CPPFLAGS) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CONFIGURE_CFLAGS)
 FINAL_CXXFLAGS = $(CPPFLAGS) $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CONFIGURE_CXXFLAGS)
-FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lnotmuch_util -Llib -lnotmuch
+FINAL_NOTMUCH_LDFLAGS = -Lutil -lnotmuch_util -Llib -lnotmuch $(LDFLAGS)
 ifeq ($(LIBDIR_IN_LDCONFIG),0)
 FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
 endif
-- 
2.34.1

[PATCH 2/3] build: move CPPFLAGS after source directory includes.

[David Bremner]

In [1] Ryan Schmidt reported a problem on macports [2] with notmuch
finding an existing installed version of notmuch.h during the build.

This is a partial fix; the user might also specify -I in CFLAGS.

[1]: id:7851CAB5-4556-4931-A0A2-37003E56C927@ryandesign.com

[2]: The problem does not arise when notmuch.h is installed into a
"system" include path that the compiler searches by default.
---
 Makefile.global | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile.global b/Makefile.global
index e6b00815..b712dd7b 100644
--- a/Makefile.global
+++ b/Makefile.global
@@ -50,8 +50,8 @@ DETACHED_SIG_FILE=$(TAR_FILE).asc
 PV_FILE=bindings/python/notmuch/version.py
 
 # Smash together user's values with our extra values
-FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CPPFLAGS) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CONFIGURE_CFLAGS)
-FINAL_CXXFLAGS = $(CPPFLAGS) $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CONFIGURE_CXXFLAGS)
+FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CPPFLAGS) $(CONFIGURE_CFLAGS)
+FINAL_CXXFLAGS = $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CPPFLAGS) $(CONFIGURE_CXXFLAGS)
 FINAL_NOTMUCH_LDFLAGS = -Lutil -lnotmuch_util -Llib -lnotmuch $(LDFLAGS)
 ifeq ($(LIBDIR_IN_LDCONFIG),0)
 FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
-- 
2.34.1

[PATCH 3/3] build: move {C,CXX}FLAGS to the end of FINAL_{C,CXX}FLAGS

[David Bremner]

In addition to avoiding problems with user specified include paths
picking up an installed version of notmuch.h, this should also enable
users to override more options (in particular they could override
warning options since the last one takes effect).

[1]: id:7851CAB5-4556-4931-A0A2-37003E56C927@ryandesign.com
---
 Makefile.global | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile.global b/Makefile.global
index b712dd7b..7a7a3c6d 100644
--- a/Makefile.global
+++ b/Makefile.global
@@ -50,8 +50,8 @@ DETACHED_SIG_FILE=$(TAR_FILE).asc
 PV_FILE=bindings/python/notmuch/version.py
 
 # Smash together user's values with our extra values
-FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CPPFLAGS) $(CONFIGURE_CFLAGS)
-FINAL_CXXFLAGS = $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CPPFLAGS) $(CONFIGURE_CXXFLAGS)
+FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(WARN_CFLAGS) $(extra_cflags) $(CPPFLAGS) $(CONFIGURE_CFLAGS) $(CFLAGS)
+FINAL_CXXFLAGS = $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CPPFLAGS) $(CONFIGURE_CXXFLAGS) $(CXXFLAGS)
 FINAL_NOTMUCH_LDFLAGS = -Lutil -lnotmuch_util -Llib -lnotmuch $(LDFLAGS)
 ifeq ($(LIBDIR_IN_LDCONFIG),0)
 FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
-- 
2.34.1

Re: [PATCH 1/3] build: move LDFLAGS after notmuch libraries.

[Tomi Ollila]

On Fri, Dec 24 2021, David Bremner wrote:

> In [1] Ryan Schmidt reported a problem on macports [2] with notmuch
> finding an existing installed version of libnotmuch during the build
> when the user specified LDFLAGS including the libnotmuch install
> directory.
>
> This change should prevent that.
>
> LDFLAGS also occurs in FINAL_LIBNOTMUCH_LDFLAGS. The only built
> library linked to that is util/libtnotmuch_util.a, and that passed as
> explicit (relative) path, and is thus not affected by -L.
>
> [1]: id:7851CAB5-4556-4931-A0A2-37003E56C927@ryandesign.com
>
> [2]: The problem does not arise when libnotmuch is installed into a
> "system" library path that the compiler/linker searches by default.

This series looks good to me, and id:87lf0thhft.fsf@tethera.net
should still work.

Tomi

> ---
>  Makefile.global | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Makefile.global b/Makefile.global
> index fe79121d..e6b00815 100644
> --- a/Makefile.global
> +++ b/Makefile.global
> @@ -52,7 +52,7 @@ PV_FILE=bindings/python/notmuch/version.py
>  # Smash together user's values with our extra values
>  FINAL_CFLAGS = -DNOTMUCH_VERSION=$(VERSION) $(CPPFLAGS) $(CFLAGS) $(WARN_CFLAGS) $(extra_cflags) $(CONFIGURE_CFLAGS)
>  FINAL_CXXFLAGS = $(CPPFLAGS) $(CXXFLAGS) $(WARN_CXXFLAGS) $(extra_cflags) $(extra_cxxflags) $(CONFIGURE_CXXFLAGS)
> -FINAL_NOTMUCH_LDFLAGS = $(LDFLAGS) -Lutil -lnotmuch_util -Llib -lnotmuch
> +FINAL_NOTMUCH_LDFLAGS = -Lutil -lnotmuch_util -Llib -lnotmuch $(LDFLAGS)
>  ifeq ($(LIBDIR_IN_LDCONFIG),0)
>  FINAL_NOTMUCH_LDFLAGS += $(RPATH_LDFLAGS)
>  endif
> -- 
> 2.34.1
>
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: [PATCH 1/3] build: move LDFLAGS after notmuch libraries.

[David Bremner]

David Bremner <david@tethera.net> writes:

> In [1] Ryan Schmidt reported a problem on macports [2] with notmuch
> finding an existing installed version of libnotmuch during the build
> when the user specified LDFLAGS including the libnotmuch install
> directory.
>
> This change should prevent that.

series applied to master, should be part of the eventual 0.35 release