Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Ludovic Courtès]

Hi Shea,

Shea Levy skribis:

> Unlike the traditional approach of installing system libraries into one
> central location like /usr/{lib,include}, the nix package manager [1]
> installs each package into it's own prefix
> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
> each package is built in its own environment determined from its
> explicitly listed dependencies, regardless of what else is installed on
> the system. Because not all package build scripts properly respect
> CFLAGS etc., we currently wrap the compiler [2] to respect custom
> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>
> Unfortunately, as you can see if you click through the link or look
> through the git history, the wrapper is quite complex (frankly, hacky)

> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Guix avoids the compiler wrapper altogether like this:

  • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.

  • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
    The comment in that file explains why the other options considered
    were unsuitable.

  • We modify the built-in “lib” spec of GCC to add the necessary -L and
    -rpath flags:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.

  • Likewise, we tell Clang where to find libc and friends:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.

This is not too intrusive and more robust than wrapping everything.

I suppose GCC and Clang could facilitate this by providing configure
options to augment the “lib” spec, specify the location of libc alone,
or something along these lines.

Thoughts?

Ludo’.
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Shea Levy via cfe-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2742 bytes --]

Hey Ludo’,

Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.

Will look at your gcc and clang patches as well, thank you!

~Shea

Ludovic Courtès <ludo-mXXj517/zsQ@public.gmane.org> writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
>     The comment in that file explains why the other options considered
>     were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
>     -rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
>   • Likewise, we tell Clang where to find libc and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Shea Levy]

[-- Attachment #1.1: Type: text/plain, Size: 2625 bytes --]

Hi Ludo’,

Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?

Thanks,
Shea

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
>     The comment in that file explains why the other options considered
>     were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
>     -rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
>   • Likewise, we tell Clang where to find libc and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

[-- Attachment #2: Type: text/plain, Size: 149 bytes --]

_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Ludovic Courtès]

Hi!

Shea Levy <shea@shealevy.com> skribis:

> Your patches look good! My biggest concern is how the ld wrapper behaves
> in the presence of response files. Have you tested that?

It surely doesn’t (yet?).

However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
so I guess it’s fine to ignore response files in the ld wrapper.

Ludo’.

¹ https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Nathan Froyd via cfe-dev]

On Tue, Oct 18, 2016 at 8:59 AM, Ludovic Courtès via cfe-dev
<cfe-dev@lists.llvm.org> wrote:
> Shea Levy <shea@shealevy.com> skribis:
>
>> Your patches look good! My biggest concern is how the ld wrapper behaves
>> in the presence of response files. Have you tested that?
>
> It surely doesn’t (yet?).
>
> However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
> the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
> so I guess it’s fine to ignore response files in the ld wrapper.

GCC will pass response files to ld when response files were used in
the invocation of GCC.

-Nathan
_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Will Dietz via cfe-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1751 bytes --]

Yes! I'm very interested in this as well!

One piece of information to add: it might be worth investigating something
more general like the configuration files used by ELLCC in controlling
their cross-compilers.

This was somewhat recently discussed here:

http://lists.llvm.org/pipermail/llvm-dev/2015-July/087907.html

And looks like the most recent description of it is here:

http://ellcc.org/blog/?p=13246

This would be rather useful in Nix for creating cross-compilers as well.

AFAIK this isn't immediately suited to your needs but I think that it's the
right direction to go
and it'd be ideal to support both use-cases if we are talking about
upstream support.

~Will


On Tue, Oct 18, 2016 at 8:46 AM Nathan Froyd via cfe-dev <
cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org> wrote:

> On Tue, Oct 18, 2016 at 8:59 AM, Ludovic Courtès via cfe-dev
> <cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org> wrote:
> > Shea Levy <shea-yfkUTty7RcRWk0Htik3J/w@public.gmane.org> skribis:
> >
> >> Your patches look good! My biggest concern is how the ld wrapper behaves
> >> in the presence of response files. Have you tested that?
> >
> > It surely doesn’t (yet?).
> >
> > However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
> > the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
> > so I guess it’s fine to ignore response files in the ld wrapper.
>
> GCC will pass response files to ld when response files were used in
> the invocation of GCC.
>
> -Nathan
> _______________________________________________
> cfe-dev mailing list
> cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

[-- Attachment #1.2: Type: text/html, Size: 3257 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev