all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
       [not found] <87r37gp8dt.fsf@shlevy-laptop.i-did-not-set--mail-host-address--so-tickle-me>
@ 2016-10-18  9:25 ` Ludovic Courtès
       [not found]   ` <87vawqm2je.fsf-mXXj517/zsQ@public.gmane.org>
  2016-10-18 11:19   ` Shea Levy
  0 siblings, 2 replies; 6+ messages in thread
From: Ludovic Courtès @ 2016-10-18  9:25 UTC (permalink / raw)
  To: nix-dev, gcc, cfe-dev, Shea Levy; +Cc: Guix-devel

Hi Shea,

Shea Levy skribis:

> Unlike the traditional approach of installing system libraries into one
> central location like /usr/{lib,include}, the nix package manager [1]
> installs each package into it's own prefix
> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
> each package is built in its own environment determined from its
> explicitly listed dependencies, regardless of what else is installed on
> the system. Because not all package build scripts properly respect
> CFLAGS etc., we currently wrap the compiler [2] to respect custom
> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>
> Unfortunately, as you can see if you click through the link or look
> through the git history, the wrapper is quite complex (frankly, hacky)

> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Guix avoids the compiler wrapper altogether like this:

  • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.

  • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
    The comment in that file explains why the other options considered
    were unsuitable.

  • We modify the built-in “lib” spec of GCC to add the necessary -L and
    -rpath flags:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.

  • Likewise, we tell Clang where to find libc and friends:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.

This is not too intrusive and more robust than wrapping everything.

I suppose GCC and Clang could facilitate this by providing configure
options to augment the “lib” spec, specify the location of libc alone,
or something along these lines.

Thoughts?

Ludo’.
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
       [not found]   ` <87vawqm2je.fsf-mXXj517/zsQ@public.gmane.org>
@ 2016-10-18 11:03     ` Shea Levy via cfe-dev
  0 siblings, 0 replies; 6+ messages in thread
From: Shea Levy via cfe-dev @ 2016-10-18 11:03 UTC (permalink / raw)
  To: Ludovic Courtès, nix-dev-rGrgPyRx506NN8uzcEdRPYRWq/SkRNHw,
	gcc-/MQLu3FmUzdAfugRpC6u6w, cfe-dev-NBbBogny7ofFcdTEL8lfRQ
  Cc: Guix-devel


[-- Attachment #1.1: Type: text/plain, Size: 2742 bytes --]

Hey Ludo’,

Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.

Will look at your gcc and clang patches as well, thank you!

~Shea

Ludovic Courtès <ludo-mXXj517/zsQ@public.gmane.org> writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
>     The comment in that file explains why the other options considered
>     were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
>     -rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
>   • Likewise, we tell Clang where to find libc and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
  2016-10-18  9:25 ` [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment Ludovic Courtès
       [not found]   ` <87vawqm2je.fsf-mXXj517/zsQ@public.gmane.org>
@ 2016-10-18 11:19   ` Shea Levy
  2016-10-18 12:59     ` Ludovic Courtès
  1 sibling, 1 reply; 6+ messages in thread
From: Shea Levy @ 2016-10-18 11:19 UTC (permalink / raw)
  To: Ludovic Courtès, nix-dev, gcc, cfe-dev; +Cc: Guix-devel


[-- Attachment #1.1: Type: text/plain, Size: 2625 bytes --]

Hi Ludo’,

Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?

Thanks,
Shea

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n296>.
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in#n42>.
>     The comment in that file explains why the other options considered
>     were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
>     -rpath flags:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gcc.scm#n218>.
>
>   • Likewise, we tell Clang where to find libc and friends:
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch>
>     <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/llvm.scm#n161>.
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

[-- Attachment #2: Type: text/plain, Size: 149 bytes --]

_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
  2016-10-18 11:19   ` Shea Levy
@ 2016-10-18 12:59     ` Ludovic Courtès
       [not found]       ` <8737jtlsnw.fsf-mXXj517/zsQ@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2016-10-18 12:59 UTC (permalink / raw)
  To: Shea Levy; +Cc: Guix-devel, gcc, nix-dev, cfe-dev

Hi!

Shea Levy <shea@shealevy.com> skribis:

> Your patches look good! My biggest concern is how the ld wrapper behaves
> in the presence of response files. Have you tested that?

It surely doesn’t (yet?).

However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
so I guess it’s fine to ignore response files in the ld wrapper.

Ludo’.

¹ https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
       [not found]       ` <8737jtlsnw.fsf-mXXj517/zsQ@public.gmane.org>
@ 2016-10-18 13:46         ` Nathan Froyd via cfe-dev
       [not found]           ` <CAGepuV+VpTJfu=RUtEioxx47QgKDCvkeN40wisGt9aceXj1mDg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Nathan Froyd via cfe-dev @ 2016-10-18 13:46 UTC (permalink / raw)
  To: Ludovic Courtès
  Cc: Guix-devel, nix-dev-rGrgPyRx506NN8uzcEdRPYRWq/SkRNHw,
	cfe-dev-NBbBogny7ofFcdTEL8lfRQ, gcc-/MQLu3FmUzdAfugRpC6u6w

On Tue, Oct 18, 2016 at 8:59 AM, Ludovic Courtès via cfe-dev
<cfe-dev@lists.llvm.org> wrote:
> Shea Levy <shea@shealevy.com> skribis:
>
>> Your patches look good! My biggest concern is how the ld wrapper behaves
>> in the presence of response files. Have you tested that?
>
> It surely doesn’t (yet?).
>
> However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
> the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
> so I guess it’s fine to ignore response files in the ld wrapper.

GCC will pass response files to ld when response files were used in
the invocation of GCC.

-Nathan
_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment
       [not found]           ` <CAGepuV+VpTJfu=RUtEioxx47QgKDCvkeN40wisGt9aceXj1mDg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2016-10-27 17:11             ` Will Dietz via cfe-dev
  0 siblings, 0 replies; 6+ messages in thread
From: Will Dietz via cfe-dev @ 2016-10-27 17:11 UTC (permalink / raw)
  To: Nathan Froyd, Ludovic Courtès
  Cc: Guix-devel, gcc-/MQLu3FmUzdAfugRpC6u6w,
	nix-dev-rGrgPyRx506NN8uzcEdRPYRWq/SkRNHw,
	cfe-dev-NBbBogny7ofFcdTEL8lfRQ


[-- Attachment #1.1: Type: text/plain, Size: 1751 bytes --]

Yes! I'm very interested in this as well!

One piece of information to add: it might be worth investigating something
more general like the configuration files used by ELLCC in controlling
their cross-compilers.

This was somewhat recently discussed here:

http://lists.llvm.org/pipermail/llvm-dev/2015-July/087907.html

And looks like the most recent description of it is here:

http://ellcc.org/blog/?p=13246

This would be rather useful in Nix for creating cross-compilers as well.

AFAIK this isn't immediately suited to your needs but I think that it's the
right direction to go
and it'd be ideal to support both use-cases if we are talking about
upstream support.

~Will


On Tue, Oct 18, 2016 at 8:46 AM Nathan Froyd via cfe-dev <
cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org> wrote:

> On Tue, Oct 18, 2016 at 8:59 AM, Ludovic Courtès via cfe-dev
> <cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org> wrote:
> > Shea Levy <shea-yfkUTty7RcRWk0Htik3J/w@public.gmane.org> skribis:
> >
> >> Your patches look good! My biggest concern is how the ld wrapper behaves
> >> in the presence of response files. Have you tested that?
> >
> > It surely doesn’t (yet?).
> >
> > However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
> > the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
> > so I guess it’s fine to ignore response files in the ld wrapper.
>
> GCC will pass response files to ld when response files were used in
> the invocation of GCC.
>
> -Nathan
> _______________________________________________
> cfe-dev mailing list
> cfe-dev-NBbBogny7ofFcdTEL8lfRQ@public.gmane.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

[-- Attachment #1.2: Type: text/html, Size: 3257 bytes --]

[-- Attachment #2: Type: text/plain, Size: 147 bytes --]

_______________________________________________
cfe-dev mailing list
cfe-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-10-27 17:11 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <87r37gp8dt.fsf@shlevy-laptop.i-did-not-set--mail-host-address--so-tickle-me>
2016-10-18  9:25 ` [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment Ludovic Courtès
     [not found]   ` <87vawqm2je.fsf-mXXj517/zsQ@public.gmane.org>
2016-10-18 11:03     ` Shea Levy via cfe-dev
2016-10-18 11:19   ` Shea Levy
2016-10-18 12:59     ` Ludovic Courtès
     [not found]       ` <8737jtlsnw.fsf-mXXj517/zsQ@public.gmane.org>
2016-10-18 13:46         ` Nathan Froyd via cfe-dev
     [not found]           ` <CAGepuV+VpTJfu=RUtEioxx47QgKDCvkeN40wisGt9aceXj1mDg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2016-10-27 17:11             ` Will Dietz via cfe-dev

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.