all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* build system option to allow CPU optimizations?
@ 2021-11-24 12:10 Ricardo Wurmus
  2021-11-26  0:07 ` zimoun
  0 siblings, 1 reply; 6+ messages in thread
From: Ricardo Wurmus @ 2021-11-24 12:10 UTC (permalink / raw)
  To: guix-devel

Hi Guix,

currently we patch source code and build systems to ensure that no 
special instructions are used that would not be portable, 
e.g. AVX2, SSE4.1 etc.  What do you think of adding a build system 
option that would allow users to restore these optimizations?

The build phases that patch out these features would have to check 
for that build system option, much like they check the TESTS? 
option before attempting to run tests.

What do you think?

-- 
Ricardo


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: build system option to allow CPU optimizations?
  2021-11-24 12:10 build system option to allow CPU optimizations? Ricardo Wurmus
@ 2021-11-26  0:07 ` zimoun
  2021-11-28 17:36   ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: zimoun @ 2021-11-26  0:07 UTC (permalink / raw)
  To: Ricardo Wurmus, guix-devel

Hi Ricardo,

On Wed, 24 Nov 2021 at 13:10, Ricardo Wurmus <rekado@elephly.net> wrote:

> The build phases that patch out these features would have to check 
> for that build system option, much like they check the TESTS? 
> option before attempting to run tests.

Then it could be a transformation.   The idea sounds good to me.


Cheers,
simon


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: build system option to allow CPU optimizations?
  2021-11-26  0:07 ` zimoun
@ 2021-11-28 17:36   ` Ludovic Courtès
  2021-11-28 18:20     ` Ricardo Wurmus
  2021-12-14  3:36     ` Maxim Cournoyer
  0 siblings, 2 replies; 6+ messages in thread
From: Ludovic Courtès @ 2021-11-28 17:36 UTC (permalink / raw)
  To: zimoun; +Cc: guix-devel

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 24 Nov 2021 at 13:10, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> The build phases that patch out these features would have to check 
>> for that build system option, much like they check the TESTS? 
>> option before attempting to run tests.
>
> Then it could be a transformation.   The idea sounds good to me.

I’ve been working on it last week with my HPC hat on.

To be clear, I think in may cases, passing ‘-march’ like you suggest is
the wrong approach; instead software should use (and usually does use)
function multi-versioning:

  https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/

I found one case though where this is not possible: C++ header-only
libraries such as Eigen contain hand-optimized vectorized routines,
selected at build time, but we end up compiling Eigen users as the
x86_64/AArch64 baseline, which is a waste.  (If you do know of other
problematic cases, I’m interested in taking a look!)

My solution to that is “package multi-versioning” via a transformation
option.  Hopefully I’ll submit preliminary patches within a week or so!

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: build system option to allow CPU optimizations?
  2021-11-28 17:36   ` Ludovic Courtès
@ 2021-11-28 18:20     ` Ricardo Wurmus
  2021-12-20 17:40       ` Ludovic Courtès
  2021-12-14  3:36     ` Maxim Cournoyer
  1 sibling, 1 reply; 6+ messages in thread
From: Ricardo Wurmus @ 2021-11-28 18:20 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel


Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> On Wed, 24 Nov 2021 at 13:10, Ricardo Wurmus 
>> <rekado@elephly.net> wrote:
>>
>>> The build phases that patch out these features would have to 
>>> check 
>>> for that build system option, much like they check the TESTS? 
>>> option before attempting to run tests.
>>
>> Then it could be a transformation.   The idea sounds good to 
>> me.
>
> I’ve been working on it last week with my HPC hat on.
>
> To be clear, I think in may cases, passing ‘-march’ like you 
> suggest is
> the wrong approach; instead software should use (and usually 
> does use)
> function multi-versioning:
>
>   https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/

It may very well be the wrong approach in principle, but I also 
think that it’s a neat escape hatch for specific use cases. 
Separating reproducibility patching makes the package 
transformation mechanism more powerful and appealing.  Much like 
respecting TESTS? makes it easy for users of modified packages to 
bypass a failing test suite, making patching of Makefiles to 
remove CPU tuning conditional would make for much less complex 
custom package definitions.

> I found one case though where this is not possible: C++ 
> header-only
> libraries such as Eigen contain hand-optimized vectorized 
> routines,
> selected at build time, but we end up compiling Eigen users as 
> the
> x86_64/AArch64 baseline, which is a waste.  (If you do know of 
> other
> problematic cases, I’m interested in taking a look!)
>
> My solution to that is “package multi-versioning” via a 
> transformation
> option.  Hopefully I’ll submit preliminary patches within a week 
> or so!

Oh, exciting!

-- 
Ricardo


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: build system option to allow CPU optimizations?
  2021-11-28 17:36   ` Ludovic Courtès
  2021-11-28 18:20     ` Ricardo Wurmus
@ 2021-12-14  3:36     ` Maxim Cournoyer
  1 sibling, 0 replies; 6+ messages in thread
From: Maxim Cournoyer @ 2021-12-14  3:36 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

Hi Ludovic,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> On Wed, 24 Nov 2021 at 13:10, Ricardo Wurmus <rekado@elephly.net> wrote:
>>
>>> The build phases that patch out these features would have to check 
>>> for that build system option, much like they check the TESTS? 
>>> option before attempting to run tests.
>>
>> Then it could be a transformation.   The idea sounds good to me.
>
> I’ve been working on it last week with my HPC hat on.
>
> To be clear, I think in may cases, passing ‘-march’ like you suggest is
> the wrong approach; instead software should use (and usually does use)
> function multi-versioning:
>
>   https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/
>
> I found one case though where this is not possible: C++ header-only
> libraries such as Eigen contain hand-optimized vectorized routines,
> selected at build time, but we end up compiling Eigen users as the
> x86_64/AArch64 baseline, which is a waste.  (If you do know of other
> problematic cases, I’m interested in taking a look!)

I think that 'atlas' is such an example of a package that uses
multi-versioning but fails to build reproducibly depending on the exact
CPU it was built.  I've reported that here [0].

[0]  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51536

Thank you,

Maxim


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: build system option to allow CPU optimizations?
  2021-11-28 18:20     ` Ricardo Wurmus
@ 2021-12-20 17:40       ` Ludovic Courtès
  0 siblings, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2021-12-20 17:40 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:

[...]

> It may very well be the wrong approach in principle, but I also think
> that it’s a neat escape hatch for specific use cases. Separating
> reproducibility patching makes the package transformation mechanism
> more powerful and appealing.  Much like respecting TESTS? makes it
> easy for users of modified packages to bypass a failing test suite,
> making patching of Makefiles to remove CPU tuning conditional would
> make for much less complex custom package definitions.
>
>> I found one case though where this is not possible: C++ header-only
>> libraries such as Eigen contain hand-optimized vectorized routines,
>> selected at build time, but we end up compiling Eigen users as the
>> x86_64/AArch64 baseline, which is a waste.  (If you do know of other
>> problematic cases, I’m interested in taking a look!)
>>
>> My solution to that is “package multi-versioning” via a
>> transformation
>> option.  Hopefully I’ll submit preliminary patches within a week or
>> so!
>
> Oh, exciting!

I forgot to mention it here, but it’s available for testing and probably
even ready to merge:

  https://issues.guix.gnu.org/52283

I think it makes an option to dismiss ‘-march’ removal unnecessary; or,
put differently, it achieves the same.

I’m interested in seeing which packages people would mark as “tunable”
and what performance gains it gives!

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-12-20 18:11 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-24 12:10 build system option to allow CPU optimizations? Ricardo Wurmus
2021-11-26  0:07 ` zimoun
2021-11-28 17:36   ` Ludovic Courtès
2021-11-28 18:20     ` Ricardo Wurmus
2021-12-20 17:40       ` Ludovic Courtès
2021-12-14  3:36     ` Maxim Cournoyer

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.