unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / code / Atom feed
* [bug#51198] [PATCH] gnu: Add b2sum.
@ 2021-10-14  4:28 Leo Famulari
  2021-10-26 16:07 ` bug#51198: " Leo Famulari
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Leo Famulari @ 2021-10-14  4:28 UTC (permalink / raw)
  To: 51198

* gnu/packages/crypto.scm (b2sum): New variable.
---
 gnu/packages/crypto.scm | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index 64cdf48d5d..9d16d64ed2 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -1,7 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2014 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015, 2017, 2018, 2019 Ricardo Wurmus <rekado@elephly.net>
-;;; Copyright © 2016, 2017, 2018, 2019 Leo Famulari <leo@famulari.name>
+;;; Copyright © 2016, 2017, 2018, 2019, 2021 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016 Lukas Gradl <lgradl@openmailbox>
 ;;; Copyright © 2016–2021 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2016, 2017 Nikita <nikita@n0.is>
@@ -838,6 +838,42 @@ (define-public libb2
 BLAKE.")
     (license license:public-domain)))
 
+(define-public b2sum
+  ;; Upstream doesn't seem to use a versioned release workflow, so build from
+  ;; a recent commit.
+  (let ((commit "54f4faa4c16ea34bcd59d16e8da46a64b259fc07")
+        (revision "0"))
+    (package
+      (name "b2sum")
+      (version (git-version "20190724" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                       (url "https://github.com/BLAKE2/BLAKE2")
+                       (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32 "04z631v0vzl52g73v390ask5fnzi5wg83lcjkjhpmmymaz0jn152"))))
+      (build-system gnu-build-system)
+      (arguments
+       `(#:make-flags (list (string-append "CC=" ,(cc-for-target))
+                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
+         #:tests? #f ; No test suite
+         #:phases
+         (modify-phases %standard-phases
+           ;; XXX Can add-before and delete be expected to work like this?
+           (add-before 'configure 'change-directory
+                       (lambda _
+                         (chdir "b2sum")))
+           (delete 'configure))))
+      (home-page "https://www.blake2.net/")
+      (synopsis "BLAKE2 checksum tool")
+      (description "BLAKE2 is a cryptographic hash function faster than MD5,
+SHA-1, SHA-2, and SHA-3, yet is at least as secure as SHA-3.")
+      ;; You may also choose to redistribute this program as Apache 2.0 or the
+      ;; OpenSSL license. See 'b2sum/b2sum.c' in the source distribution.
+      (license license:cc0))))
+
 (define-public rhash
   (package
     (name "rhash")
-- 
2.33.0





^ permalink raw reply related	[flat|nested] 17+ messages in thread

* bug#51198: [PATCH] gnu: Add b2sum.
  2021-10-14  4:28 [bug#51198] [PATCH] gnu: Add b2sum Leo Famulari
@ 2021-10-26 16:07 ` Leo Famulari
  2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
  2021-11-27 23:32 ` [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter Leo Famulari
  2 siblings, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2021-10-26 16:07 UTC (permalink / raw)
  To: 51198-done

Pushed as 3d51fd19faee59d9455eebbb957cf6498ecc1220




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-14  4:28 [bug#51198] [PATCH] gnu: Add b2sum Leo Famulari
  2021-10-26 16:07 ` bug#51198: " Leo Famulari
@ 2021-10-26 22:26 ` Nicolò Balzarotti
  2021-10-26 22:53   ` Leo Famulari
                     ` (2 more replies)
  2021-11-27 23:32 ` [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter Leo Famulari
  2 siblings, 3 replies; 17+ messages in thread
From: Nicolò Balzarotti @ 2021-10-26 22:26 UTC (permalink / raw)
  To: Leo Famulari, 51198

Leo Famulari <leo@famulari.name> writes:

> * gnu/packages/crypto.scm (b2sum): New variable.

Hi, I guix pulled and wanted to try this, but I get

>> b2sum --help
> Illegal instruction

(I'm on an old X200 thinkpad).  Might it be that build is not
deterministic (and we should disable some optimization)?

Thanks, Nicolò




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
@ 2021-10-26 22:53   ` Leo Famulari
  2021-10-26 22:54     ` Leo Famulari
  2021-10-26 23:16   ` Leo Famulari
  2022-03-18 17:12   ` bug#51198: " Leo Famulari
  2 siblings, 1 reply; 17+ messages in thread
From: Leo Famulari @ 2021-10-26 22:53 UTC (permalink / raw)
  To: Nicolò Balzarotti; +Cc: 51198

On Wed, Oct 27, 2021 at 12:26:01AM +0200, Nicolò Balzarotti wrote:
> Leo Famulari <leo@famulari.name> writes:
> 
> > * gnu/packages/crypto.scm (b2sum): New variable.
> 
> Hi, I guix pulled and wanted to try this, but I get
> 
> >> b2sum --help
> > Illegal instruction
> 
> (I'm on an old X200 thinkpad).  Might it be that build is not
> deterministic (and we should disable some optimization)?

I guess it's because the package built with "-march=native".

What's the right choice for this parameter?

https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html

CC-ing Mark Weaver because he often has good advice on this subject.

'gnu/packages' does include some "-march=" examples.




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-26 22:53   ` Leo Famulari
@ 2021-10-26 22:54     ` Leo Famulari
  0 siblings, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2021-10-26 22:54 UTC (permalink / raw)
  To: Nicolò Balzarotti; +Cc: Mark H Weaver, 51198

On Tue, Oct 26, 2021 at 06:53:55PM -0400, Leo Famulari wrote:
> I guess it's because the package built with "-march=native".
> 
> What's the right choice for this parameter?
> 
> https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
> 
> CC-ing Mark Weaver because he often has good advice on this subject.
> 
> 'gnu/packages' does include some "-march=" examples.

Er, CC-ing for real now




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
  2021-10-26 22:53   ` Leo Famulari
@ 2021-10-26 23:16   ` Leo Famulari
  2021-10-27  8:44     ` Nicolò Balzarotti
  2022-03-18 17:12   ` bug#51198: " Leo Famulari
  2 siblings, 1 reply; 17+ messages in thread
From: Leo Famulari @ 2021-10-26 23:16 UTC (permalink / raw)
  To: Nicolò Balzarotti; +Cc: 51198

On Wed, Oct 27, 2021 at 12:26:01AM +0200, Nicolò Balzarotti wrote:
> (I'm on an old X200 thinkpad).  Might it be that build is not
> deterministic (and we should disable some optimization)?

As a test, I replaced march=native with march=x86-64.

My benchmark is hashing a ~7GB tree of files [0].

Once the files are cached by the kernel:
"march=native": ~13 seconds
"march=x86-64": ~20 seconds

I wonder if some of the other values of -march are faster...

Or maybe the package could use #:substitutable #f, since b2sum is a very
quick build.

Although BLAKE3 / b3sum [1] is radically faster, b2sum is nice because
it's not in Rust, which is still suboptimal on Guix. With b3sum, my
benchmark takes less than 2 seconds!

[0]
`LC_ALL=C; find "$1" -type f -print0 | sort -z | xargs -0 b2sum | time b2sum`

[1] https://bugs.gnu.org/51289




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-26 23:16   ` Leo Famulari
@ 2021-10-27  8:44     ` Nicolò Balzarotti
  2021-10-27 15:38       ` Leo Famulari
  0 siblings, 1 reply; 17+ messages in thread
From: Nicolò Balzarotti @ 2021-10-27  8:44 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 51198

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> On Wed, Oct 27, 2021 at 12:26:01AM +0200, Nicolò Balzarotti wrote:
>> (I'm on an old X200 thinkpad).  Might it be that build is not
>> deterministic (and we should disable some optimization)?
>
> As a test, I replaced march=native with march=x86-64.
>
> [...]
>
> Or maybe the package could use #:substitutable #f, since b2sum is a very
> quick build.

This blog post [fn:1] on guix-hpc address the "Pre-built binaries
vs. performance" dilemma.

I guess the easiest way is to provide a variant (b2sum-avx or something
like that) with avx enabled.  Else, I'd just go with the unoptimized
version as it happens for many other packages, but let's hear from
others.

Thanks, Nicolò

[fn:1] https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-27  8:44     ` Nicolò Balzarotti
@ 2021-10-27 15:38       ` Leo Famulari
  2021-10-27 15:56         ` Ludovic Courtès
  0 siblings, 1 reply; 17+ messages in thread
From: Leo Famulari @ 2021-10-27 15:38 UTC (permalink / raw)
  To: Nicolò Balzarotti; +Cc: 51198

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

On Wed, Oct 27, 2021 at 10:44:45AM +0200, Nicolò Balzarotti wrote:
> This blog post [fn:1] on guix-hpc address the "Pre-built binaries
> vs. performance" dilemma.
>
> [fn:1] https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/

It's helpful, but it doesn't look like we are using that method in Guix
anymore. It was removed in this commit:

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=969adb235ee34decb65255e1ea821ff0e221ed3d

I guess it learned how to do runtime feature detection?

> I guess the easiest way is to provide a variant (b2sum-avx or something
> like that) with avx enabled.  Else, I'd just go with the unoptimized
> version as it happens for many other packages, but let's hear from
> others.

The easiest thing is require local building, since it's an extremely
cheap build. Only 1.5 seconds on my laptop, total (not just the build
phase). With blis, one had to consider the lengthy build time.

Because performance is critical for a tool like this, and because it's
cheap to build, I've attached a patch to require local building. This is
easier for me than creating a set of package variants that will need to
be expanded for each new generation of CPUs :)

[-- Attachment #2: 0001-gnu-b2sum-Build-on-the-local-machine.patch --]
[-- Type: text/plain, Size: 1776 bytes --]

From 8b862425310cf631c30c823eb0fa2bfd79d36823 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Wed, 27 Oct 2021 11:29:40 -0400
Subject: [PATCH] gnu: b2sum: Build on the local machine.

* gnu/packages/crypto.scm (b2sum)[arguments]: Set '#:substitutable? #f'.
---
 gnu/packages/crypto.scm | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index 3acd147f25..de8bcf1d70 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -858,9 +858,18 @@ (define-public b2sum
                  (base32 "04z631v0vzl52g73v390ask5fnzi5wg83lcjkjhpmmymaz0jn152"))))
       (build-system gnu-build-system)
       (arguments
-       `(#:make-flags (list (string-append "CC=" ,(cc-for-target))
-                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
+       `(;; By default, b2sum uses the compiler to generate instructions
+         ;; tailored to the CPU of the running machine, using "-march=native".
+         ;; This gives a ~1.5x speedup on a Core i5-6300U with a large dataset
+         ;; paged in, whereas compilation of b2sum takes ~1.5 seconds.
+         ;; b2sum does not support run-time feature detection:
+         ;; https://github.com/BLAKE2/BLAKE2/issues/1
+         ;; For more information, see the discussion beginning here:
+         ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51198#13
+         #:substitutable? #f
          #:tests? #f ; No test suite
+         #:make-flags (list (string-append "CC=" ,(cc-for-target))
+                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
          #:phases
          (modify-phases %standard-phases
            (add-before 'build 'change-directory
-- 
2.33.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-27 15:38       ` Leo Famulari
@ 2021-10-27 15:56         ` Ludovic Courtès
  2021-11-03 17:32           ` Leo Famulari
  0 siblings, 1 reply; 17+ messages in thread
From: Ludovic Courtès @ 2021-10-27 15:56 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 51198, Nicolò Balzarotti

Hi!

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Oct 27, 2021 at 10:44:45AM +0200, Nicolò Balzarotti wrote:
>> This blog post [fn:1] on guix-hpc address the "Pre-built binaries
>> vs. performance" dilemma.
>>
>> [fn:1] https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/
>
> It's helpful, but it doesn't look like we are using that method in Guix
> anymore. It was removed in this commit:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=969adb235ee34decb65255e1ea821ff0e221ed3d
>
> I guess it learned how to do runtime feature detection?

BLIS has run-time/load-time feature detection now (which is the main
approach the blog argues for).

>> I guess the easiest way is to provide a variant (b2sum-avx or something
>> like that) with avx enabled.  Else, I'd just go with the unoptimized
>> version as it happens for many other packages, but let's hear from
>> others.
>
> The easiest thing is require local building, since it's an extremely
> cheap build. Only 1.5 seconds on my laptop, total (not just the build
> phase). With blis, one had to consider the lengthy build time.
>
> Because performance is critical for a tool like this, and because it's
> cheap to build, I've attached a patch to require local building. This is
> easier for me than creating a set of package variants that will need to
> be expanded for each new generation of CPUs :)
>
>>From 8b862425310cf631c30c823eb0fa2bfd79d36823 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Wed, 27 Oct 2021 11:29:40 -0400
> Subject: [PATCH] gnu: b2sum: Build on the local machine.
>
> * gnu/packages/crypto.scm (b2sum)[arguments]: Set '#:substitutable? #f'.
> ---
>  gnu/packages/crypto.scm | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
> index 3acd147f25..de8bcf1d70 100644
> --- a/gnu/packages/crypto.scm
> +++ b/gnu/packages/crypto.scm
> @@ -858,9 +858,18 @@ (define-public b2sum
>                   (base32 "04z631v0vzl52g73v390ask5fnzi5wg83lcjkjhpmmymaz0jn152"))))
>        (build-system gnu-build-system)
>        (arguments
> -       `(#:make-flags (list (string-append "CC=" ,(cc-for-target))
> -                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
> +       `(;; By default, b2sum uses the compiler to generate instructions
> +         ;; tailored to the CPU of the running machine, using "-march=native".
> +         ;; This gives a ~1.5x speedup on a Core i5-6300U with a large dataset
> +         ;; paged in, whereas compilation of b2sum takes ~1.5 seconds.
> +         ;; b2sum does not support run-time feature detection:
> +         ;; https://github.com/BLAKE2/BLAKE2/issues/1
> +         ;; For more information, see the discussion beginning here:
> +         ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51198#13
> +         #:substitutable? #f

It’s okay but not entirely sufficient: on a cluster setup, you typically
talk to a daemon that’s on another machine, so the CPU features it’ll
detect there may be different from those you’ll use.  Likewise, as
discussed on IRC, you’d also need #:local-build? #t.

Anyway, I’d suggest using ‘guix hash -H blake2s-256’ or similar (it uses
libgcrypt, which does the right thing).  I think the latest Coreutils
provide a generic ‘cksum’, too, that probably does the right thing.

Ludo’.




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-10-27 15:56         ` Ludovic Courtès
@ 2021-11-03 17:32           ` Leo Famulari
  2021-11-06 16:45             ` Ludovic Courtès
  0 siblings, 1 reply; 17+ messages in thread
From: Leo Famulari @ 2021-11-03 17:32 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 51198, Nicolò Balzarotti

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Wed, Oct 27, 2021 at 05:56:31PM +0200, Ludovic Courtès wrote:
> It’s okay but not entirely sufficient: on a cluster setup, you typically
> talk to a daemon that’s on another machine, so the CPU features it’ll
> detect there may be different from those you’ll use.  Likewise, as
> discussed on IRC, you’d also need #:local-build? #t.

It seems that #:local-build? is not available in gnu-build-system. Is
that correct?

Using the attached patch, building fails with "Unrecognized keyword:
#:local-build?".

Is there a way to turn it on for packages using gnu-build-system?

[-- Attachment #2: 0001-gnu-b2sum-Build-on-the-local-machine.patch --]
[-- Type: text/plain, Size: 1829 bytes --]

From 77a063e797849332b784decab32f29be3d13f031 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Wed, 27 Oct 2021 11:29:40 -0400
Subject: [PATCH] gnu: b2sum: Build on the local machine.

* gnu/packages/crypto.scm (b2sum)[arguments]: Set '#:substitutable? #f' and
'#:local-build? #t'.
---
 gnu/packages/crypto.scm | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index f48fc23c1f..7ef8504eb7 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -858,9 +858,19 @@ (define-public b2sum
                  (base32 "04z631v0vzl52g73v390ask5fnzi5wg83lcjkjhpmmymaz0jn152"))))
       (build-system gnu-build-system)
       (arguments
-       `(#:make-flags (list (string-append "CC=" ,(cc-for-target))
-                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
+       `(;; By default, b2sum uses the compiler to generate instructions
+         ;; tailored to the CPU of the running machine, using "-march=native".
+         ;; This gives a ~1.5x speedup on a Core i5-6300U with a large dataset
+         ;; paged in, whereas compilation of b2sum takes ~1.5 seconds.
+         ;; b2sum does not support run-time feature detection:
+         ;; https://github.com/BLAKE2/BLAKE2/issues/1
+         ;; For more information, see the discussion beginning here:
+         ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51198#13
+         #:substitutable? #f
+         #:local-build? #t
          #:tests? #f ; No test suite
+         #:make-flags (list (string-append "CC=" ,(cc-for-target))
+                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
          #:phases
          (modify-phases %standard-phases
            (add-before 'build 'change-directory
-- 
2.33.1


^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-11-03 17:32           ` Leo Famulari
@ 2021-11-06 16:45             ` Ludovic Courtès
  2021-11-27 23:36               ` Leo Famulari
  0 siblings, 1 reply; 17+ messages in thread
From: Ludovic Courtès @ 2021-11-06 16:45 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 51198, Nicolò Balzarotti

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Oct 27, 2021 at 05:56:31PM +0200, Ludovic Courtès wrote:
>> It’s okay but not entirely sufficient: on a cluster setup, you typically
>> talk to a daemon that’s on another machine, so the CPU features it’ll
>> detect there may be different from those you’ll use.  Likewise, as
>> discussed on IRC, you’d also need #:local-build? #t.
>
> It seems that #:local-build? is not available in gnu-build-system. Is
> that correct?

Oh right, it’s missing from (guix build-system gnu).  We could easily
add it.

Though again, it would remain an approximation of what we really want.

HTH!

Ludo’.




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter.
  2021-10-14  4:28 [bug#51198] [PATCH] gnu: Add b2sum Leo Famulari
  2021-10-26 16:07 ` bug#51198: " Leo Famulari
  2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
@ 2021-11-27 23:32 ` Leo Famulari
  2021-11-27 23:32   ` [bug#51198] [PATCH v2 2/2] gnu: b2sum: Build on the local machine Leo Famulari
  2021-12-01 15:34   ` [bug#51198] [PATCH] gnu: Add b2sum zimoun
  2 siblings, 2 replies; 17+ messages in thread
From: Leo Famulari @ 2021-11-27 23:32 UTC (permalink / raw)
  To: 51198

* guix/build-system/gnu.scm (gnu-build, gnu-cross-build): Add #:local-build?
parameter and pass it to 'build-expression->derivation'.
---
 guix/build-system/gnu.scm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/guix/build-system/gnu.scm b/guix/build-system/gnu.scm
index 6b481ad45c..fc923f0c5a 100644
--- a/guix/build-system/gnu.scm
+++ b/guix/build-system/gnu.scm
@@ -349,6 +349,7 @@ (define* (gnu-build store name input-drvs
                     (build (nix-system->gnu-triplet system))
                     (imported-modules %gnu-build-system-modules)
                     (modules %default-modules)
+                    (local-build? #f)
                     (substitutable? #t)
                     allowed-references
                     disallowed-references)
@@ -429,6 +430,7 @@ (define guile-for-build
                                 #:inputs input-drvs
                                 #:outputs outputs
                                 #:modules imported-modules
+                                #:local-build? local-build?
                                 #:substitutable? substitutable?
 
                                 #:allowed-references
@@ -504,6 +506,7 @@ (define* (gnu-cross-build store name
                           (build (nix-system->gnu-triplet system))
                           (imported-modules %gnu-build-system-modules)
                           (modules %default-modules)
+                          (local-build? #f)
                           (substitutable? #t)
                           allowed-references
                           disallowed-references)
@@ -595,6 +598,7 @@ (define guile-for-build
                                 #:inputs (append native-drvs target-drvs)
                                 #:outputs outputs
                                 #:modules imported-modules
+                                #:local-build? local-build?
                                 #:substitutable? substitutable?
 
                                 #:allowed-references
-- 
2.33.1





^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH v2 2/2] gnu: b2sum: Build on the local machine.
  2021-11-27 23:32 ` [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter Leo Famulari
@ 2021-11-27 23:32   ` Leo Famulari
  2021-12-01 15:34   ` [bug#51198] [PATCH] gnu: Add b2sum zimoun
  1 sibling, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2021-11-27 23:32 UTC (permalink / raw)
  To: 51198

* gnu/packages/crypto.scm (b2sum)[arguments]: Set '#:substitutable? #f' and
'#:local-build? #t'.
---
 gnu/packages/crypto.scm | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index b6f2010a80..d4c6717eb5 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -858,9 +858,19 @@ (define-public b2sum
                  (base32 "04z631v0vzl52g73v390ask5fnzi5wg83lcjkjhpmmymaz0jn152"))))
       (build-system gnu-build-system)
       (arguments
-       `(#:make-flags (list (string-append "CC=" ,(cc-for-target))
-                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
+       `(;; By default, b2sum uses the compiler to generate instructions
+         ;; tailored to the CPU of the running machine, using "-march=native".
+         ;; This gives a ~1.5x speedup on a Core i5-6300U with a large dataset
+         ;; paged in, whereas compilation of b2sum takes ~1.5 seconds.
+         ;; b2sum does not support run-time feature detection:
+         ;; https://github.com/BLAKE2/BLAKE2/issues/1
+         ;; For more information, see the discussion beginning here:
+         ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51198#13
+         #:substitutable? #f
+         #:local-build? #t
          #:tests? #f ; No test suite
+         #:make-flags (list (string-append "CC=" ,(cc-for-target))
+                            (string-append "PREFIX=" (assoc-ref %outputs "out")))
          #:phases
          (modify-phases %standard-phases
            (add-before 'build 'change-directory
-- 
2.33.1





^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-11-06 16:45             ` Ludovic Courtès
@ 2021-11-27 23:36               ` Leo Famulari
  0 siblings, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2021-11-27 23:36 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 51198, Nicolò Balzarotti

On Sat, Nov 06, 2021 at 05:45:01PM +0100, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> > It seems that #:local-build? is not available in gnu-build-system. Is
> > that correct?
> 
> Oh right, it’s missing from (guix build-system gnu).  We could easily
> add it.

I've sent patches to make this change. I checked that the derivation of
'hello' is unchanged when building without cross-compilation.

https://issues.guix.gnu.org/51198#12




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-11-27 23:32 ` [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter Leo Famulari
  2021-11-27 23:32   ` [bug#51198] [PATCH v2 2/2] gnu: b2sum: Build on the local machine Leo Famulari
@ 2021-12-01 15:34   ` zimoun
  2021-12-01 17:50     ` Leo Famulari
  1 sibling, 1 reply; 17+ messages in thread
From: zimoun @ 2021-12-01 15:34 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 51198

Hi Leo,

On Sat, 27 Nov 2021 at 18:32, Leo Famulari <leo@famulari.name> wrote:
> * guix/build-system/gnu.scm (gnu-build, gnu-cross-build): Add #:local-build?
> parameter and pass it to 'build-expression->derivation'.

The manual needs to be updated, I guess.  Something there:

<https://guix.gnu.org/manual/devel/en/guix.html#index-gnu_002dbuild_002dsystem>


This gnu-build-system triggers a world rebuild, isn’t it?  Maybe, it
could go to core-updates-frozen now, no? (Now = before merge :-))

Cheers,
simon




^ permalink raw reply	[flat|nested] 17+ messages in thread

* [bug#51198] [PATCH] gnu: Add b2sum.
  2021-12-01 15:34   ` [bug#51198] [PATCH] gnu: Add b2sum zimoun
@ 2021-12-01 17:50     ` Leo Famulari
  0 siblings, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2021-12-01 17:50 UTC (permalink / raw)
  To: zimoun; +Cc: 51198

On Wed, Dec 01, 2021 at 04:34:09PM +0100, zimoun wrote:
> On Sat, 27 Nov 2021 at 18:32, Leo Famulari <leo@famulari.name> wrote:
> > * guix/build-system/gnu.scm (gnu-build, gnu-cross-build): Add #:local-build?
> > parameter and pass it to 'build-expression->derivation'.
> 
> The manual needs to be updated, I guess.  Something there:
> 
> <https://guix.gnu.org/manual/devel/en/guix.html#index-gnu_002dbuild_002dsystem>

Good idea, I'll send a revised patch series.

> This gnu-build-system triggers a world rebuild, isn’t it?  Maybe, it
> could go to core-updates-frozen now, no? (Now = before merge :-))

As far as I can tell, this does not change any derivations, except of
course if a package definition uses the new #:local-build? option. So it
should be okay for the master branch, if it's a change that we want to
make.




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#51198: [PATCH] gnu: Add b2sum.
  2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
  2021-10-26 22:53   ` Leo Famulari
  2021-10-26 23:16   ` Leo Famulari
@ 2022-03-18 17:12   ` Leo Famulari
  2 siblings, 0 replies; 17+ messages in thread
From: Leo Famulari @ 2022-03-18 17:12 UTC (permalink / raw)
  To: Nicolò Balzarotti; +Cc: 51198-done

On Wed, Oct 27, 2021 at 12:26:01AM +0200, Nicolò Balzarotti wrote:
> Leo Famulari <leo@famulari.name> writes:
> 
> > * gnu/packages/crypto.scm (b2sum): New variable.
> 
> Hi, I guix pulled and wanted to try this, but I get
> 
> >> b2sum --help
> > Illegal instruction
> 
> (I'm on an old X200 thinkpad).  Might it be that build is not
> deterministic (and we should disable some optimization)?

I just pushed commit 386adb6df0e00e7170df0cdfaf0e04c38d0f3e11, which
removes the "march=native" compiler flag, so that the package should
work generically.

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=386adb6df0e00e7170df0cdfaf0e04c38d0f3e11

This commit also enables the 'tunable?' property of the package, so that
one can use this program with the highest performance in a way that is
tailored to their hardware.

Just add the --tune command-line option to any Guix command that
provides this package and it should do the right thing.

Thanks Nicolò for reporting the problem and thanks to Ludovic for
developing the solution.




^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2022-03-18 17:13 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-14  4:28 [bug#51198] [PATCH] gnu: Add b2sum Leo Famulari
2021-10-26 16:07 ` bug#51198: " Leo Famulari
2021-10-26 22:26 ` [bug#51198] " Nicolò Balzarotti
2021-10-26 22:53   ` Leo Famulari
2021-10-26 22:54     ` Leo Famulari
2021-10-26 23:16   ` Leo Famulari
2021-10-27  8:44     ` Nicolò Balzarotti
2021-10-27 15:38       ` Leo Famulari
2021-10-27 15:56         ` Ludovic Courtès
2021-11-03 17:32           ` Leo Famulari
2021-11-06 16:45             ` Ludovic Courtès
2021-11-27 23:36               ` Leo Famulari
2022-03-18 17:12   ` bug#51198: " Leo Famulari
2021-11-27 23:32 ` [bug#51198] [PATCH v2 1/2] build-system/gnu: Add #:local-build? parameter Leo Famulari
2021-11-27 23:32   ` [bug#51198] [PATCH v2 2/2] gnu: b2sum: Build on the local machine Leo Famulari
2021-12-01 15:34   ` [bug#51198] [PATCH] gnu: Add b2sum zimoun
2021-12-01 17:50     ` Leo Famulari

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).