[bug#58903] [PATCH] gnu: nix: Update to 2.11.1.

[bug#58903] [PATCH] gnu: nix: Update to 2.11.1.

[phodina via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 144 bytes --]

Hi,

here's update of Nix daemon. Unfortunately, there is one test that doesn't return the location '/nix/store' so it's substituted.

----
Petr

[-- Attachment #1.2: Type: text/html, Size: 296 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-nix-Update-to-2.11.1.patch --]
[-- Type: text/x-patch; name=0001-gnu-nix-Update-to-2.11.1.patch, Size: 4344 bytes --]

From 6c9fe1aa54189afa36a85610e0a537581a495ad4 Mon Sep 17 00:00:00 2001
From: Petr Hodina <phodina@protonmail.com>
Date: Sun, 30 Oct 2022 19:00:09 +0100
Subject: [PATCH] gnu: nix: Update to 2.11.1.

* gnu/packages/package-management.scm (nix): New variable.
* gnu/packages/patches/nix-dont-build-html-doc.diff: Update patch.

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9872d99312..a0c741df1c 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -21,6 +21,7 @@
 ;;; Copyright © 2021, 2022 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
 ;;; Copyright © 2022 Zhu Zihao <all_but_last@163.com>
+;;; Copyright © 2022 Petr Hodina <phodina@protonmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -56,6 +57,7 @@ (define-module (gnu packages package-management)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages cmake)
   #:use-module (gnu packages cpio)
+  #:use-module (gnu packages cpp)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
   #:use-module (gnu packages databases)
@@ -739,7 +741,7 @@ (define-public guix-modules
 (define-public nix
   (package
     (name "nix")
-    (version "2.5.1")
+    (version "2.11.1")
     (source
      (origin
        (method git-fetch)
@@ -748,7 +750,7 @@ (define-public nix
              (commit version)))
        (file-name (git-file-name name version))
        (sha256
-        (base32 "1m8rmv8i6lg83pmalvjlq1fn8mcghn3ngjv3kw1kqsa45ymj5sqq"))
+        (base32 "1sbl3fm3xcwcdf80xlfp743iw6pdx8rq8p272az03x1l9kk7l9d8"))
        (patches
         (search-patches "nix-dont-build-html-doc.diff"))))
     (build-system gnu-build-system)
@@ -757,6 +759,11 @@ (define-public nix
       #:configure-flags #~(list "--sysconfdir=/etc" "--enable-gc")
       #:phases
       #~(modify-phases %standard-phases
+          (add-after 'unpack 'fix-build
+		   (lambda* _
+		    ;; FIXME: Disable failing ASSERT
+		    (substitute* "src/libexpr/tests/primops.cc"
+			 ((".*/nix/store.*") ""))))
           (replace 'install
             ;; Don't try & fail to create subdirectories in /etc, but keep them
             ;; in the output as examples.
@@ -782,6 +789,7 @@ (define-public nix
                    bzip2
                    curl
                    editline
+                   json-modern-cxx
                    libarchive
                    libgc
                    libseccomp
diff --git a/gnu/packages/patches/nix-dont-build-html-doc.diff b/gnu/packages/patches/nix-dont-build-html-doc.diff
index 79142bc215..db110b80b8 100644
--- a/gnu/packages/patches/nix-dont-build-html-doc.diff
+++ b/gnu/packages/patches/nix-dont-build-html-doc.diff
@@ -1,26 +1,18 @@
-"--enable-doc-gen" configure flag will enable HTML document generation that requires mdbook.
-We can't simply disable it because we need manpages.
-
-Author: Zhu Zihao <all_but_last@163.com>
-
-diff --git a/doc/manual/local.mk b/doc/manual/local.mk
-index e43d9f2fb..c323d1847 100644
---- a/doc/manual/local.mk
-+++ b/doc/manual/local.mk
-@@ -69,8 +69,6 @@ $(d)/builtins.json: $(bindir)/nix
- 	$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(bindir)/nix __dump-builtins > $@.tmp
- 	@mv $@.tmp $@
+--- a/doc/manual/local.mk	2022-10-30 18:52:44.091425741 +0100
++++ b/doc/manual/local.mk	2022-10-30 18:53:00.026549841 +0100
+@@ -77,7 +77,6 @@
  
--# Generate the HTML manual.
+ # Generate the HTML manual.
+ html: $(docdir)/manual/index.html
 -install: $(docdir)/manual/index.html
  
  # Generate 'nix' manpages.
  install: $(mandir)/man1/nix3-manpages
-@@ -94,7 +92,5 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
+@@ -101,7 +100,5 @@
  	done
  	@touch $@
  
--$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md $(call rwildcard, $(d)/src, *.md)
+-$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md
 -	$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(DESTDIR)$(docdir)/manual
  
  endif
-- 
2.37.2

[bug#58903] Specify the build dir for Nix

[phodina via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 788 bytes --]

Hello,

I'd like to upstream this patch to Nix service and hear your thoughts on the matter.

The issue is that there are some packages that fail to build even on realtively powerful machine due to resource exhaustion - not enough RAM.

The solution for that is to extract and build the derivation on block device instead of tmpfs.

The solution for this is to specify the environment variable TMPDIR​ and point it e.g. to `/var/tmp​`.

Unfortunately I'm not aware of any other way how to specify I'd like to build just this package in other directory (other than `/tmp`).

Also is there some other way how Guix handles this issue? I might be also used in Nix.

FIY the reason I have to use Nix is to build the Android image [1]

1 https://github.com/danielfullmer/robotnix

----
Petr

[-- Attachment #1.2: Type: text/html, Size: 1147 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-services-nix-Add-more-configuration-fields.patch --]
[-- Type: text/x-patch; name=0001-services-nix-Add-more-configuration-fields.patch, Size: 3705 bytes --]

From 2d20cede80daf21ad355438a0d041b6dcba4a20b Mon Sep 17 00:00:00 2001
From: Petr Hodina <phodina@protonmail.com>
Date: Mon, 31 Oct 2022 16:12:38 +0100
Subject: [PATCH] services: nix: Add more configuration fields.

* gnu/services/nix.scm (<nix-configuration>)[build-dir]: New field.
(nix-service-etc, nix-shepherd-service): Take them into account.
* doc/guix.texi (Nix): Update it.

diff --git a/doc/guix.texi b/doc/guix.texi
index 80fb3bc47f..5eed0112d2 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -37111,6 +37111,26 @@ The Nix package to use.
 @item @code{sandbox} (default: @code{#t})
 Specifies whether builds are sandboxed by default.
 
+@item @code{build-dir} (default: @code{"/tmp"})
+Specifies build directory. This is useful to change if there is not enough RAM
+on the machine (e.g. embedded targets) or if building big packages (e.g.
+chromium).
+
+Normally the default location is @code{"/tmp"} which is mostly represented by
+@code{tmpfs}. Therefore this solution is fast and does not perform any IO to the
+nonvolatile block storage (unless swap is enabled). However, the user might
+encounter build failure when building large packages (e.g. @code{chromium})
+which take a lot of space when unpacking, the build artifacts consume more space
+and the linking stage might fail due to insufficient amount of RAM for LTO as
+all the object files are loaded into memory.
+
+This can be overcome by storing the intermediate files of the derivation in
+other locations such as @code{"/var/tmp/"} that are usually storaged on block
+device.
+
+Be aware that on embedded targets this option will result in more flash wear out
+due to large amount of writes.
+
 @item @code{build-sandbox-items} (default: @code{'()})
 This is a list of strings or objects appended to the
 @code{build-sandbox-items} field of the configuration file.
diff --git a/gnu/services/nix.scm b/gnu/services/nix.scm
index df04a85c22..ee49c0d594 100644
--- a/gnu/services/nix.scm
+++ b/gnu/services/nix.scm
@@ -54,6 +54,8 @@ (define-record-type* <nix-configuration>
                        (default nix))
   (sandbox             nix-configuration-sandbox ;boolean
                        (default #t))
+  (build-dir           nix-configuration-build-dir ;string
+                       (default "/tmp"))
   (build-sandbox-items nix-configuration-build-sandbox-items ;list of strings
                        (default '()))
   (extra-config        nix-configuration-extra-config ;list of strings
@@ -106,7 +108,7 @@ (define (nix-activation _)
 
 (define nix-service-etc
   (match-lambda
-    (($ <nix-configuration> package sandbox build-sandbox-items extra-config)
+    (($ <nix-configuration> package sandbox build-dir build-sandbox-items extra-config)
      (let ((ref-file (references-file package)))
        `(("nix/nix.conf"
           ,(computed-file
@@ -130,7 +132,7 @@ (define internal-sandbox-paths
 (define nix-shepherd-service
   ;; Return a <shepherd-service> for Nix.
   (match-lambda
-    (($ <nix-configuration> package _ _ _ extra-options)
+    (($ <nix-configuration> package _ build-dir _ _ extra-options)
      (list
       (shepherd-service
        (provision '(nix-daemon))
@@ -138,7 +140,10 @@ (define nix-shepherd-service
        (requirement '())
        (start #~(make-forkexec-constructor
                  (list (string-append #$package "/bin/nix-daemon")
-                       #$@extra-options)))
+                       #$@extra-options)
+                 #:environment-variables
+                   (list (string-append "TMPDIR=" build-dir)
+                         "PATH=/run/current-system/profile/bin")))
        (respawn? #f)
        (stop #~(make-kill-destructor)))))))
 
-- 
2.37.2

[bug#58903] Specify the build dir for Nix

[Tobias Geerinckx-Rice via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

Heyo,

phodina 写道：
> The issue is that there are some packages that fail to build 
> even on realtively powerful machine due to resource exhaustion - 
> not enough
> RAM.

Yes, this is an issue in Guix as well, e.g., when building disc 
images.

> Unfortunately I'm not aware of any other way how to specify I'd 
> like to build just this package in other directory (other than 
> `/tmp`).
>
> Also is there some other way how Guix handles this issue? I 
> might be also used in Nix.

No, Guix uses the exact same mechanism (and possibly code), 
inherited directly from Nix.

>  (build-dir           nix-configuration-build-dir ;string
>                       (default "/tmp"))

‘directory’.  Plz, no unnec. abbrev.

The concept LGTM.  Calling it a ‘build directory’ might be 
editorialising too much.  Does Nix use it for anything else?

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

[bug#58903] Specify the build dir for Nix

[phodina via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Hi Tobias,

thanks for feedback!

> 
> > (build-dir nix-configuration-build-dir ;string
> > (default "/tmp"))
> 
> 
> ‘directory’. Plz, no unnec. abbrev.
> 
> The concept LGTM. Calling it a ‘build directory’ might be editorialising too much. Does Nix use it for anything else?

I've updated the patch so there is no abbreviation. I asked on Nix matrix channel and was told just for derivations. But I'm not expert so take it with grain of salt.

Ludo',
you have a lot insight into the build infrastructure and you also took the concepts from Nix.

What do you think about this approach? It should not be premanently set to block device as you loose throughput and increase latency, but it's neccessary for some derivations as not everybody can have powerful machine at their disposal.

Is there some way to share the approach with guix daemon? If I remeber correctly the derivation is offloaded to nix backend - the contents of `nix` directory (haven't yet had the chance to go through the C code).

I guess the machines behind 'https://ci.guix.gnu.org' have enough resources. Or is there some script that handles large derivations? 

Also this might be interesting problem for embedded devices though offloading is definitely the best approach.

----
Petr

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-services-nix-Add-more-configuration-fields.patch --]
[-- Type: text/x-patch; name=v2-0001-services-nix-Add-more-configuration-fields.patch, Size: 3744 bytes --]

From d71d48d0fd5d4803abbb50900e0d9aa3c374e7e1 Mon Sep 17 00:00:00 2001
From: Petr Hodina <phodina@protonmail.com>
Date: Mon, 31 Oct 2022 16:12:38 +0100
Subject: [PATCH v2] services: nix: Add more configuration fields.

* gnu/services/nix.scm (<nix-configuration>)[build-directory]: New field.
(nix-service-etc, nix-shepherd-service): Take them into account.
* doc/guix.texi (Nix): Update it.

diff --git a/doc/guix.texi b/doc/guix.texi
index 80fb3bc47f..825cf0831d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -37111,6 +37111,26 @@ The Nix package to use.
 @item @code{sandbox} (default: @code{#t})
 Specifies whether builds are sandboxed by default.
 
+@item @code{build-directory} (default: @code{"/tmp"})
+Specifies build directory. This is useful to change if there is not enough RAM
+on the machine (e.g. embedded targets) or if building big packages (e.g.
+chromium).
+
+Normally the default location is @code{"/tmp"} which is mostly represented by
+@code{tmpfs}. Therefore this solution is fast and does not perform any IO to the
+nonvolatile block storage (unless swap is enabled). However, the user might
+encounter build failure when building large packages (e.g. @code{chromium})
+which take a lot of space when unpacking, the build artifacts consume more space
+and the linking stage might fail due to insufficient amount of RAM for LTO as
+all the object files are loaded into memory.
+
+This can be overcome by storing the intermediate files of the derivation in
+other locations such as @code{"/var/tmp/"} that are usually storaged on block
+device.
+
+Be aware that on embedded targets this option will result in more flash wear out
+due to large amount of writes.
+
 @item @code{build-sandbox-items} (default: @code{'()})
 This is a list of strings or objects appended to the
 @code{build-sandbox-items} field of the configuration file.
diff --git a/gnu/services/nix.scm b/gnu/services/nix.scm
index df04a85c22..dcf994b603 100644
--- a/gnu/services/nix.scm
+++ b/gnu/services/nix.scm
@@ -54,6 +54,8 @@ (define-record-type* <nix-configuration>
                        (default nix))
   (sandbox             nix-configuration-sandbox ;boolean
                        (default #t))
+  (build-directory     nix-configuration-build-directory ;string
+                       (default "/tmp"))
   (build-sandbox-items nix-configuration-build-sandbox-items ;list of strings
                        (default '()))
   (extra-config        nix-configuration-extra-config ;list of strings
@@ -106,7 +108,7 @@ (define (nix-activation _)
 
 (define nix-service-etc
   (match-lambda
-    (($ <nix-configuration> package sandbox build-sandbox-items extra-config)
+    (($ <nix-configuration> package sandbox build-directory build-sandbox-items extra-config)
      (let ((ref-file (references-file package)))
        `(("nix/nix.conf"
           ,(computed-file
@@ -130,7 +132,7 @@ (define internal-sandbox-paths
 (define nix-shepherd-service
   ;; Return a <shepherd-service> for Nix.
   (match-lambda
-    (($ <nix-configuration> package _ _ _ extra-options)
+    (($ <nix-configuration> package _ build-directory _ _ extra-options)
      (list
       (shepherd-service
        (provision '(nix-daemon))
@@ -138,7 +140,10 @@ (define nix-shepherd-service
        (requirement '())
        (start #~(make-forkexec-constructor
                  (list (string-append #$package "/bin/nix-daemon")
-                       #$@extra-options)))
+                       #$@extra-options)
+                 #:environment-variables
+                   (list (string-append "TMPDIR=" build-directory)
+                         "PATH=/run/current-system/profile/bin")))
        (respawn? #f)
        (stop #~(make-kill-destructor)))))))
 
-- 
2.37.2

[bug#58903] [PATCH] gnu: nix: Update to 2.11.1.

[Ludovic Courtès]

Hi,

phodina <phodina@protonmail.com> skribis:

> I've updated the patch so there is no abbreviation. I asked on Nix matrix channel and was told just for derivations. But I'm not expert so take it with grain of salt.
>
> Ludo',
> you have a lot insight into the build infrastructure and you also took the concepts from Nix.

The manual documents it in the context of guix-daemon (info "(guix)
Build Environment Setup"):

     You can influence the directory where the daemon stores build trees
  via the ‘TMPDIR’ environment variable.  However, the build tree within
  the chroot is always called ‘/tmp/guix-build-NAME.drv-0’, where NAME is
  the derivation name—e.g., ‘coreutils-8.24’.  This way, the value of
  ‘TMPDIR’ does not leak inside build environments, which avoids
  discrepancies in cases where build processes capture the name of their
  build tree.

> I guess the machines behind 'https://ci.guix.gnu.org' have enough resources. Or is there some script that handles large derivations? 

On Guix System, /tmp is not on tmpfs by default (maybe something to
change?); that’s also the case on build machines.

> From d71d48d0fd5d4803abbb50900e0d9aa3c374e7e1 Mon Sep 17 00:00:00 2001
> From: Petr Hodina <phodina@protonmail.com>
> Date: Mon, 31 Oct 2022 16:12:38 +0100
> Subject: [PATCH v2] services: nix: Add more configuration fields.
>
> * gnu/services/nix.scm (<nix-configuration>)[build-directory]: New field.
> (nix-service-etc, nix-shepherd-service): Take them into account.
> * doc/guix.texi (Nix): Update it.

[...]

> +@item @code{build-directory} (default: @code{"/tmp"})
> +Specifies build directory.

Rather something like: “The directory where build directory are stored
during builds.”

> This is useful to change if there is not enough RAM
> +on the machine (e.g. embedded targets) or if building big packages (e.g.
> +chromium).

“This is useful to change if, for example, the default location does not
have enough space to hold build trees for big packages.”

(There’s no ‘chromium’ package in Guix.)

Then, instead of a long explanation, I’d just write something like:

  This is similar to setting the @env{TMPDIR} environment variable for
  @command{guix-daemon}.  @ref{Build Environment Setup, @env{TMPDIR}},
  for more info.

Could you send an updated patch?

Thanks,
Ludo’.

[bug#58903] [PATCH] gnu: nix: Update to 2.11.1.

[phodina via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 254 bytes --]

Hello Ludo’,

here's the updated patch with the text as you suggested. I removed the previous long paragraph which was to detailed and would only lengthen the Guix documentation :)

Also good idea to reference the guix daemon. Thanks.

----
Petr

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-services-nix-Add-more-configuration-fields.patch --]
[-- Type: text/x-patch; name=v3-0001-services-nix-Add-more-configuration-fields.patch, Size: 3122 bytes --]

From 5b38fc4945224df40ceca3736ee40becbda51853 Mon Sep 17 00:00:00 2001
From: Petr Hodina <phodina@protonmail.com>
Date: Mon, 31 Oct 2022 16:12:38 +0100
Subject: [PATCH v3] services: nix: Add more configuration fields.

* gnu/services/nix.scm (<nix-configuration>)[build-directory]: New field.
(nix-service-etc, nix-shepherd-service): Take them into account.
* doc/guix.texi (Nix): Update it.

diff --git a/doc/guix.texi b/doc/guix.texi
index 50487a5172..69ac5de066 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -37209,6 +37209,15 @@ The Nix package to use.
 @item @code{sandbox} (default: @code{#t})
 Specifies whether builds are sandboxed by default.
 
+@item @code{build-directory} (default: @code{"/tmp"})
+The directory where build directory are stored during builds.
+This is useful to change if, for example, the default location does not
+have enough space to hold build trees for big packages.
+
+This is similar to setting the @env{TMPDIR} environment variable for
+@command{guix-daemon}.  @ref{Build Environment Setup, @env{TMPDIR}},
+for more info.
+
 @item @code{build-sandbox-items} (default: @code{'()})
 This is a list of strings or objects appended to the
 @code{build-sandbox-items} field of the configuration file.
diff --git a/gnu/services/nix.scm b/gnu/services/nix.scm
index df04a85c22..dcf994b603 100644
--- a/gnu/services/nix.scm
+++ b/gnu/services/nix.scm
@@ -54,6 +54,8 @@ (define-record-type* <nix-configuration>
                        (default nix))
   (sandbox             nix-configuration-sandbox ;boolean
                        (default #t))
+  (build-directory     nix-configuration-build-directory ;string
+                       (default "/tmp"))
   (build-sandbox-items nix-configuration-build-sandbox-items ;list of strings
                        (default '()))
   (extra-config        nix-configuration-extra-config ;list of strings
@@ -106,7 +108,7 @@ (define (nix-activation _)
 
 (define nix-service-etc
   (match-lambda
-    (($ <nix-configuration> package sandbox build-sandbox-items extra-config)
+    (($ <nix-configuration> package sandbox build-directory build-sandbox-items extra-config)
      (let ((ref-file (references-file package)))
        `(("nix/nix.conf"
           ,(computed-file
@@ -130,7 +132,7 @@ (define internal-sandbox-paths
 (define nix-shepherd-service
   ;; Return a <shepherd-service> for Nix.
   (match-lambda
-    (($ <nix-configuration> package _ _ _ extra-options)
+    (($ <nix-configuration> package _ build-directory _ _ extra-options)
      (list
       (shepherd-service
        (provision '(nix-daemon))
@@ -138,7 +140,10 @@ (define nix-shepherd-service
        (requirement '())
        (start #~(make-forkexec-constructor
                  (list (string-append #$package "/bin/nix-daemon")
-                       #$@extra-options)))
+                       #$@extra-options)
+                 #:environment-variables
+                   (list (string-append "TMPDIR=" build-directory)
+                         "PATH=/run/current-system/profile/bin")))
        (respawn? #f)
        (stop #~(make-kill-destructor)))))))
 
-- 
2.38.1

bug#58903: [PATCH] gnu: nix: Update to 2.11.1.

[Ludovic Courtès]

Hi,

phodina <phodina@protonmail.com> skribis:

> From 5b38fc4945224df40ceca3736ee40becbda51853 Mon Sep 17 00:00:00 2001
> From: Petr Hodina <phodina@protonmail.com>
> Date: Mon, 31 Oct 2022 16:12:38 +0100
> Subject: [PATCH v3] services: nix: Add more configuration fields.
>
> * gnu/services/nix.scm (<nix-configuration>)[build-directory]: New field.
> (nix-service-etc, nix-shepherd-service): Take them into account.
> * doc/guix.texi (Nix): Update it.

Applied, thanks!

Ludo’.