bug#74217: Bootstrapping Zig with no Binary Blobs

bug#74217: Bootstrapping Zig with no Binary Blobs

[Ekaitz Zarraga]

Hi,

In order to include modern versions of Zig (Zig 0.12+) in Guix, we need 
to remove the binary blobs.

I open this issue to track this effort and store information about the 
process.

Some Guix user is trying to achieve the same goal:

https://git.jakstys.lt/motiejus/zig-repro

And discussing about it here:

https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11

We could use that effort as a reference and package it to Guix.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

Hi Ekaitz,

On Wed, 06 Nov 2024 05:47:48 +0800,
Ekaitz Zarraga wrote:
>
> Hi,
>
> In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> remove the binary blobs.
>
> I open this issue to track this effort and store information about the process.
>
> Some Guix user is trying to achieve the same goal:
>
> https://git.jakstys.lt/motiejus/zig-repro
>
> And discussing about it here:
>
> https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11


Great news, thanks for sharing!


> We could use that effort as a reference and package it to Guix.


Excited to know Zig developers are willing to help with bootstrapping!  We
shouldn't miss this chance :)

I'll look into replicating the current progress this weekend.  Anyone reading
this mail wants to join in?


Thanks

bug#74217: Bootstrapping Zig with no Binary Blobs

[Noé Lopez via Bug reports for GNU Guix]

>Excited to know Zig developers are willing to help with bootstrapping!  We
>shouldn't miss this chance :)
>
>I'll look into replicating the current progress this weekend.  Anyone reading
>this mail wants to join in?

Hey Hilton and Ekaitz,

I’m interested in this :) From what I can see the current effort in the
zig-repro repository is very well made and we should just need to
replicate each step with the correct dependencies, hoping they exist 🤞

I can try to make the first few steps as packages tomorrow, WDYT?

Good evening,
Noé

bug#74217: Bootstrapping Zig with no Binary Blobs

[Ekaitz Zarraga]

Hi

> I can try to make the first few steps as packages tomorrow, WDYT?

Sure!
Send the patches to this issue and I'll review when I have time.

Thanks for the energy!

bug#74217: [PATCH 0/2] Initial step on bootstrapping Zig.

[Hilton Chain via Bug reports for GNU Guix]

Finished step 00 from Motiejus Jakštys's script[1], this should make later
work on Guix side easier.

[1]: https://git.jakstys.lt/motiejus/zig-repro/src/branch/main/run

Hilton Chain (2):
  gnu: Add zig-0.10.0-610-bootstrap.
  gnu: Add zig-0.10.0-610.

 gnu/local.mk                                  |   1 +
 ...10.0-610-bootstrap-resolve-conflicts.patch |  87 ++++++++++
 gnu/packages/zig.scm                          | 151 +++++++++++++++++-
 3 files changed, 238 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch


base-commit: 2a6d96425eea57dc6dd48a2bec16743046e32e06
--
2.46.0

bug#74217: [PATCH 1/2] gnu: Add zig-0.10.0-610-bootstrap.

[Hilton Chain via Bug reports for GNU Guix]

* gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch: New
file.
* gnu/local.mk (dist_patch_DATA): Regisiter it.
* gnu/packages/zig.scm (zig-0.10.0-538-source,zig-0.10.0-539-patch)
(zig-0.10.0-542-patch,zig-0.10.0-610-bootstrap): New variables.

Change-Id: I132bbad34f40b919b4573e02d0f40eb4a007a26c
---
 gnu/local.mk                                  |   1 +
 ...10.0-610-bootstrap-resolve-conflicts.patch |  87 +++++++++++++++
 gnu/packages/zig.scm                          | 105 +++++++++++++++++-
 3 files changed, 192 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index ae902a5ab2..2cc2ea4c81 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -2358,6 +2358,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/xygrib-newer-proj.patch			\
   %D%/packages/patches/yggdrasil-extra-config.patch	\
   %D%/packages/patches/zig-0.9-riscv-support.patch		\
+  %D%/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch	\
   %D%/packages/patches/zig-use-baseline-cpu-by-default.patch	\
   %D%/packages/patches/zig-use-system-paths.patch		\
   %D%/packages/patches/zsh-egrep-failing-test.patch		\
diff --git a/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
new file mode 100644
index 0000000000..5ad5ffc249
--- /dev/null
+++ b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
@@ -0,0 +1,87 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 1c03faf1e9..89406eb1b2 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -846,16 +846,17 @@ else()
+ endif()
+ 
+ set(ZIG_BUILD_ARGS
+-  --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
+-  "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
+-  "-Denable-llvm"
+-  ${ZIG_RELEASE_ARG}
+-  ${ZIG_STATIC_ARG}
+-  ${ZIG_NO_LIB_ARG}
+-  ${ZIG_SINGLE_THREADED_ARG}
+-  "-Dtarget=${ZIG_TARGET_TRIPLE}"
+-  "-Dcpu=${ZIG_TARGET_MCPU}"
+-  "-Dversion-string=${RESOLVED_ZIG_VERSION}"
++    --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
++    "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
++    "-Denable-llvm"
++    "-Denable-stage1"
++    ${ZIG_RELEASE_ARG}
++    ${ZIG_STATIC_ARG}
++    ${ZIG_NO_LIB_ARG}
++    ${ZIG_SINGLE_THREADED_ARG}
++    "-Dtarget=${ZIG_TARGET_TRIPLE}"
++    "-Dcpu=${ZIG_TARGET_MCPU}"
++    "-Dversion-string=${RESOLVED_ZIG_VERSION}"
+ )
+ 
+ add_custom_target(stage3 ALL
+diff --git a/build.zig b/build.zig
+index cf0e092326..7f80c3e1df 100644
+--- a/build.zig
++++ b/build.zig
+@@ -142,7 +142,8 @@ pub fn build(b: *Builder) !void {
+     const force_gpa = b.option(bool, "force-gpa", "Force the compiler to use GeneralPurposeAllocator") orelse false;
+     const link_libc = b.option(bool, "force-link-libc", "Force self-hosted compiler to link libc") orelse (enable_llvm or only_c);
+     const sanitize_thread = b.option(bool, "sanitize-thread", "Enable thread-sanitization") orelse false;
+-    const strip = b.option(bool, "strip", "Omit debug information");
++    const strip = b.option(bool, "strip", "Omit debug information") orelse false;
++    const use_zig0 = b.option(bool, "zig0", "Bootstrap using zig0") orelse false;
+     const value_tracing = b.option(bool, "value-tracing", "Enable extra state tracking to help troubleshoot bugs in the compiler (using the std.debug.Trace API)") orelse false;
+ 
+     const mem_leak_frames: u32 = b.option(u32, "mem-leak-frames", "How many stack frames to print when a memory leak occurs. Tests get 2x this amount.") orelse blk: {
+@@ -151,7 +152,22 @@ pub fn build(b: *Builder) !void {
+         break :blk 4;
+     };
+ 
+-    const exe = addCompilerStep(b);
++    if (only_c) {
++        target.ofmt = .c;
++    }
++
++    const main_file: ?[]const u8 = mf: {
++        if (!have_stage1) break :mf "src/main.zig";
++        if (use_zig0) break :mf null;
++        break :mf "src/stage1.zig";
++    };
++
++    const exe = b.addExecutable("zig", main_file);
++
++    const compile_step = b.step("compile", "Build the self-hosted compiler");
++    compile_step.dependOn(&exe.step);
++
++    exe.stack_size = stack_size;
+     exe.strip = strip;
+     exe.sanitize_thread = sanitize_thread;
+     exe.build_id = b.option(bool, "build-id", "Include a build id note") orelse false;
+diff --git a/src/translate_c/ast.zig b/src/translate_c/ast.zig
+index 20e4259725..bc0f002c21 100644
+--- a/src/translate_c/ast.zig
++++ b/src/translate_c/ast.zig
+@@ -1448,6 +1448,12 @@ fn renderNode(c: *Context, node: Node) Allocator.Error!NodeIndex {
+         .optional_type => return renderPrefixOp(c, node, .optional_type, .question_mark, "?"),
+         .address_of => {
+             const payload = node.castTag(.address_of).?.data;
++            if (c.zig_is_stage1 and payload.tag() == .fn_identifier)
++                return try c.addNode(.{
++                    .tag = .identifier,
++                    .main_token = try c.addIdentifier(payload.castTag(.fn_identifier).?.data),
++                    .data = undefined,
++                });
+ 
+             const ampersand = try c.addToken(.ampersand, "&");
+             const base = if (payload.tag() == .fn_identifier)
diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 6994d48818..68907fd04e 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -23,13 +23,15 @@ (define-module (gnu packages zig)
   #:use-module (guix gexp)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cmake)
   #:use-module (gnu packages)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages llvm)
-  #:use-module (gnu packages llvm-meta))
+  #:use-module (gnu packages llvm-meta)
+  #:use-module (gnu packages web))
 
 (define-public zig-0.9
   (package
@@ -196,4 +198,105 @@ (define-public zig-0.10
     (properties `((max-silent-time . 9600)
                   ,@(clang-compiler-cpu-architectures "15")))))
 
+(define zig-0.10.0-538-source
+  ;; "std: added eql to DynamicBitSet and DynamicBitSetUnmanaged"
+  (let ((commit "bf316e550671cc71eb498b3cf799493627bb0fdc")
+        (revision "538"))
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/ziglang/zig")
+            (commit commit)))
+      (file-name (git-file-name "zig" (git-version "0.10" revision commit)))
+      (sha256
+       (base32 "1dchc2bp842jlw0byssqzindv8cigpqcj2hk3752667jrrww13vv")))))
+
+(define zig-0.10.0-539-patch
+  ;; "remove `-fstage1` option"
+  (let ((commit "28514476ef8c824c3d189d98f23d0f8d23e496ea"))
+    (origin
+      (method url-fetch)
+      (uri (string-append
+            "https://github.com/ziglang/zig/commit/" commit ".patch"))
+      (sha256
+       (base32 "0qxxiafg2sd5rr4xhw0c12rygd7zh1rmf3x8hfialyxmsbi5pfxp")))))
+
+(define zig-0.10.0-542-patch
+  ;; "actually remove stage1"
+  (let ((commit "3ba916584db5485c38ebf2390e8d22bc6d81bf8e"))
+    (origin
+      (method url-fetch)
+      (uri (string-append
+            "https://github.com/ziglang/zig/commit/" commit ".patch"))
+      (sha256
+       (base32 "1l09gmbr3vqzinb63kvaskgs1d0mvm1m7w3ai3ngwg5zlabyya35")))))
+
+;; Build zig1.wasm from source.
+(define zig-0.10.0-610-bootstrap
+  (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+        (revision "610"))
+    (package
+      (inherit zig-0.10)
+      (name "zig")
+      (version (git-version "0.10.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/ziglang/zig")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (modules '((guix build utils)))
+                (snippet '(delete-file "stage1/zig1.wasm.zst"))
+                (sha256
+                 (base32
+                  "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+      (arguments
+       (substitute-keyword-arguments (package-arguments zig-0.10)
+         ((#:phases phases '%standard-phases)
+          #~(modify-phases #$phases
+              (add-after 'unpack 'prepare-source
+                (lambda* (#:key native-inputs inputs #:allow-other-keys)
+                  (copy-recursively "." "../source-backup")
+                  ;; Revert "actually remove stage1".
+                  (invoke "patch" "--reverse" "--strip=1"
+                          "--input" #+zig-0.10.0-542-patch)
+                  ;; Revert "remove `-fstage1` option".
+                  (false-if-exception
+                   (invoke "patch" "--reverse" "--strip=1"
+                           "--input" #+zig-0.10.0-539-patch))
+                  ;; Resolve conflicts in previous patching.
+                  (invoke
+                   "patch" "--forward" "--strip=1" "--input"
+                   #+(local-file
+                      (search-patch
+                       "zig-0.10.0-610-bootstrap-resolve-conflicts.patch")))
+                  ;; Restore build system.
+                  (rename-file "stage1/config.zig.in" "src/config.zig.in")
+                  (substitute* "src/config.zig.in"
+                    (("(have_stage1 = )false" _ prefix)
+                     (string-append prefix "true")))
+                  (for-each
+                   (lambda (file)
+                     (copy-file (string-append #+zig-0.10.0-538-source "/" file)
+                                file))
+                   '("build.zig" "CMakeLists.txt"))))
+              (add-after 'install 'build-zig1
+                (lambda _
+                  (copy-recursively "../source-backup" ".")
+                  (invoke (string-append #$output "/bin/zig")
+                          "build" "update-zig1" "--verbose")))
+              (add-after 'build-zig1 'install-zig1
+                (lambda _
+                  (install-file "stage1/zig1.wasm.zst"
+                                (string-append #$output:zig1 "/bin"))))
+              (replace 'check
+                (lambda* (#:key tests? #:allow-other-keys)
+                  (when tests?
+                    (invoke (string-append #$output "/bin/zig")
+                            "test" "-I" "test" "test/behavior.zig"))))))))
+      (native-inputs
+       (modify-inputs (package-native-inputs zig-0.10)
+         (prepend binaryen)))
+      (outputs '("out" "zig1")))))
+
 (define-public zig zig-0.10)
-- 
2.46.0

bug#74217: [PATCH 2/2] gnu: Add zig-0.10.0-610.

[Hilton Chain via Bug reports for GNU Guix]

*  gnu/packages/zig.scm (zig-0.10.0-610): New variable.

Change-Id: I277a7f5e9781e89d7ad7cd108fec9afcf8cd23d9
---
 gnu/packages/zig.scm | 46 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 68907fd04e..4174dba38b 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -299,4 +299,50 @@ (define zig-0.10.0-610-bootstrap
          (prepend binaryen)))
       (outputs '("out" "zig1")))))
 
+;; Bootstrap with our zig1.wasm.
+(define zig-0.10.0-610
+  (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+        (revision "610"))
+    (package
+      (inherit zig-0.10)
+      (name "zig")
+      (version (git-version "0.10.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/ziglang/zig")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (modules '((guix build utils)))
+                (snippet '(delete-file "stage1/zig1.wasm.zst"))
+                (sha256
+                 (base32
+                  "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+      (arguments
+       (substitute-keyword-arguments (package-arguments zig-0.10)
+         ((#:phases phases '%standard-phases)
+          #~(modify-phases #$phases
+              (add-after 'unpack 'unpack-zig1
+                (lambda* (#:key native-inputs inputs #:allow-other-keys)
+                  (install-file (search-input-file
+                                 (or native-inputs inputs) "bin/zig1.wasm.zst")
+                                "stage1")))
+              (add-after 'install 'update-zig1
+                (lambda _
+                  (invoke (string-append #$output "/bin/zig")
+                          "build" "update-zig1" "--verbose")))
+              (add-after 'update-zig1 'install-zig1
+                (lambda _
+                  (install-file "stage1/zig1.wasm.zst"
+                                (string-append #$output:zig1 "/bin"))))
+              (replace 'check
+                (lambda* (#:key tests? #:allow-other-keys)
+                  (when tests?
+                    (invoke (string-append #$output "/bin/zig")
+                            "test" "-I" "test" "test/behavior.zig"))))))))
+      (native-inputs
+       (modify-inputs (package-native-inputs zig-0.10)
+         (prepend binaryen `(,zig-0.10.0-610-bootstrap "zig1"))))
+      (outputs '("out" "zig1")))))
+
 (define-public zig zig-0.10)
-- 
2.46.0

bug#74217: [PATCH 0/2] Initial step on bootstrapping Zig.

[Hilton Chain via Bug reports for GNU Guix]

I have added a wip-zig-bootstrap[1] branch.

Variables are not exported, you can build them locally with the following
command:
--8<---------------cut here---------------start------------->8---
ZIG=$(
	sed 's/^\(define(-public)? (zig-.*-.*)/\2/p' gnu/packages/zig.scm \
		--quiet --regexp-extended |
		tail --lines=1
)

./pre-inst-env guix build --expression="(@@ (gnu packages zig) $ZIG)"
--8<---------------cut here---------------end--------------->8---

Thanks
---
[1]: https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-zig-bootstrap

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

On Thu, Nov 07, 2024 at 09:19:23AM +0800, Hilton Chain wrote:
> Hi Ekaitz,
> 
> On Wed, 06 Nov 2024 05:47:48 +0800,
> Ekaitz Zarraga wrote:
> >
> > Hi,
> >
> > In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> > remove the binary blobs.
> >
> > I open this issue to track this effort and store information about the process.
> >
> > Some Guix user is trying to achieve the same goal:
> >
> > https://git.jakstys.lt/motiejus/zig-repro
> >
> > And discussing about it here:
> >
> > https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11
> 
> 
> Great news, thanks for sharing!
> 
> 
> > We could use that effort as a reference and package it to Guix.
> 
> 
> Excited to know Zig developers are willing to help with bootstrapping!  We
> shouldn't miss this chance :)
> 
> I'll look into replicating the current progress this weekend.  Anyone reading
> this mail wants to join in?

Ok, you've convinced me.  I guess I finally need to get zig@0.10
building correctly on riscv64.

Expect plenty of FIXUP and SQUASH commits in the future.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

Hi Efraim,

On Mon, 11 Nov 2024 19:42:25 +0800,
Efraim Flashner wrote:
>
> Ok, you've convinced me.  I guess I finally need to get zig@0.10
> building correctly on riscv64.
>
> Expect plenty of FIXUP and SQUASH commits in the future.

I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
when the branch is ready to be worked on :)

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Mon, Nov 11, 2024 at 07:56:30PM +0800, Hilton Chain wrote:
> Hi Efraim,
> 
> On Mon, 11 Nov 2024 19:42:25 +0800,
> Efraim Flashner wrote:
> >
> > Ok, you've convinced me.  I guess I finally need to get zig@0.10
> > building correctly on riscv64.
> >
> > Expect plenty of FIXUP and SQUASH commits in the future.
> 
> I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
> when the branch is ready to be worked on :)

No worries. I'm still getting a look around the branch. And if you
force-push then my regular push will be rejected and I can rebase
whatever I have.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

Hello everyone,

With a new forced push (and new rebuilds, sorry), wip-zig-bootstrap is mostly
ready (for me) now.  Please take a look at the first few commits, as I'm
changing Zig's behavior there, here are some additional notes:

  To Efraim: Can adding pkg-config to native-inputs avoid the ncdu snippet?

I'm building this branch on my personal Cuirass instance[1][2], for x86_64-linux
and aarch64-linux (qemu-binfmt), in previous revisions I have indentified
reproduciblity issue, and aarch64 builds timed-out.  I haven't investigated them
yet.

Then for what's the RUNPATH issue I have mentioned in commits:
+ For current zig@0.10 on Guix master: glibc is missing from RUNPATH, which
fails the validate-runpath check.
+ For zig@0.11, some other inputs are missing, making the binary failing to run on Guix[3].
+ dan also mentioned privately to me that they needed to add paths from
LIBRARY_PATH to RUNPATH for their own projects, so programs built by Zig is also
affected.

I think this is due to Zig's implemention of its own linking logic, which
bypasses our ld-wrapper.

I'm not going to implement ld-wrapper within Zig. :) So my proposed workaround
in wip-zig-bootstrap is to patch the handling logic added for Guix:

(In lib/std/zig/system/NativePaths.zig)
--8<---------------cut here---------------start------------->8---
         // Distros like guix don't use FHS, so they rely on environment
         // variables to search for headers and libraries.
         // We use os.getenv here since this part won't be executed on
         // windows, to get rid of unnecessary error handling.
-        if (std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
+        if (std.posix.getenv("CROSS_C_INCLUDE_PATH") orelse std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
             var it = mem.tokenizeScalar(u8, c_include_path, ':');
             while (it.next()) |dir| {
                 try self.addIncludeDir(dir);
             }
         }

-        if (std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
+        if (std.posix.getenv("CROSS_CPLUS_INCLUDE_PATH") orelse std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
             var it = mem.tokenizeScalar(u8, cplus_include_path, ':');
             while (it.next()) |dir| {
                 try self.addIncludeDir(dir);
             }
         }

-        if (std.posix.getenv("LIBRARY_PATH")) |library_path| {
+        if (std.posix.getenv("CROSS_LIBRARY_PATH") orelse std.posix.getenv("LIBRARY_PATH")) |library_path| {
             var it = mem.tokenizeScalar(u8, library_path, ':');
             while (it.next()) |dir| {
                 try self.addLibDir(dir);
+                try self.addRPath(dir);
             }
         }
     }
--8<---------------cut here---------------end--------------->8---

Adding directories from CROSS_LIBRARY_PATH or LIBRARY_PATH to RUNPATH, "CROSS_"
part is for our cross toolchain, I haven't tested it yet.

I think this behavior change is reasonable since the search path
(CROSS_)?LIBRARY_PATH is only automatically set by our compilers.

I added this change to 0.9 as well to make all Zigs behave consistently.  I also
used shrink-runpath phase from meson-build-system in Zig and zig-build-system.

I want to move shrink-runpath to (guix build utils) and export it too, so that
it can be used easier.  But I'm not sure if this change will trigger rebuilds of
other packages, so I didn't do it.

Thanks to Guile, for builds not managed by guix-daemon, something like the
following script can be used, we can ship a program-file if we agree on this
workaround.
--8<---------------cut here---------------start------------->8---
(use-modules (guix build meson-build-system))

(define shrink-runpath
  (assoc-ref %standard-phases 'shrink-runpath))

(define (main directories)
  (for-each (lambda (dir)
              (false-if-exception
               (shrink-runpath
                #:elf-directories '(".")
                #:outputs `(("out" . ,dir)))))
            directories))

(main (cdr (command-line)))
--8<---------------cut here---------------end--------------->8---
Usage: guile <file-with-above-content> DIRECTORY...


Thanks
---
[1]: https://ci.boiledscript.com/jobset/guix-zig
[2]: https://substitute.boiledscript.com, if you want to challenge it.
[3]: https://github.com/ziglang/zig/issues/18434

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 6399 bytes --]

On Thu, Nov 14, 2024 at 12:46:59AM +0800, Hilton Chain wrote:
> Hello everyone,
> 
> With a new forced push (and new rebuilds, sorry), wip-zig-bootstrap is mostly
> ready (for me) now.  Please take a look at the first few commits, as I'm
> changing Zig's behavior there, here are some additional notes:
> 
>   To Efraim: Can adding pkg-config to native-inputs avoid the ncdu snippet?

Unfortunately no. The code reads:
    exe.root_module.linkSystemLibrary("ncursesw", .{});
    exe.root_module.linkSystemLibrary("libzstd", .{});

and so it searches (something like the following, I lost the log):

/usr/lib/liblibzstd.so
/usr/lib/liblibzstd.a
...
/gnu/store/...-glibc.../lib/liblibzstd.so
/gnu/store/...-glibc.../lib/liblibzstd.a
...
/gnu/store/...-zstd.../lib/liblibzstd.so
/gnu/store/...-zstd.../lib/liblibzstd.a

so it looks like it automatically adds the 'lib' at the front. When I've
built out to ncdu again I'll check it again.

> I'm building this branch on my personal Cuirass instance[1][2], for x86_64-linux
> and aarch64-linux (qemu-binfmt), in previous revisions I have indentified
> reproduciblity issue, and aarch64 builds timed-out.  I haven't investigated them
> yet.

I've also been having berlin build zig on the wip-zig-bootstrap branch
for x86_64. Unfortunately there's quite a bit to go on aarch64 for it to
get there, but I can confirm from my machine that it's working.

> Then for what's the RUNPATH issue I have mentioned in commits:
> + For current zig@0.10 on Guix master: glibc is missing from RUNPATH, which
> fails the validate-runpath check.
> + For zig@0.11, some other inputs are missing, making the binary failing to run on Guix[3].
> + dan also mentioned privately to me that they needed to add paths from
> LIBRARY_PATH to RUNPATH for their own projects, so programs built by Zig is also
> affected.

I didn't test it. ncdu@2.3 builds with zig-0.11, so that's an option for
testing it out.

https://dev.yorhel.nl/ncdu/changes2

> I think this is due to Zig's implemention of its own linking logic, which
> bypasses our ld-wrapper.

I wonder if switching from lld to make-lld-wrapper would make a
difference here.

> I'm not going to implement ld-wrapper within Zig. :) So my proposed workaround
> in wip-zig-bootstrap is to patch the handling logic added for Guix:
> 
> (In lib/std/zig/system/NativePaths.zig)
> --8<---------------cut here---------------start------------->8---
>          // Distros like guix don't use FHS, so they rely on environment
>          // variables to search for headers and libraries.
>          // We use os.getenv here since this part won't be executed on
>          // windows, to get rid of unnecessary error handling.
> -        if (std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
> +        if (std.posix.getenv("CROSS_C_INCLUDE_PATH") orelse std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
>              var it = mem.tokenizeScalar(u8, c_include_path, ':');
>              while (it.next()) |dir| {
>                  try self.addIncludeDir(dir);
>              }
>          }
> 
> -        if (std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
> +        if (std.posix.getenv("CROSS_CPLUS_INCLUDE_PATH") orelse std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
>              var it = mem.tokenizeScalar(u8, cplus_include_path, ':');
>              while (it.next()) |dir| {
>                  try self.addIncludeDir(dir);
>              }
>          }
> 
> -        if (std.posix.getenv("LIBRARY_PATH")) |library_path| {
> +        if (std.posix.getenv("CROSS_LIBRARY_PATH") orelse std.posix.getenv("LIBRARY_PATH")) |library_path| {
>              var it = mem.tokenizeScalar(u8, library_path, ':');
>              while (it.next()) |dir| {
>                  try self.addLibDir(dir);
> +                try self.addRPath(dir);
>              }
>          }
>      }
> --8<---------------cut here---------------end--------------->8---
> 
> Adding directories from CROSS_LIBRARY_PATH or LIBRARY_PATH to RUNPATH, "CROSS_"
> part is for our cross toolchain, I haven't tested it yet.

I like this, and it seems like it should make it work for cross
compiling zig programs. That's part of why I added the updated ncdu
commit, to use it for testing.

> I think this behavior change is reasonable since the search path
> (CROSS_)?LIBRARY_PATH is only automatically set by our compilers.
> 
> I added this change to 0.9 as well to make all Zigs behave consistently.  I also
> used shrink-runpath phase from meson-build-system in Zig and zig-build-system.
> 
> I want to move shrink-runpath to (guix build utils) and export it too, so that
> it can be used easier.  But I'm not sure if this change will trigger rebuilds of
> other packages, so I didn't do it.

I found that there was still the full LIBRARY_PATH embedded in the ncdu
binary as a string. So with that I'm not sure about using RPath instead
of LibDir for the LIBRARY_PATH.

> Thanks to Guile, for builds not managed by guix-daemon, something like the
> following script can be used, we can ship a program-file if we agree on this
> workaround.
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix build meson-build-system))
> 
> (define shrink-runpath
>   (assoc-ref %standard-phases 'shrink-runpath))
> 
> (define (main directories)
>   (for-each (lambda (dir)
>               (false-if-exception
>                (shrink-runpath
>                 #:elf-directories '(".")
>                 #:outputs `(("out" . ,dir)))))
>             directories))
> 
> (main (cdr (command-line)))
> --8<---------------cut here---------------end--------------->8---
> Usage: guile <file-with-above-content> DIRECTORY...
> 
> 
> Thanks
> ---
> [1]: https://ci.boiledscript.com/jobset/guix-zig
> [2]: https://substitute.boiledscript.com, if you want to challenge it.
> [3]: https://github.com/ziglang/zig/issues/18434

I spent a bunch of time trying to get zig-0.10 or 0.10.0-610 to build on
riscv64 and ppc64le but haven't been able to crack it yet.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 14 Nov 2024 02:10:44 +0800,
Efraim Flashner wrote:
>
> > I think this is due to Zig's implemention of its own linking logic, which
> > bypasses our ld-wrapper.
>
> I wonder if switching from lld to make-lld-wrapper would make a
> difference here.

Thanks for mentioning!  This should do the work!

I thought Zig was using lld as a library for linking.  Just looked at
src/link/Elf.zig, this is not true, it invokes ld.lld.

I have more time today.  I'll test this out and see if lld-as-ld-wrapper can
also add glibc RUNPATH.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 14 Nov 2024 07:40:05 +0800,
Hilton Chain wrote:
>
> On Thu, 14 Nov 2024 02:10:44 +0800,
> Efraim Flashner wrote:
> >
> > > I think this is due to Zig's implemention of its own linking logic, which
> > > bypasses our ld-wrapper.
> >
> > I wonder if switching from lld to make-lld-wrapper would make a
> > difference here.
>
> Thanks for mentioning!  This should do the work!
>
> I thought Zig was using lld as a library for linking.  Just looked at
> src/link/Elf.zig, this is not true, it invokes ld.lld.
>
> I have more time today.  I'll test this out and see if lld-as-ld-wrapper can
> also add glibc RUNPATH.

Zig manages linking to libc seperately, if this approach works, we have to
keep validate-runpath? off for Zig.  Reference to ld.lld can't be patched
since it's both native-inputs (for building Zig) and inputs (for using the
built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 14 Nov 2024 09:05:41 +0800,
Hilton Chain wrote:
>
> On Thu, 14 Nov 2024 07:40:05 +0800,
> Hilton Chain wrote:
> >
> > I thought Zig was using lld as a library for linking.  Just looked at
> > src/link/Elf.zig, this is not true, it invokes ld.lld.
> >
> > I have more time today.  I'll test this out and see if lld-as-ld-wrapper can
> > also add glibc RUNPATH.
>
> Zig manages linking to libc seperately, if this approach works, we have to
> keep validate-runpath? off for Zig.  Reference to ld.lld can't be patched
> since it's both native-inputs (for building Zig) and inputs (for using the
> built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?

ld.lld is invoked in 'zig clang' route, sorry for the noise.

Currently I'm 1. modifying each-lib-rpath option of 'zig build'.  2. passing
libc to linker.  I'll write details on this when succeed.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 14 Nov 2024 14:05:52 +0800,
Hilton Chain wrote:
>
> Currently I'm 1. modifying each-lib-rpath option of 'zig build'.  2. passing
> libc to linker.  I'll write details on this when succeed.

1. Modification about each-lib-rpath.
--8<---------------cut here---------------start------------->8---
-feach-lib-rpath               Ensure adding rpath for each used dynamic library
--8<---------------cut here---------------end--------------->8---

This option is on implicitly for native builds.  This implicity is what our Zig
currently solely relies on.

I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
set.  This approach is better than my previous one since it only adds needed
libraries.


2. Pass libc to Zig's linker
This was the behavior in 0.9, but changed due to issue on macOS[1].  (btw, our
CPLUS_INCLUDE_PATH also has issue with macOS target[2]).  RUNPATH for glibc was
missing because of this, since it's the linker handling each-lib-rpath.

Since we do not support macOS anyway, can we restore this behavior?


I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
find dynamic linker, env is chosen for it's well-known).  We have patched this
reference, not sure if it will cause issue for cross-building Zig.


Thanks
---
[1]: https://github.com/ziglang/zig/issues/10765
[2]: https://github.com/ziglang/zig/issues/18063

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

On Thu, Nov 14, 2024 at 05:22:17PM +0800, Hilton Chain wrote:
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> >
> > Currently I'm 1. modifying each-lib-rpath option of 'zig build'.  2. passing
> > libc to linker.  I'll write details on this when succeed.
> 
> 1. Modification about each-lib-rpath.
> --8<---------------cut here---------------start------------->8---
> -feach-lib-rpath               Ensure adding rpath for each used dynamic library
> --8<---------------cut here---------------end--------------->8---
> 
> This option is on implicitly for native builds.  This implicity is what our Zig
> currently solely relies on.
> 
> I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
> set.  This approach is better than my previous one since it only adds needed
> libraries.
> 
> 
> 2. Pass libc to Zig's linker
> This was the behavior in 0.9, but changed due to issue on macOS[1].  (btw, our
> CPLUS_INCLUDE_PATH also has issue with macOS target[2]).  RUNPATH for glibc was
> missing because of this, since it's the linker handling each-lib-rpath.
> 
> Since we do not support macOS anyway, can we restore this behavior?

At worst I could see adding a comment that it would likely break future
macOS cross-compiles.  I don't see an issue with either putting it back
unconditionally or trying to make it conditional based on
(%current-system) or the contents of (%current-target-system).

> 
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known).  We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

We use search-input-file in the replacement, so it should choose the
cross-binutils for the replacement /bin/env, so it shouldn't be a
problem.

> 
> Thanks
> ---
> [1]: https://github.com/ziglang/zig/issues/10765
> [2]: https://github.com/ziglang/zig/issues/18063

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Motiejus Jakštys]

On Thu, Nov 14, 2024 at 11:22 AM Hilton Chain <hako@ultrarare.space> wrote:
>
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known).  We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

This file is only consulted when `-target=native`. I.e. when it needs
to compile for the host. If target is specified, it will not consult
that file.

Just verified with zig 0.13.0:

$ strace -f -e openat zig cc hello.c -o hello |& grep -w env
openat(AT_FDCWD,
"/nix/store/sf6y4arqcm100rnnl3dhpg732i774zp6-coreutils-9.5/bin/env",
O_RDONLY|O_NOCTTY|O_CLOEXEC) = 5
$ strace -f -e openat zig cc -target x86_64-linux-gnu.2.32 hello.c -o
hello |& grep -w env
$

Motiejus

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 14 Nov 2024 17:41:04 +0800,
Efraim Flashner wrote:
>
> [1  <text/plain; utf-8 (quoted-printable)>]
> On Thu, Nov 14, 2024 at 05:22:17PM +0800, Hilton Chain wrote:
> > 2. Pass libc to Zig's linker
> > This was the behavior in 0.9, but changed due to issue on macOS[1].  (btw, our
> > CPLUS_INCLUDE_PATH also has issue with macOS target[2]).  RUNPATH for glibc was
> > missing because of this, since it's the linker handling each-lib-rpath.
> >
> > Since we do not support macOS anyway, can we restore this behavior?
>
> At worst I could see adding a comment that it would likely break future
> macOS cross-compiles.  I don't see an issue with either putting it back
> unconditionally or trying to make it conditional based on
> (%current-system) or the contents of (%current-target-system).

Bad news: Zig 0.12 changed[1] behavior of each_lib_rpath, it won't filter
libraries now.

Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)

Another forced push, I have ensured consistent behavior for
(CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
passing '-lc' to linker.

Who said not going to implement a ld-wrapper within Zig?  :P
Fortunately it was already there :)

BTW, adding pkg-config to native-inputs works for ncdu.

> > I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> > find dynamic linker, env is chosen for it's well-known).  We have patched this
> > reference, not sure if it will cause issue for cross-building Zig.
>
> We use search-input-file in the replacement, so it should choose the
> cross-binutils for the replacement /bin/env, so it shouldn't be a
> problem.


On Thu, 14 Nov 2024 17:47:23 +0800,
Motiejus Jakštys wrote:
>
> This file is only consulted when `-target=native`. I.e. when it needs
> to compile for the host. If target is specified, it will not consult
> that file.
>
> Just verified with zig 0.13.0:
>
> $ strace -f -e openat zig cc hello.c -o hello |& grep -w env
> openat(AT_FDCWD,
> "/nix/store/sf6y4arqcm100rnnl3dhpg732i774zp6-coreutils-9.5/bin/env",
> O_RDONLY|O_NOCTTY|O_CLOEXEC) = 5
> $ strace -f -e openat zig cc -target x86_64-linux-gnu.2.32 hello.c -o
> hello |& grep -w env
> $

Thanks!  Then it should be in inputs.  I don't want to add coreutils to it, so I
patched reference of /usr/bin/env to clang++ (it's both in inputs and RUNPATH)
instead.

---
[1]: https://github.com/ziglang/zig/commit/852e7e24b5f15b489463bdabb0039e2a424e5ee6

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Fri, 15 Nov 2024 11:29:10 +0800,
Hilton Chain wrote:
>
> Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
>
> Another forced push, I have ensured consistent behavior for
> (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> passing '-lc' to linker.
>
> Who said not going to implement a ld-wrapper within Zig?  :P
> Fortunately it was already there :)
>
> BTW, adding pkg-config to native-inputs works for ncdu.

I have locally made the "use-system-paths" patch larger so that Zig can really
honor "CROSS_" environment variables.

The next issue is cross building with pkg-config.  Zig only invokes
"pkg-config", but we don't have a "pkg-config" with search path for target
inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
dynamic linker path, I'll look into it soon.

Also for reproducibility, bin/zig is the only file differs and here's the diff,
I don't know about this part so I currently have no idea on fixing it.
--8<---------------cut here---------------start------------->8---
--- /gnu/store/gqdi4drfn3js5cwgfmlpkyfm2xf3l5b0-zig-0.10.1/bin/zig
+++ cuirass/gqdi4drfn3js5cwgfmlpkyfm2xf3l5b0-zig-0.10.1/bin/zig
├── readelf --wide --decompress --string-dump=.rodata {}
│ @@ -77024,14 +77024,16 @@
│    [149be0]  +�&
│    [149bf9]  )&
│    [149c12]  %
│    [149c28]  VO$
│    [149c40]  D�(
│    [149c59]  >$
│    [149c70]  8�%
│ +  [149c94]  ;
│ +  [149ca0]  ;
│    [149ca8]  '
│    [149cb0]  uespemos�odnarodarenegylsetybdet
│    [149cf0]  p��
│    [149d11]  O'
│    [149d21]  5&
│    [149d31]  f&
│    [149d40]  SJ'
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
--- /gnu/store/466cm9xpjqg80iqracj4qirsrdha1rnk-zig-0.11.0/bin/zig
+++ cuirass/466cm9xpjqg80iqracj4qirsrdha1rnk-zig-0.11.0/bin/zig
├── readelf --wide --decompress --string-dump=.rodata {}
│ @@ -64905,14 +64905,16 @@
│    [ 5ae48]  xpnt4win2kvistawin10ws2003win8_1win10_th2win10_rs1win10_rs2win10_rs3win10_rs4win10_rs5win10_19h1
│    [ 5aef0]
│    [ 5aef8]  #
│    [ 5af00]  %
│    [ 5af08]  %
│    [ 5af10]  &
│    [ 5af78]  celfhexrawmachospirvdxcontainer
│ +  [ 5afc0]  ;
│ +  [ 5aff8]  ;
│    [ 5b070]  E
│    [ 5b0b4]  ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_
│    [ 5b350]
│    [ 5b380]  @
│    [ 5b3e0]
│    [ 5b420]  ]
│    [ 5b5e0]  %
--8<---------------cut here---------------end--------------->8---

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Fri, 15 Nov 2024 22:30:40 +0800,
Hilton Chain wrote:
>
> I have locally made the "use-system-paths" patch larger so that Zig can really
> honor "CROSS_" environment variables.
>
> The next issue is cross building with pkg-config.  Zig only invokes
> "pkg-config", but we don't have a "pkg-config" with search path for target
> inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
> dynamic linker path, I'll look into it soon.

Adding a file with content like the following and passing --libc <this file> to
zig works, RUNPATH is correct and no need to set CC then.

--8<---------------cut here---------------start------------->8---
include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
msvc_lib_dir=
kernel32_lib_dir=
gcc_dir=
--8<---------------cut here---------------end--------------->8---

For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
output binary, I'll find a way to fix it.

> Also for reproducibility, bin/zig is the only file differs and here's the diff,
> I don't know about this part so I currently have no idea on fixing it.

This seem to be an upstream issue, Zig is reproducible only on the same machine.
I'll verify it and report to upstream.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Motiejus Jakštys]

On Sat, Nov 16, 2024 at 8:55 AM Hilton Chain <hako@ultrarare.space> wrote:
>
> On Fri, 15 Nov 2024 22:30:40 +0800,
> Hilton Chain wrote:
> >
> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
>
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

Zig defaults to `-march=native` when building for the host. Try
ZIG_TARGET_MCPU=baseline when building stage3.

Motiejus

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sat, 16 Nov 2024 15:13:40 +0800,
Motiejus Jakštys wrote:
> 
> On Sat, Nov 16, 2024 at 8:55 AM Hilton Chain <hako@ultrarare.space> wrote:
> >
> > On Fri, 15 Nov 2024 22:30:40 +0800,
> > Hilton Chain wrote:
> > >
> > > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > > I don't know about this part so I currently have no idea on fixing it.
> >
> > This seem to be an upstream issue, Zig is reproducible only on the same machine.
> > I'll verify it and report to upstream.
> 
> Zig defaults to `-march=native` when building for the host. Try
> ZIG_TARGET_MCPU=baseline when building stage3.

Yes, ZIG_TARGET_MCPU=baseline is passed to cmake.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

On Sat, Nov 16, 2024 at 02:54:55PM +0800, Hilton Chain wrote:
> On Fri, 15 Nov 2024 22:30:40 +0800,
> Hilton Chain wrote:
> >
> > I have locally made the "use-system-paths" patch larger so that Zig can really
> > honor "CROSS_" environment variables.
> >
> > The next issue is cross building with pkg-config.  Zig only invokes
> > "pkg-config", but we don't have a "pkg-config" with search path for target
> > inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
> > dynamic linker path, I'll look into it soon.

I tried adding pkg-config-for-build as a work-around but it wasn't
enough without touching the zig compiler's source too, which I didn't
attempt yesterday.

> Adding a file with content like the following and passing --libc <this file> to
> zig works, RUNPATH is correct and no need to set CC then.
> 
> --8<---------------cut here---------------start------------->8---
> include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
> msvc_lib_dir=
> kernel32_lib_dir=
> gcc_dir=
> --8<---------------cut here---------------end--------------->8---

Is this the layout of the file expected? That doesn't look too hard to
create in the build-system if necessary.

> For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
> output binary, I'll find a way to fix it.

I was going to say to take a look at gcc-2.95, where we point all the
linkers for all the architectures to whatever the target architecture
is, but that won't work here since we have 1 zig binary and it can
compile for any architecture.

I'm going to suggest against adding a cross-libc for all the different
architectures as an input, that would be crazy.

(ins)efraim@3900XT ~/workspace/zig$ git grep 'ld-linux-aarch64.so.1'
lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_LINUX_AARCH64_SO             "ld-linux-aarch64.so.1"
lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_SO                           "ld-linux-aarch64.so.1"
lib/std/Build.zig:/// that contains the path `aarch64-linux-gnu/lib/ld-linux-aarch64.so.1`.
lib/std/Target.zig:                .aarch64 => init("/lib/ld-linux-aarch64.so.1"),
Binary file stage1/zig1.wasm matches

Would it be possible to change the init("/path/to/ld.so") part to the
zig equivalent of (search-input-file inputs "/path/to/ld.so"), and then
when it is used from Guix the cross-libc will already be in the PATH and
therefore findable from zig's search through the vector¹ of the paths
inside PATH?

¹ I know it's not true but in my mind a vector and an array are the same
thing.

> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
> 
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 17 Nov 2024 01:03:59 +0800,
Efraim Flashner wrote:
>
> [1  <text/plain; utf-8 (quoted-printable)>]
> On Sat, Nov 16, 2024 at 02:54:55PM +0800, Hilton Chain wrote:
> > On Fri, 15 Nov 2024 22:30:40 +0800,
> > Hilton Chain wrote:
> > >
> > > I have locally made the "use-system-paths" patch larger so that Zig can really
> > > honor "CROSS_" environment variables.
> > >
> > > The next issue is cross building with pkg-config.  Zig only invokes
> > > "pkg-config", but we don't have a "pkg-config" with search path for target
> > > inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
> > > dynamic linker path, I'll look into it soon.
>
> I tried adding pkg-config-for-build as a work-around but it wasn't
> enough without touching the zig compiler's source too, which I didn't
> attempt yesterday.
>
> > Adding a file with content like the following and passing --libc <this file> to
> > zig works, RUNPATH is correct and no need to set CC then.
> >
> > --8<---------------cut here---------------start------------->8---
> > include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> > sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> > crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
> > msvc_lib_dir=
> > kernel32_lib_dir=
> > gcc_dir=
> > --8<---------------cut here---------------end--------------->8---
>
> Is this the layout of the file expected? That doesn't look too hard to
> create in the build-system if necessary.
>
> > For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
> > output binary, I'll find a way to fix it.
>
> I was going to say to take a look at gcc-2.95, where we point all the
> linkers for all the architectures to whatever the target architecture
> is, but that won't work here since we have 1 zig binary and it can
> compile for any architecture.
>
> I'm going to suggest against adding a cross-libc for all the different
> architectures as an input, that would be crazy.
>
> (ins)efraim@3900XT ~/workspace/zig$ git grep 'ld-linux-aarch64.so.1'
> lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_LINUX_AARCH64_SO             "ld-linux-aarch64.so.1"
> lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_SO                           "ld-linux-aarch64.so.1"
> lib/std/Build.zig:/// that contains the path `aarch64-linux-gnu/lib/ld-linux-aarch64.so.1`.
> lib/std/Target.zig:                .aarch64 => init("/lib/ld-linux-aarch64.so.1"),
> Binary file stage1/zig1.wasm matches
>
> Would it be possible to change the init("/path/to/ld.so") part to the
> zig equivalent of (search-input-file inputs "/path/to/ld.so"), and then
> when it is used from Guix the cross-libc will already be in the PATH and
> therefore findable from zig's search through the vector¹ of the paths
> inside PATH?
>
> ¹ I know it's not true but in my mind a vector and an array are the same
> thing.

I have added a GUIX_ZIG_LIBC_DIR environment variable, to be set as output path
of cross-libc or libc by zig-build-system, patched Zig to search it and
concatenate it with "/lib/ld...".

Also added the file for --libc option in zig-build-system.  Cross compilation is
available now.  (only available in 0.12 for now, I'll port the patches to other
versions when I get up.)

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sat, 16 Nov 2024 14:54:55 +0800,
Hilton Chain wrote:
>
> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
>
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

Opened a GitHub issue: <https://github.com/ziglang/zig/issues/22002>.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1754 bytes --]

On Fri, Nov 15, 2024 at 10:30:40PM +0800, Hilton Chain wrote:
> On Fri, 15 Nov 2024 11:29:10 +0800,
> Hilton Chain wrote:
> >
> > Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
> >
> > Another forced push, I have ensured consistent behavior for
> > (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> > passing '-lc' to linker.
> >
> > Who said not going to implement a ld-wrapper within Zig?  :P
> > Fortunately it was already there :)
> >
> > BTW, adding pkg-config to native-inputs works for ncdu.
> 
> I have locally made the "use-system-paths" patch larger so that Zig can really
> honor "CROSS_" environment variables.
> 
> The next issue is cross building with pkg-config.  Zig only invokes
> "pkg-config", but we don't have a "pkg-config" with search path for target
> inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
> dynamic linker path, I'll look into it soon.

I found a patch after the 0.13.0 release that switches from hardcoding
pkg-config to using the PKG_CONFIG environment variable and falling back
to pkg-config, so I backported it to 0.12 and was able to use that and
guix's regular pkg-config package.  I've added those patches to the
wip-zig-bootstrap tree.

We now have a couple of phases that are before the 'build phase, do you
think it'd be better to consolidate them into a 'configure phase?
There's no 'configure' script to run, but it does do a lot of
preparation before the actual 'build phase...

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 17 Nov 2024 15:16:13 +0800,
Efraim Flashner wrote:
> 
> [1  <text/plain; utf-8 (quoted-printable)>]
> On Fri, Nov 15, 2024 at 10:30:40PM +0800, Hilton Chain wrote:
> > On Fri, 15 Nov 2024 11:29:10 +0800,
> > Hilton Chain wrote:
> > >
> > > Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
> > >
> > > Another forced push, I have ensured consistent behavior for
> > > (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> > > passing '-lc' to linker.
> > >
> > > Who said not going to implement a ld-wrapper within Zig?  :P
> > > Fortunately it was already there :)
> > >
> > > BTW, adding pkg-config to native-inputs works for ncdu.
> > 
> > I have locally made the "use-system-paths" patch larger so that Zig can really
> > honor "CROSS_" environment variables.
> > 
> > The next issue is cross building with pkg-config.  Zig only invokes
> > "pkg-config", but we don't have a "pkg-config" with search path for target
> > inputs.  I can add a pkg-config-for-zig to workaround this, and then... It's
> > dynamic linker path, I'll look into it soon.
> 
> I found a patch after the 0.13.0 release that switches from hardcoding
> pkg-config to using the PKG_CONFIG environment variable and falling back
> to pkg-config, so I backported it to 0.12 and was able to use that and
> guix's regular pkg-config package.  I've added those patches to the
> wip-zig-bootstrap tree.

Thanks, I have ported all patches and pushed.  GUIX_ZIG_LIBC_DIR is changed to
GUIX_ZIG_GLIBC_LINKER and is set as full path in Guix side because I don't want
mess with strings in Zig side...

> We now have a couple of phases that are before the 'build phase, do you
> think it'd be better to consolidate them into a 'configure phase?
> There's no 'configure' script to run, but it does do a lot of
> preparation before the actual 'build phase...

I have merged these phases into configure, forgot to change commit log though.


The reproducibility issue is related to kernel version from target ("native" by
default) information, to address this we need to specify a target for native
builds too.[1]

---
[1]: https://github.com/ziglang/zig/issues/22002#issuecomment-2480933071

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 17 Nov 2024 22:51:48 +0800,
Hilton Chain wrote:
>
> The reproducibility issue is related to kernel version from target ("native" by
> default) information, to address this we need to specify a target for native
> builds too.

Specifying a target makes Zig behave like in cross-compilation, fortunately
workaround for zig-build-system still applies here too.

Marked the commit as DRAFT since I'm still building it.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Sun, 17 Nov 2024 22:51:48 +0800,
Hilton Chain wrote:
>
> Thanks, I have ported all patches and pushed.  GUIX_ZIG_LIBC_DIR is changed to
> GUIX_ZIG_GLIBC_LINKER and is set as full path in Guix side because I don't want
> mess with strings in Zig side...

Reworked this, patched Zig to search dynamic linker in CROSS_LIBRARY_PATH or
LIBRARY_PATH (added in the use-system-paths patch), no extra environment
variable introduced now.

> > We now have a couple of phases that are before the 'build phase, do you
> > think it'd be better to consolidate them into a 'configure phase?
> > There's no 'configure' script to run, but it does do a lot of
> > preparation before the actual 'build phase...
>
> I have merged these phases into configure, forgot to change commit log though.

Moved zig-target and zig-build-system's configure phase to (guix build
zig-utils) to avoid dependency issue, as building Zig now uses them too.

Removed nonfree files (IETF RFC documents) from Zig source.

I'll take a look at Zig package manager support.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Tue, 19 Nov 2024 21:13:50 +0800,
Hilton Chain wrote:
>
> I'll take a look at Zig package manager support.

Added support for Zig package manager (build.zig.zon) and switched default Zig
to zig-0.13, please see the two DRAFT commits.  I have also updated and added
some packages as examples.

Since Zig has command line option changes in 0.11 (those are affecting us:
"-Drelease-safe" -> "-Doptimeze=ReleaseSafe", new "-j"), I added a zig-arguments
procedure which returns an alist mapping command line options.

Motiejus raised concern about Zig's bundled abilist file[1][2], which is
generated from sources of various glibc versions.  How should we treat it?

[1]: https://github.com/ziglang/zig/tree/master/lib/libc/glibc
[2]: https://github.com/ziglang/glibc-abi-tool

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

I started to learn Zig these days and adjusted patches for Zig a bit.

Other than this, I have added a install-source? argument to zig-build-system, it
can now install package source to "/src/<name>-<version>".

"-Dtarget" is used in zig-build-system's native builds as well, same as Zig for
reproducibility.

Made "#:zig-inputs" a private keyword, its value will be passed to inputs like
cargo-build-system (the build system doesn't extract the whole closure for now,
this can be added in the future depending on how Zig ecosystem develops).

Set ZIG_GLOBAL_CACHE_DIR and ZIG_LOCAL_CACHE_DIR to "/tmp/zig-cache", and set
ZIG_LIBC instead of using "zig build --libc", for convenience.

I think this is mostly ready ( again :) ).  If there's nothing else missing,
I'll post the current status and call for packages to test the build system on
guix-devel, together with the concern on generated files.

bug#74217: Bootstrapping Zig with no Binary Blobs

[Motiejus Jakštys]

On Thu, Nov 28, 2024 at 1:08 PM Hilton Chain <hako@ultrarare.space> wrote:
>
> <...> together with the concern on generated files.

Hi Hilton,

The only remaining, to my knowledge, binary file is `abilists`, which,
once you have `zig` binary, can be generated this way:

git clone https://github.com/ziglang/glibc-abi-tool; cd glibc-abi-tool
git checkout fc5d0a7046b76795e4219f8f168e118ec29fbc53
/path/to/zig-0.13/bin/zig run consolidate.zig
mv abilists /path/to/zig/lib/libc/glibc/abilist

For 0.12.1:
rm -fr glibc/2.39
sed -i '133d' consolidate.zig
/path/to/zig-0.12.1/bin/zig run consolidate.zig
mv abilists /path/to/zig/lib/libc/glibc/abilist

I wasn't able to generate it for 0.11, but perhaps it's not as
important, as I imagine there are very few people, if any, still
developing on 0.11. Since one needs abilists *only* to cross-compile
to non-guix glibc targets, it sounds like a vanishingly small use case
for Guix to support.

Motiejus

bug#74217: Bootstrapping Zig with no Binary Blobs

[Hilton Chain via Bug reports for GNU Guix]

On Thu, 28 Nov 2024 20:41:10 +0800,
Motiejus Jakštys wrote:
>
> On Thu, Nov 28, 2024 at 1:08 PM Hilton Chain <hako@ultrarare.space> wrote:
> >
> > <...> together with the concern on generated files.
>
> Hi Hilton,
>
> The only remaining, to my knowledge, binary file is `abilists`, which,
> once you have `zig` binary, can be generated this way:
>
> git clone https://github.com/ziglang/glibc-abi-tool; cd glibc-abi-tool
> git checkout fc5d0a7046b76795e4219f8f168e118ec29fbc53
> /path/to/zig-0.13/bin/zig run consolidate.zig
> mv abilists /path/to/zig/lib/libc/glibc/abilist
>
> For 0.12.1:
> rm -fr glibc/2.39
> sed -i '133d' consolidate.zig
> /path/to/zig-0.12.1/bin/zig run consolidate.zig
> mv abilists /path/to/zig/lib/libc/glibc/abilist
>
> I wasn't able to generate it for 0.11, but perhaps it's not as
> important, as I imagine there are very few people, if any, still
> developing on 0.11. Since one needs abilists *only* to cross-compile
> to non-guix glibc targets, it sounds like a vanishingly small use case
> for Guix to support.
>
> Motiejus

Thanks!  Then I'll keep abilists removed before we can reproduce one :)