[bug#41574] gnu: Add xed.

[bug#41574] gnu: Add xed.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 1626 bytes --]

This patch packages up Intel's X86 Encoder and Decoder library and associated
cli tool "examples."

A few things of note:

1) The build uses Intel's custom Python build tool `mbuild' so we have to
   manually handle the main build phases. We may need to add explicit options
   to the build script invocation so that build variables (e.g. CFLAGS etc.)
   propogate correctly. These don't look to be set in the environment, so what
   variables should we pick be picking up and from where?

2) The group of tests under `tests/tests-avx512pf' seems to be failing. A user
   on the irc channel also cross-checked for me and confirmed the same. This
   program isn't actually *executing* the avx instructions, so I don't think
   the failing test are specific to the executing cpu. Anyway, I opted to leave
   this test in the source commented out.

3) The commands provided by the `out' output are pretty poorly documented and
   have dumb names. I suspose this is becase the utilities are branded as just
   "examples" of using the library. Anyway, this is a case where the only
   reasonable documentation is the source code, so I provide that for the
   utilities in the `doc' output.

4) Finally, the `devel' output supplies the library and headers proper.

5) The package name `xed' potentially collides with the package from
   http://xed.sourceforge.net/. We don't currently have the latter yet, but I
   mention this just in case there is a good way to proactively handle this up
   front.

Thoughts? I threw this together just because I wanted it myself but figured
it's worth sharing.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-xed.patch --]
[-- Type: text/x-patch, Size: 5656 bytes --]

From 9ec113c2bfed745ffe28bdbc4510fa799bea5199 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Thu, 28 May 2020 07:32:28 +0900
Subject: [PATCH] gnu: Add xed.
To: guix-patches@gnu.org

* gnu/packages/assembly.scm (xed): New variable.
---
 gnu/packages/assembly.scm | 92 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)

diff --git a/gnu/packages/assembly.scm b/gnu/packages/assembly.scm
index c775603445..f0293fe54d 100644
--- a/gnu/packages/assembly.scm
+++ b/gnu/packages/assembly.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2019 Andy Tai <atai@atai.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Christopher Lemmer Webber <cwebber@dustycloud.org>
+;;; Copyright © 2020 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -371,3 +372,94 @@ It understands mnemonics and generates code for NMOS 6502s (such
 as 6502A, 6504, 6507, 6510, 7501, 8500, 8501, 8502 ...),
  CMOS 6502s (65C02 and Rockwell R65C02) and the 65816.")
     (license license:gpl2)))
+
+(define-public xed
+  (package
+    (name "xed")
+    (version "11.2.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "https://github.com/intelxed/xed/archive/"
+                           version ".tar.gz"))
+       (sha256
+        (base32 "1dl23wxz0dlkbcw6k78njnz2yc8a8yr4lp979sd9x1lnz74malvi"))
+       (file-name (string-append name "-" version ".tar.gz"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("python-2" ,python-2)
+       ("python-3" ,python-3)
+       ("mbuild"
+        ,(let ((name "mbuild")
+               (version "0.2496"))
+           (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/intelxed/mbuild.git")
+                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
+             (sha256
+              (base32
+               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
+             (file-name (git-file-name name version)))))))
+    (arguments
+     `(#:phases
+       ;; Upstream uses the custom Python build tool `mbuild', so we munge
+       ;; gnu-build-system to fit.  The build process for this package is
+       ;; documented at https://intelxed.github.io/build-manual/.
+       (let* ((build-dir "build")
+              (kit-dir "kit"))
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (replace 'build
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((mbuild (assoc-ref inputs "mbuild")))
+                 ;; The build system requires that `mbuild' be in a sibling
+                 ;; directory to the project's source.
+                 (symlink mbuild "../mbuild")
+                 (invoke "./mfile.py"
+                         (string-append "--build-dir=" build-dir)
+                         (string-append "--install-dir=" kit-dir)
+                         "examples"
+                         "doc"
+                         "install"))))
+           (replace 'check
+             (lambda _
+               (invoke "tests/run-cmd.py"
+                       (string-append "--build-dir=" kit-dir "/bin")
+                       "--tests" "tests/tests-base"
+                       "--tests" "tests/tests-avx512"
+                       ; "--tests" "tests/tests-avx512pf"  ; Broken
+                       "--tests" "tests/tests-cet"
+                       "--tests" "tests/tests-via"
+                       "--tests" "tests/tests-syntax"
+                       "--tests" "tests/tests-xop")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (doc (assoc-ref outputs "doc"))
+                      (devel (assoc-ref outputs "devel"))
+                      (pkg (strip-store-file-name out)))
+                 (copy-recursively (string-append kit-dir "/bin")
+                                   (string-append out "/bin"))
+                 (copy-recursively (string-append kit-dir "/examples")
+                                   (string-append doc "/share/doc/" pkg
+                                                  "/examples"))
+                 (copy-recursively (string-append kit-dir "/include")
+                                   (string-append devel "/include"))
+                 (copy-recursively (string-append kit-dir "/lib")
+                                   (string-append devel "/lib")))))))))
+    (outputs '("out" "devel" "doc"))
+    (home-page "https://intelxed.github.io/")
+    (synopsis "Encoder and decoder for x86 (IA32 and Intel64) instructions")
+    (description "The X86 Encoder Decoder (XED) is a software library and
+for encoding and decoding X86 (IA32 and Intel64) instructions.  The decoder
+takes sequences of 1-15 bytes along with machine mode information and produces
+a data structure describing the opcode, operands, and flags.  The encoder takes
+a similar data structure and produces a sequence of 1 to 15 bytes.  Disassembly
+is essentially a printing pass on the data structure.
+
+The default output contains a family of command line wrappers @code{xed*}
+around the library with the development files themselves in the @code{devel}
+output.  See the @code{doc} output for documentation and examples about the
+library, including the source code to the @code{xed*} utilities.")
+    (license license:asl2.0)))
-- 
2.26.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#41574] gnu: Add xev.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 405 bytes --]

Updated patch to fix three issues:

1) Change output name `doc' to `src' and install under src/<name>-<version>/,
2) Change output name `devel' to `lib', and
3) Delete extraneous *.py files from `lib' output.

However, in the course of the above, I ran into an issue with a
non-deterministic build. For now I'm sharing the current state of the patch but
am looking into fixing the determinism.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-xed.patch --]
[-- Type: text/x-patch, Size: 6101 bytes --]

From 4a4073c3b7ba588ab65c406d018336524a4c29b9 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Thu, 28 May 2020 07:32:28 +0900
Subject: [PATCH] gnu: Add xed.

* gnu/packages/assembly.scm (xed): New variable.
---
 gnu/packages/assembly.scm | 96 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 96 insertions(+)

diff --git a/gnu/packages/assembly.scm b/gnu/packages/assembly.scm
index c775603445..0af9f2a237 100644
--- a/gnu/packages/assembly.scm
+++ b/gnu/packages/assembly.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2019 Andy Tai <atai@atai.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Christopher Lemmer Webber <cwebber@dustycloud.org>
+;;; Copyright © 2020 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -47,6 +48,7 @@
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages python)
   #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages shells)
   #:use-module (gnu packages xml)
   #:use-module ((guix utils)
                 #:select (%current-system)))
@@ -371,3 +373,97 @@ It understands mnemonics and generates code for NMOS 6502s (such
 as 6502A, 6504, 6507, 6510, 7501, 8500, 8501, 8502 ...),
  CMOS 6502s (65C02 and Rockwell R65C02) and the 65816.")
     (license license:gpl2)))
+
+(define-public xed
+  (package
+    (name "xed")
+    (version "11.2.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "https://github.com/intelxed/xed/archive/"
+                           version ".tar.gz"))
+       (sha256
+        (base32 "1dl23wxz0dlkbcw6k78njnz2yc8a8yr4lp979sd9x1lnz74malvi"))
+       (file-name (string-append name "-" version ".tar.gz"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("python-2" ,python-2)
+       ("python-3" ,python-3)
+       ("tcsh" ,tcsh)
+       ("mbuild"
+        ,(let ((name "mbuild")
+               (version "0.2496"))
+           (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/intelxed/mbuild.git")
+                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
+             (sha256
+              (base32
+               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
+             (file-name (git-file-name name version)))))))
+    (outputs '("out" "lib" "src"))
+    (arguments
+     `(#:phases
+       ;; Upstream uses the custom Python build tool `mbuild', so we munge
+       ;; gnu-build-system to fit.  The build process for this package is
+       ;; documented at https://intelxed.github.io/build-manual/.
+       (let* ((build-dir "build")
+              (kit-dir "kit"))
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (replace 'build
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((mbuild (assoc-ref inputs "mbuild")))
+                 ;; The build system requires that `mbuild' be in a sibling
+                 ;; directory to the project's source.
+                 (symlink mbuild "../mbuild")
+                 (invoke "./mfile.py"
+                         (string-append "--build-dir=" build-dir)
+                         (string-append "--install-dir=" kit-dir)
+                         "examples"
+                         "doc"
+                         "install"))))
+           (replace 'check
+             (lambda _
+               (invoke "tests/run-cmd.py"
+                       ;; Skip broken test group `tests/tests-avx512pf'.
+                       (string-append "--build-dir=" kit-dir "/bin")
+                       "--tests" "tests/tests-base"
+                       "--tests" "tests/tests-avx512"
+                       "--tests" "tests/tests-cet"
+                       "--tests" "tests/tests-via"
+                       "--tests" "tests/tests-syntax"
+                       "--tests" "tests/tests-xop")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (lib (assoc-ref outputs "lib"))
+                      (src (assoc-ref outputs "src"))
+                      (pkg (strip-store-file-name out))
+                      (src-dir (string-append src "/src/" pkg "/examples")))
+                 (copy-recursively (string-append kit-dir "/bin")
+                                   (string-append out "/bin"))
+                 (copy-recursively (string-append kit-dir "/examples") src-dir)
+                 (copy-recursively (string-append kit-dir "/include") src-dir)
+                 ;; Discard residual build system files.
+                 (for-each delete-file (find-files src-dir "\\.py$"))
+                 (copy-recursively (string-append kit-dir "/include")
+                                   (string-append lib "/include"))
+                 (copy-recursively (string-append kit-dir "/lib")
+                                   (string-append lib "/lib")))))))))
+    (home-page "https://intelxed.github.io/")
+    (synopsis "Encoder and decoder for x86 (IA32 and Intel64) instructions")
+    (description "The X86 Encoder Decoder (XED) is a software library and
+for encoding and decoding X86 (IA32 and Intel64) instructions.  The decoder
+takes sequences of 1-15 bytes along with machine mode information and produces
+a data structure describing the opcode, operands, and flags.  The encoder takes
+a similar data structure and produces a sequence of 1 to 15 bytes.  Disassembly
+is essentially a printing pass on the data structure.
+
+The library and development files are under the @code{lib} output, with a
+family of command line utility wrappers in the default output.  Each of the cli
+tools is named like @code{xed*}.  Documentation for the cli tools is sparse, so
+see the @code{src} output for the corresponding source code.")
+    (license license:asl2.0)))
-- 
2.26.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#41574] gnu: Add intel-xev.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 182 bytes --]

This patch makes two main changes:

1) Fixes upstream's source of non-determinism!
2) Renames packages from `xed` to `intel-xed`,

along with a few other minor improvements.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-intel-xed.patch --]
[-- Type: text/x-patch, Size: 13635 bytes --]

From 4e0d690a702fbfc983cf2d981d4f07f1eb79ede3 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Thu, 28 May 2020 07:32:28 +0900
Subject: [PATCH] gnu: Add intel-xed.
To: guix-patches@gnu.org

* gnu/packages/assembly.scm (intel-xed): New variable.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/assembly.scm                     | 105 +++++++++++++++++-
 .../intel-xed-fix-nondeterminism.patch        | 100 +++++++++++++++++
 3 files changed, 202 insertions(+), 4 deletions(-)
 create mode 100644 gnu/packages/patches/intel-xed-fix-nondeterminism.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index babcb8f6ad..9cefb31235 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1103,6 +1103,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ilmbase-fix-test-arm.patch		\
   %D%/packages/patches/inetutils-hurd.patch			\
   %D%/packages/patches/inkscape-poppler-0.76.patch		\
+  %D%/packages/patches/intel-xed-fix-nondeterminism.patch	\
   %D%/packages/patches/intltool-perl-compatibility.patch	\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
diff --git a/gnu/packages/assembly.scm b/gnu/packages/assembly.scm
index c9582bc596..c78400f5fc 100644
--- a/gnu/packages/assembly.scm
+++ b/gnu/packages/assembly.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2019 Andy Tai <atai@atai.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Christopher Lemmer Webber <cwebber@dustycloud.org>
+;;; Copyright © 2020 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -47,6 +48,7 @@
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages python)
   #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages shells)
   #:use-module (gnu packages xml)
   #:use-module ((guix utils)
                 #:select (%current-system)))
@@ -149,14 +151,14 @@ to the clients.")
 (define-public fasm
   (package
     (name "fasm")
-    (version "1.73.24")
+    (version "1.73.22")
     (source
      (origin
        (method url-fetch)
        (uri (string-append "https://flatassembler.net/fasm-"
                            version ".tgz"))
        (sha256
-        (base32 "142vxhs8mh8isvlzq7ir0asmqda410phzxmk9gk9b43dldskkj7k"))))
+        (base32 "1pb0rcfdsb0h89khjjrbikz5wjdllavj3ajim0rcyh7x12xr1hw5"))))
     (build-system gnu-build-system)
     (arguments
      `(#:tests? #f                      ; no tests exist
@@ -347,14 +349,14 @@ Supported architectures are:
 (define-public xa
   (package
     (name "xa")
-    (version "2.3.11")
+    (version "2.3.10")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://www.floodgap.com/retrotech/xa"
                                   "/dists/xa-" version ".tar.gz"))
               (sha256
                (base32
-                "0b81r7mvzqxgnbbmhixcnrf9nc72v1nqaw19k67221g3k561dwij"))))
+                "0y5sd247g11jfk5msxy91hz2nhpy7smj125dzfyfhjsjnqk5nyw6"))))
     (build-system gnu-build-system)
     (arguments
      `(#:tests? #f   ; TODO: custom test harness, not sure how it works
@@ -371,3 +373,98 @@ It understands mnemonics and generates code for NMOS 6502s (such
 as 6502A, 6504, 6507, 6510, 7501, 8500, 8501, 8502 ...),
  CMOS 6502s (65C02 and Rockwell R65C02) and the 65816.")
     (license license:gpl2)))
+
+(define-public intel-xed
+  (package
+    (name "intel-xed")
+    (version "11.2.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/intelxed/xed.git")
+             (commit "40125558530137444b4ee6fd26b445bfa105b543")))
+       (sha256 (base32 "1jffayski2gpd54vaska7fmiwnnia8v3cka4nfyzjgl8xsky9v2s"))
+       (file-name (git-file-name name version))
+       (patches (search-patches "intel-xed-fix-nondeterminism.patch"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("python-2" ,python-2)
+       ("python-3" ,python-3)
+       ("tcsh" ,tcsh)
+       ("mbuild"
+        ,(let ((name "mbuild")
+               (version "0.2496"))
+           (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/intelxed/mbuild.git")
+                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
+             (sha256
+              (base32
+               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
+             (file-name (git-file-name name version)))))))
+    (outputs '("out" "lib" "src"))
+    (arguments
+     `(#:phases
+       ;; Upstream uses the custom Python build tool `mbuild', so we munge
+       ;; gnu-build-system to fit.  The build process for this package is
+       ;; documented at https://intelxed.github.io/build-manual/.
+       (let* ((build-dir "build")
+              (kit-dir "kit"))
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (replace 'build
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((mbuild (assoc-ref inputs "mbuild")))
+                 (setenv "PYTHONPATH" (string-append
+                                        (getenv "PYTHONPATH") ":" mbuild))
+                 (invoke "./mfile.py"
+                         (string-append "--build-dir=" build-dir)
+                         (string-append "--install-dir=" kit-dir)
+                         "examples"
+                         "doc"
+                         "install"))))
+           (replace 'check
+             (lambda _
+               (invoke "tests/run-cmd.py"
+                       ;; Skip broken test group `tests/tests-avx512pf'.
+                       (string-append "--build-dir=" kit-dir "/bin")
+                       "--tests" "tests/tests-base"
+                       "--tests" "tests/tests-avx512"
+                       "--tests" "tests/tests-cet"
+                       "--tests" "tests/tests-via"
+                       "--tests" "tests/tests-syntax"
+                       "--tests" "tests/tests-xop")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (lib (assoc-ref outputs "lib"))
+                      (src (assoc-ref outputs "src"))
+                      (pkg (strip-store-file-name out))
+                      (src-dir (string-append src "/src/" pkg "/examples")))
+                 (copy-recursively (string-append kit-dir "/bin")
+                                   (string-append out "/bin"))
+                 (copy-recursively (string-append kit-dir "/examples") src-dir)
+                 (copy-recursively (string-append kit-dir "/include") src-dir)
+                 ;; Discard residual build system files.
+                 (for-each delete-file (find-files src-dir "\\.py$"))
+                 (copy-recursively (string-append kit-dir "/include")
+                                   (string-append lib "/include"))
+                 (copy-recursively (string-append kit-dir "/lib")
+                                   (string-append lib "/lib"))
+                 #t)))))))
+    (home-page "https://intelxed.github.io/")
+    (synopsis "Encoder and decoder for x86 (IA32 and Intel64) instructions")
+    (description "The Intel X86 Encoder Decoder (XED) is a software library and
+for encoding and decoding X86 (IA32 and Intel64) instructions.  The decoder
+takes sequences of 1-15 bytes along with machine mode information and produces
+a data structure describing the opcode, operands, and flags.  The encoder takes
+a similar data structure and produces a sequence of 1 to 15 bytes.  Disassembly
+is essentially a printing pass on the data structure.
+
+The library and development files are under the @code{lib} output, with a
+family of command line utility wrappers in the default output.  Each of the cli
+tools is named like @code{xed*}.  Documentation for the cli tools is sparse, so
+see the @code{src} output for the corresponding source code.")
+    (license license:asl2.0)))
diff --git a/gnu/packages/patches/intel-xed-fix-nondeterminism.patch b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
new file mode 100644
index 0000000000..657f7e979d
--- /dev/null
+++ b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
@@ -0,0 +1,100 @@
+diff --git a/pysrc/ild_codegen.py b/pysrc/ild_codegen.py
+index 628ec45..a9bff79 100755
+--- a/pysrc/ild_codegen.py
++++ b/pysrc/ild_codegen.py
+@@ -188,14 +188,14 @@ def gen_l2_func_list(agi, target_nt_dict, arg_nt_dict,
+                      ild_t_member):
+     """generate L2 functions"""
+     l2_func_list = []
+-    for (nt_name,array) in target_nt_dict.items():
++    for (nt_name,array) in sorted(target_nt_dict.items()):
+         target_opname = array.get_target_opname()
+         if array.is_const_lookup_fun():
+             fo = gen_const_l2_function(agi, nt_name,
+                                 target_opname, ild_t_member)
+             l2_func_list.append(fo)
+         else:
+-            for arg_nt_seq,arg_arr in arg_nt_dict.items():
++            for arg_nt_seq,arg_arr in sorted(arg_nt_dict.items()):
+                 fo = gen_scalable_l2_function(agi, nt_name,
+                      target_opname, ild_t_member, arg_arr, list(arg_nt_seq))
+                 l2_func_list.append(fo)
+diff --git a/pysrc/ild_disp.py b/pysrc/ild_disp.py
+index 942c036..cf80e29 100755
+--- a/pysrc/ild_disp.py
++++ b/pysrc/ild_disp.py
+@@ -350,7 +350,8 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     disp_dict = _gen_l3_array_dict(agi, disp_nts, _disp_token)
+ 
+     
+-    nt_arr_list = list(brdisp_dict.values()) + list(disp_dict.values())
++    nt_arr_list = ([v for (k,v) in sorted(brdisp_dict.items())] +
++                   [v for (k,v) in sorted(disp_dict.items())])
+     #create function that calls all initialization functions
+     init_f = ild_nt.gen_init_function(nt_arr_list, 'xed_ild_disp_l3_init')
+     
+@@ -367,7 +368,7 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     l2_functions = []
+     eosz_op = ild_eosz.get_target_opname()
+     easz_op = ild_easz.get_target_opname()
+-    for nt_name,array in list(disp_dict.items()) + list(brdisp_dict.items()):
++    for nt_name,array in sorted(disp_dict.items()) + sorted(brdisp_dict.items()):
+         #Some DISP NTs depend on EOSZ, others on EASZ, we need to know
+         #that when we generate L2 functions
+         if eosz_op in array.get_arg_names():
+diff --git a/pysrc/ild_easz.py b/pysrc/ild_easz.py
+index 02cd691..c53b9f2 100755
+--- a/pysrc/ild_easz.py
++++ b/pysrc/ild_easz.py
+@@ -165,9 +165,10 @@ def work(agi, united_lookup, easz_nts, ild_gendir, debug):
+             return
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()),
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values,
+                                        'xed_ild_easz_init')
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _easz_c_fn, 
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _easz_c_fn, 
+                           mbuild.join('include-private', _easz_header_fn),
+                           init_f)
+     getter_fos = []
+diff --git a/pysrc/ild_eosz.py b/pysrc/ild_eosz.py
+index 6643bc3..89d2d89 100755
+--- a/pysrc/ild_eosz.py
++++ b/pysrc/ild_eosz.py
+@@ -200,10 +200,11 @@ def work(agi, united_lookup, eosz_nts, ild_gendir, debug):
+             return None
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()), 
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values, 
+                                       'xed_ild_eosz_init')
+     #dump init and lookup functions for EOSZ sequences
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _eosz_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _eosz_c_fn,
+                           mbuild.join('include-private', _eosz_header_fn),
+                           init_f)
+     #generate EOSZ getter functions - they get xed_decoded_inst_t*
+diff --git a/pysrc/ild_imm.py b/pysrc/ild_imm.py
+index 51c413c..0530bae 100755
+--- a/pysrc/ild_imm.py
++++ b/pysrc/ild_imm.py
+@@ -322,12 +322,14 @@ def work(agi, united_lookup, imm_nts, ild_gendir, eosz_dict,
+                                      level='l3')
+         nt_dict[nt_name] = array
+ 
++    nt_dict_values = [v for (k,v) in sorted(nt_dict.items())]
++
+     #create function that calls all initialization functions for L3
+-    init_f = ild_nt.gen_init_function(list(nt_dict.values()),
++    init_f = ild_nt.gen_init_function(nt_dict_values,
+                                       'xed_ild_imm_l3_init')
+     
+     #dump L3 functions
+-    ild_nt.dump_lu_arrays(agi, list(nt_dict.values()), _l3_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_dict_values, _l3_c_fn,
+                           mbuild.join('include-private',_l3_header_fn),
+                           init_f)
+     
-- 
2.27.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#41574] gnu: Add intel-xev.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3775 bytes --]

elaexuotee--- via Guix-patches via <guix-patches@gnu.org> writes:

> This patch makes two main changes:
>
> 1) Fixes upstream's source of non-determinism!
> 2) Renames packages from `xed` to `intel-xed`,
>
> along with a few other minor improvements.
>
> From 4e0d690a702fbfc983cf2d981d4f07f1eb79ede3 Mon Sep 17 00:00:00 2001
> From: "B. Wilson" <elaexuotee@wilsonb.com>
> Date: Thu, 28 May 2020 07:32:28 +0900
> Subject: [PATCH] gnu: Add intel-xed.
> To: guix-patches@gnu.org
>
> * gnu/packages/assembly.scm (intel-xed): New variable.
> ---
>  gnu/local.mk                                  |   1 +
>  gnu/packages/assembly.scm                     | 105 +++++++++++++++++-
>  .../intel-xed-fix-nondeterminism.patch        | 100 +++++++++++++++++
>  3 files changed, 202 insertions(+), 4 deletions(-)
>  create mode 100644 gnu/packages/patches/intel-xed-fix-nondeterminism.patch

[...]

> @@ -149,14 +151,14 @@ to the clients.")
>  (define-public fasm
>    (package
>      (name "fasm")
> -    (version "1.73.24")
> +    (version "1.73.22")
>      (source
>       (origin
>         (method url-fetch)
>         (uri (string-append "https://flatassembler.net/fasm-"
>                             version ".tgz"))
>         (sha256
> -        (base32 "142vxhs8mh8isvlzq7ir0asmqda410phzxmk9gk9b43dldskkj7k"))))
> +        (base32 "1pb0rcfdsb0h89khjjrbikz5wjdllavj3ajim0rcyh7x12xr1hw5"))))
>      (build-system gnu-build-system)
>      (arguments
>       `(#:tests? #f                      ; no tests exist
> @@ -347,14 +349,14 @@ Supported architectures are:
>  (define-public xa
>    (package
>      (name "xa")
> -    (version "2.3.11")
> +    (version "2.3.10")
>      (source (origin
>                (method url-fetch)
>                (uri (string-append "https://www.floodgap.com/retrotech/xa"
>                                    "/dists/xa-" version ".tar.gz"))
>                (sha256
>                 (base32
> -                "0b81r7mvzqxgnbbmhixcnrf9nc72v1nqaw19k67221g3k561dwij"))))
> +                "0y5sd247g11jfk5msxy91hz2nhpy7smj125dzfyfhjsjnqk5nyw6"))))

Were these two downgrades intended?  I'm assuming no, since the new
package don't appear to use them.

[...]

> +(define-public intel-xed
> +  (package
> +    (name "intel-xed")
> +    (version "11.2.0")
> +    (source
> +     (origin
> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://github.com/intelxed/xed.git")
> +             (commit "40125558530137444b4ee6fd26b445bfa105b543")))
> +       (sha256 (base32 "1jffayski2gpd54vaska7fmiwnnia8v3cka4nfyzjgl8xsky9v2s"))
> +       (file-name (git-file-name name version))
> +       (patches (search-patches "intel-xed-fix-nondeterminism.patch"))))
> +    (build-system gnu-build-system)
> +    (native-inputs
> +     `(("python-2" ,python-2)
> +       ("python-3" ,python-3)

Does it work to use 'python-wrapper' instead of providing both Python 2
and Python 3 here?

> +    (outputs '("out" "lib" "src"))

Is the src output used for other things than documentation?  If not, I
think we can drop it and let users do 'guix build --source intel-xed'
instead.  The description should be modified accordingly.

Apart from this the package LGTM.  Probably it should have:

  (supported-systems '("x86_64-linux" "i686-linux"))

too?

[...]

> diff --git a/gnu/packages/patches/intel-xed-fix-nondeterminism.patch b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
> new file mode 100644
> index 0000000000..657f7e979d
> --- /dev/null
> +++ b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
> @@ -0,0 +1,100 @@
> +diff --git a/pysrc/ild_codegen.py b/pysrc/ild_codegen.py

Can you add a short description at the top of the patch file explaining
what it does any why?

Can you send an updated patch?  Thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#41574] gnu: Add intel-xev.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 1201 bytes --]

Marius,

Thanks for taking a look at this.

> Were these two downgrades intended?  I'm assuming no, since the new
> package don't appear to use them.

Definitely not. Looks like I was sloppy with a local rebase. Thanks for
catching this.

> Does it work to use 'python-wrapper' instead of providing both Python 2
> and Python 3 here?

Beautiful; 'python-wrapper' is exactly what I was looking for.

> Is the src output used for other things than documentation?  If not, I
> think we can drop it and let users do 'guix build --source intel-xed'
> instead.  The description should be modified accordingly.

Sounds emminently reasonable to me.

> Apart from this the package LGTM.  Probably it should have:
> 
>   (supported-systems '("x86_64-linux" "i686-linux"))
> 
> too?

I'm not so sure, actually. The tool and library simply facilitate translating
to/from x86 opcodes, but as far as I can tell it doesn't actually *execute* any
architecture-specific instructions.

> Can you add a short description at the top of the patch file explaining
> what it does any why?

Oh, neat. I didn't know this was possible.

> Can you send an updated patch?  Thanks!

Done!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-intel-xed.patch --]
[-- Type: text/x-patch, Size: 12533 bytes --]

From a90ef5e79d863201d990d607c2926400654bfd9b Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Thu, 28 May 2020 07:32:28 +0900
Subject: [PATCH] gnu: Add intel-xed.
To: guix-patches@gnu.org

* gnu/packages/assembly.scm (intel-xed): New variable.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/assembly.scm                     |  89 ++++++++++++++
 .../intel-xed-fix-nondeterminism.patch        | 113 ++++++++++++++++++
 3 files changed, 203 insertions(+)
 create mode 100644 gnu/packages/patches/intel-xed-fix-nondeterminism.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1b9fabd2ad..69884b0451 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1104,6 +1104,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ilmbase-fix-test-arm.patch		\
   %D%/packages/patches/inetutils-hurd.patch			\
   %D%/packages/patches/inkscape-poppler-0.76.patch		\
+  %D%/packages/patches/intel-xed-fix-nondeterminism.patch	\
   %D%/packages/patches/intltool-perl-compatibility.patch	\
   %D%/packages/patches/iputils-libcap-compat.patch		\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
diff --git a/gnu/packages/assembly.scm b/gnu/packages/assembly.scm
index c9582bc596..452d458272 100644
--- a/gnu/packages/assembly.scm
+++ b/gnu/packages/assembly.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2019 Andy Tai <atai@atai.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Christopher Lemmer Webber <cwebber@dustycloud.org>
+;;; Copyright © 2020 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -47,6 +48,7 @@
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages python)
   #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages shells)
   #:use-module (gnu packages xml)
   #:use-module ((guix utils)
                 #:select (%current-system)))
@@ -371,3 +373,90 @@ It understands mnemonics and generates code for NMOS 6502s (such
 as 6502A, 6504, 6507, 6510, 7501, 8500, 8501, 8502 ...),
  CMOS 6502s (65C02 and Rockwell R65C02) and the 65816.")
     (license license:gpl2)))
+
+(define-public intel-xed
+  (package
+    (name "intel-xed")
+    (version "11.2.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/intelxed/xed.git")
+             (commit "40125558530137444b4ee6fd26b445bfa105b543")))
+       (sha256 (base32 "1jffayski2gpd54vaska7fmiwnnia8v3cka4nfyzjgl8xsky9v2s"))
+       (file-name (git-file-name name version))
+       (patches (search-patches "intel-xed-fix-nondeterminism.patch"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("python-wrapper" ,python-wrapper)
+       ("tcsh" ,tcsh)
+       ("mbuild"
+        ,(let ((name "mbuild")
+               (version "0.2496"))
+           (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/intelxed/mbuild.git")
+                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
+             (sha256
+              (base32
+               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
+             (file-name (git-file-name name version)))))))
+    (outputs '("out" "lib" "src"))
+    (arguments
+     `(#:phases
+       ;; Upstream uses the custom Python build tool `mbuild', so we munge
+       ;; gnu-build-system to fit.  The build process for this package is
+       ;; documented at https://intelxed.github.io/build-manual/.
+       (let* ((build-dir "build")
+              (kit-dir "kit"))
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (replace 'build
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((mbuild (assoc-ref inputs "mbuild")))
+                 (setenv "PYTHONPATH" (string-append
+                                        (getenv "PYTHONPATH") ":" mbuild))
+                 (invoke "./mfile.py"
+                         (string-append "--build-dir=" build-dir)
+                         (string-append "--install-dir=" kit-dir)
+                         "examples"
+                         "doc"
+                         "install"))))
+           (replace 'check
+             (lambda _
+               ;; Skip broken test group `tests/tests-avx512pf'.
+               (invoke "tests/run-cmd.py"
+                       (string-append "--build-dir=" kit-dir "/bin")
+                       "--tests" "tests/tests-base"
+                       "--tests" "tests/tests-avx512"
+                       "--tests" "tests/tests-cet"
+                       "--tests" "tests/tests-via"
+                       "--tests" "tests/tests-syntax"
+                       "--tests" "tests/tests-xop")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (lib (assoc-ref outputs "lib")))
+                 (copy-recursively (string-append kit-dir "/bin")
+                                   (string-append out "/bin"))
+                 (copy-recursively (string-append kit-dir "/include")
+                                   (string-append lib "/include"))
+                 (copy-recursively (string-append kit-dir "/lib")
+                                   (string-append lib "/lib"))
+                 #t)))))))
+    (home-page "https://intelxed.github.io/")
+    (synopsis "Encoder and decoder for x86 (IA32 and Intel64) instructions")
+    (description "The Intel X86 Encoder Decoder (XED) is a software library and
+for encoding and decoding X86 (IA32 and Intel64) instructions.  The decoder
+takes sequences of 1-15 bytes along with machine mode information and produces
+a data structure describing the opcode, operands, and flags.  The encoder takes
+a similar data structure and produces a sequence of 1 to 15 bytes.  Disassembly
+is essentially a printing pass on the data structure.
+
+The library and development files are under the @code{lib} output, with a
+family of command line utility wrappers in the default output.  Each of the cli
+tools is named like @code{xed*}.  Documentation for the cli tools is sparse, so
+this is a case where ``the code is the documentation.''")
+    (license license:asl2.0)))
diff --git a/gnu/packages/patches/intel-xed-fix-nondeterminism.patch b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
new file mode 100644
index 0000000000..c81bd0edde
--- /dev/null
+++ b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
@@ -0,0 +1,113 @@
+This patch removes sources of build non-determinism in the upstream sources.
+
+In particular, many of the compiled sources are generated with Python code,
+which in turn uses dictionaries to index the output C functions.  However,
+iterators over Python dictionaries have no guaranteed order, thus resulting in
+the C functions being output in a random order between builds.
+
+The patch below fixes this by forcing an order during output in several key
+places.  Note, however, that future updates may uncover new such places that
+just happen to be non-problematic at the time of this patch.  If you are
+reading this due to finding such issues, feel free to contact me at
+elaexuotee@wilsonb.com for help.
+
+diff --git a/pysrc/ild_codegen.py b/pysrc/ild_codegen.py
+index 628ec45..a9bff79 100755
+--- a/pysrc/ild_codegen.py
++++ b/pysrc/ild_codegen.py
+@@ -188,14 +188,14 @@ def gen_l2_func_list(agi, target_nt_dict, arg_nt_dict,
+                      ild_t_member):
+     """generate L2 functions"""
+     l2_func_list = []
+-    for (nt_name,array) in target_nt_dict.items():
++    for (nt_name,array) in sorted(target_nt_dict.items()):
+         target_opname = array.get_target_opname()
+         if array.is_const_lookup_fun():
+             fo = gen_const_l2_function(agi, nt_name,
+                                 target_opname, ild_t_member)
+             l2_func_list.append(fo)
+         else:
+-            for arg_nt_seq,arg_arr in arg_nt_dict.items():
++            for arg_nt_seq,arg_arr in sorted(arg_nt_dict.items()):
+                 fo = gen_scalable_l2_function(agi, nt_name,
+                      target_opname, ild_t_member, arg_arr, list(arg_nt_seq))
+                 l2_func_list.append(fo)
+diff --git a/pysrc/ild_disp.py b/pysrc/ild_disp.py
+index 942c036..cf80e29 100755
+--- a/pysrc/ild_disp.py
++++ b/pysrc/ild_disp.py
+@@ -350,7 +350,8 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     disp_dict = _gen_l3_array_dict(agi, disp_nts, _disp_token)
+ 
+     
+-    nt_arr_list = list(brdisp_dict.values()) + list(disp_dict.values())
++    nt_arr_list = ([v for (k,v) in sorted(brdisp_dict.items())] +
++                   [v for (k,v) in sorted(disp_dict.items())])
+     #create function that calls all initialization functions
+     init_f = ild_nt.gen_init_function(nt_arr_list, 'xed_ild_disp_l3_init')
+     
+@@ -367,7 +368,7 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     l2_functions = []
+     eosz_op = ild_eosz.get_target_opname()
+     easz_op = ild_easz.get_target_opname()
+-    for nt_name,array in list(disp_dict.items()) + list(brdisp_dict.items()):
++    for nt_name,array in sorted(disp_dict.items()) + sorted(brdisp_dict.items()):
+         #Some DISP NTs depend on EOSZ, others on EASZ, we need to know
+         #that when we generate L2 functions
+         if eosz_op in array.get_arg_names():
+diff --git a/pysrc/ild_easz.py b/pysrc/ild_easz.py
+index 02cd691..c53b9f2 100755
+--- a/pysrc/ild_easz.py
++++ b/pysrc/ild_easz.py
+@@ -165,9 +165,10 @@ def work(agi, united_lookup, easz_nts, ild_gendir, debug):
+             return
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()),
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values,
+                                        'xed_ild_easz_init')
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _easz_c_fn, 
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _easz_c_fn, 
+                           mbuild.join('include-private', _easz_header_fn),
+                           init_f)
+     getter_fos = []
+diff --git a/pysrc/ild_eosz.py b/pysrc/ild_eosz.py
+index 6643bc3..89d2d89 100755
+--- a/pysrc/ild_eosz.py
++++ b/pysrc/ild_eosz.py
+@@ -200,10 +200,11 @@ def work(agi, united_lookup, eosz_nts, ild_gendir, debug):
+             return None
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()), 
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values, 
+                                       'xed_ild_eosz_init')
+     #dump init and lookup functions for EOSZ sequences
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _eosz_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _eosz_c_fn,
+                           mbuild.join('include-private', _eosz_header_fn),
+                           init_f)
+     #generate EOSZ getter functions - they get xed_decoded_inst_t*
+diff --git a/pysrc/ild_imm.py b/pysrc/ild_imm.py
+index 51c413c..0530bae 100755
+--- a/pysrc/ild_imm.py
++++ b/pysrc/ild_imm.py
+@@ -322,12 +322,14 @@ def work(agi, united_lookup, imm_nts, ild_gendir, eosz_dict,
+                                      level='l3')
+         nt_dict[nt_name] = array
+ 
++    nt_dict_values = [v for (k,v) in sorted(nt_dict.items())]
++
+     #create function that calls all initialization functions for L3
+-    init_f = ild_nt.gen_init_function(list(nt_dict.values()),
++    init_f = ild_nt.gen_init_function(nt_dict_values,
+                                       'xed_ild_imm_l3_init')
+     
+     #dump L3 functions
+-    ild_nt.dump_lu_arrays(agi, list(nt_dict.values()), _l3_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_dict_values, _l3_c_fn,
+                           mbuild.join('include-private',_l3_header_fn),
+                           init_f)
+     
-- 
2.27.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#41574] gnu: Add intel-xev.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3517 bytes --]

elaexuotee@wilsonb.com writes:

> Marius,
>
> Thanks for taking a look at this.
>
>> Were these two downgrades intended?  I'm assuming no, since the new
>> package don't appear to use them.
>
> Definitely not. Looks like I was sloppy with a local rebase. Thanks for
> catching this.

OK.

>> Does it work to use 'python-wrapper' instead of providing both Python 2
>> and Python 3 here?
>
> Beautiful; 'python-wrapper' is exactly what I was looking for.

Great.

>> Is the src output used for other things than documentation?  If not, I
>> think we can drop it and let users do 'guix build --source intel-xed'
>> instead.  The description should be modified accordingly.
>
> Sounds emminently reasonable to me.

The new patch still has a "src" output, even though it does not seem to
use it.

>> Apart from this the package LGTM.  Probably it should have:
>> 
>>   (supported-systems '("x86_64-linux" "i686-linux"))
>> 
>> too?
>
> I'm not so sure, actually. The tool and library simply facilitate translating
> to/from x86 opcodes, but as far as I can tell it doesn't actually *execute* any
> architecture-specific instructions.

OK.

>> Can you add a short description at the top of the patch file explaining
>> what it does any why?
>
> Oh, neat. I didn't know this was possible.

Nice job on the comment.  :-)

>> Can you send an updated patch?  Thanks!
>
> Done!

Looks like I missed a couple of things in the first round, sorry about
that.  Here it comes...

> From a90ef5e79d863201d990d607c2926400654bfd9b Mon Sep 17 00:00:00 2001
> From: "B. Wilson" <elaexuotee@wilsonb.com>
> Date: Thu, 28 May 2020 07:32:28 +0900
> Subject: [PATCH] gnu: Add intel-xed.
> To: guix-patches@gnu.org
>
> * gnu/packages/assembly.scm (intel-xed): New variable.

Please also mention the new patch and change to local.mk in the commit
message.

[...]

> +(define-public intel-xed
> +  (package
> +    (name "intel-xed")
> +    (version "11.2.0")
> +    (source
> +     (origin
> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://github.com/intelxed/xed.git")
> +             (commit "40125558530137444b4ee6fd26b445bfa105b543")))

Use the "11.2.0" tag here instead of the commit hash.

> +       (sha256 (base32 "1jffayski2gpd54vaska7fmiwnnia8v3cka4nfyzjgl8xsky9v2s"))
> +       (file-name (git-file-name name version))
> +       (patches (search-patches "intel-xed-fix-nondeterminism.patch"))))
> +    (build-system gnu-build-system)
> +    (native-inputs
> +     `(("python-wrapper" ,python-wrapper)
> +       ("tcsh" ,tcsh)
> +       ("mbuild"
> +        ,(let ((name "mbuild")
> +               (version "0.2496"))
> +           (origin
> +             (method git-fetch)
> +             (uri (git-reference
> +                   (url "https://github.com/intelxed/mbuild.git")
> +                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
> +             (sha256
> +              (base32
> +               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
> +             (file-name (git-file-name name version)))))))

Can you add a comment about where you got that version number from?
Also, would it make sense to package this separately?  Can be done later
though.

> +    (outputs '("out" "lib" "src"))

As mentioned before, the 'src' output can go.

Apart from these minor issues, I think it's good to go.  \o/

Can you send an updated patch?  TIA!  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#41574] gnu: Add intel-xev.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Marius Bakke <marius@gnu.org> writes:

>> +(define-public intel-xed
>> +  (package
>> +    (name "intel-xed")
>> +    (version "11.2.0")
>> +    (source
>> +     (origin
>> +       (method git-fetch)
>> +       (uri (git-reference
>> +             (url "https://github.com/intelxed/xed.git")
>> +             (commit "40125558530137444b4ee6fd26b445bfa105b543")))
>
> Use the "11.2.0" tag here instead of the commit hash.

Actually, I meant the 'version' variable here, as in (commit version).

Patch overload!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#41574] gnu: Add intel-xev.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 1169 bytes --]

> Please also mention the new patch and change to local.mk in the commit
> message.

Looks like I also forgot to mention the patch itself. Thanks.

> Use the "11.2.0" tag here instead of the commit hash.

Done. As you mentioned in a follow-up email, I replaced the commit string with
the `version' token which happily happens to match the name of the version tag
we want.

> Can you add a comment about where you got that version number from?
> Also, would it make sense to package this separately?  Can be done later
> though.

I went ahead and added a comment explaining both of these decisions. The reason
for including mbuild in this way instead of packaging it separately is simply
that it seems to be a home-grown build system just for XED (I have no idea
why). There are other projects called "mbuild" and I didn't want to pollute the
package namespace unnecessarily.

> > +    (outputs '("out" "lib" "src"))
> 
> As mentioned before, the 'src' output can go.

Argh. Completely missed that. Nice catch.

> Apart from these minor issues, I think it's good to go.  \o/
> 
> Can you send an updated patch?  TIA!  :-)

Here you go!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-intel-xed.patch --]
[-- Type: text/x-patch, Size: 13009 bytes --]

From aeaae07bbec8e2405ec3be40098b9a6b3fcc0a3b Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Thu, 28 May 2020 07:32:28 +0900
Subject: [PATCH] gnu: Add intel-xed.
To: guix-patches@gnu.org

* gnu/packages/assembly.scm (intel-xed): New variable.
* gnu/packages/patches/intel-xed-fix-nondeterminism.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add reference to new patch.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/assembly.scm                     |  94 +++++++++++++++
 .../intel-xed-fix-nondeterminism.patch        | 113 ++++++++++++++++++
 3 files changed, 208 insertions(+)
 create mode 100644 gnu/packages/patches/intel-xed-fix-nondeterminism.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index a321575799..c70ee59b8f 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1107,6 +1107,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ilmbase-fix-test-arm.patch		\
   %D%/packages/patches/inetutils-hurd.patch			\
   %D%/packages/patches/inkscape-poppler-0.76.patch		\
+  %D%/packages/patches/intel-xed-fix-nondeterminism.patch	\
   %D%/packages/patches/intltool-perl-compatibility.patch	\
   %D%/packages/patches/iputils-libcap-compat.patch		\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
diff --git a/gnu/packages/assembly.scm b/gnu/packages/assembly.scm
index c9582bc596..39e14d452c 100644
--- a/gnu/packages/assembly.scm
+++ b/gnu/packages/assembly.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2019 Andy Tai <atai@atai.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Christopher Lemmer Webber <cwebber@dustycloud.org>
+;;; Copyright © 2020 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -47,6 +48,7 @@
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages python)
   #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages shells)
   #:use-module (gnu packages xml)
   #:use-module ((guix utils)
                 #:select (%current-system)))
@@ -371,3 +373,95 @@ It understands mnemonics and generates code for NMOS 6502s (such
 as 6502A, 6504, 6507, 6510, 7501, 8500, 8501, 8502 ...),
  CMOS 6502s (65C02 and Rockwell R65C02) and the 65816.")
     (license license:gpl2)))
+
+(define-public intel-xed
+  (package
+    (name "intel-xed")
+    (version "11.2.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/intelxed/xed.git")
+             (commit version)))
+       (sha256 (base32 "1jffayski2gpd54vaska7fmiwnnia8v3cka4nfyzjgl8xsky9v2s"))
+       (file-name (git-file-name name version))
+       (patches (search-patches "intel-xed-fix-nondeterminism.patch"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("python-wrapper" ,python-wrapper)
+       ("tcsh" ,tcsh)
+       ;; As of the time of writing this comment, mbuild does not exist in the
+       ;; Python Package Index and seems to only be used by intel-xed, so we
+       ;; opt to include it here instead of packaging separately.  Note also
+       ;; that the git repository contains no version tags, so we directly
+       ;; reference the "version" variable from setup.py instead.
+       ("mbuild"
+        ,(let ((name "mbuild")
+               (version "0.2496"))
+           (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/intelxed/mbuild.git")
+                   (commit "5304b94361fccd830c0e2417535a866b79c1c297")))
+             (sha256
+              (base32
+               "0r3avc3035aklqxcnc14rlmmwpj3jp09vbcbwynhvvmcp8srl7dl"))
+             (file-name (git-file-name name version)))))))
+    (outputs '("out" "lib"))
+    (arguments
+     `(#:phases
+       ;; Upstream uses the custom Python build tool `mbuild', so we munge
+       ;; gnu-build-system to fit.  The build process for this package is
+       ;; documented at https://intelxed.github.io/build-manual/.
+       (let* ((build-dir "build")
+              (kit-dir "kit"))
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (replace 'build
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((mbuild (assoc-ref inputs "mbuild")))
+                 (setenv "PYTHONPATH" (string-append
+                                       (getenv "PYTHONPATH") ":" mbuild))
+                 (invoke "./mfile.py"
+                         (string-append "--build-dir=" build-dir)
+                         (string-append "--install-dir=" kit-dir)
+                         "examples"
+                         "doc"
+                         "install"))))
+           (replace 'check
+             (lambda _
+               ;; Skip broken test group `tests/tests-avx512pf'.
+               (invoke "tests/run-cmd.py"
+                       (string-append "--build-dir=" kit-dir "/bin")
+                       "--tests" "tests/tests-base"
+                       "--tests" "tests/tests-avx512"
+                       "--tests" "tests/tests-cet"
+                       "--tests" "tests/tests-via"
+                       "--tests" "tests/tests-syntax"
+                       "--tests" "tests/tests-xop")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (lib (assoc-ref outputs "lib")))
+                 (copy-recursively (string-append kit-dir "/bin")
+                                   (string-append out "/bin"))
+                 (copy-recursively (string-append kit-dir "/include")
+                                   (string-append lib "/include"))
+                 (copy-recursively (string-append kit-dir "/lib")
+                                   (string-append lib "/lib"))
+                 #t)))))))
+    (home-page "https://intelxed.github.io/")
+    (synopsis "Encoder and decoder for x86 (IA32 and Intel64) instructions")
+    (description "The Intel X86 Encoder Decoder (XED) is a software library and
+for encoding and decoding X86 (IA32 and Intel64) instructions.  The decoder
+takes sequences of 1-15 bytes along with machine mode information and produces
+a data structure describing the opcode, operands, and flags.  The encoder takes
+a similar data structure and produces a sequence of 1 to 15 bytes.  Disassembly
+is essentially a printing pass on the data structure.
+
+The library and development files are under the @code{lib} output, with a
+family of command line utility wrappers in the default output.  Each of the cli
+tools is named like @code{xed*}.  Documentation for the cli tools is sparse, so
+this is a case where ``the code is the documentation.''")
+    (license license:asl2.0)))
diff --git a/gnu/packages/patches/intel-xed-fix-nondeterminism.patch b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
new file mode 100644
index 0000000000..c81bd0edde
--- /dev/null
+++ b/gnu/packages/patches/intel-xed-fix-nondeterminism.patch
@@ -0,0 +1,113 @@
+This patch removes sources of build non-determinism in the upstream sources.
+
+In particular, many of the compiled sources are generated with Python code,
+which in turn uses dictionaries to index the output C functions.  However,
+iterators over Python dictionaries have no guaranteed order, thus resulting in
+the C functions being output in a random order between builds.
+
+The patch below fixes this by forcing an order during output in several key
+places.  Note, however, that future updates may uncover new such places that
+just happen to be non-problematic at the time of this patch.  If you are
+reading this due to finding such issues, feel free to contact me at
+elaexuotee@wilsonb.com for help.
+
+diff --git a/pysrc/ild_codegen.py b/pysrc/ild_codegen.py
+index 628ec45..a9bff79 100755
+--- a/pysrc/ild_codegen.py
++++ b/pysrc/ild_codegen.py
+@@ -188,14 +188,14 @@ def gen_l2_func_list(agi, target_nt_dict, arg_nt_dict,
+                      ild_t_member):
+     """generate L2 functions"""
+     l2_func_list = []
+-    for (nt_name,array) in target_nt_dict.items():
++    for (nt_name,array) in sorted(target_nt_dict.items()):
+         target_opname = array.get_target_opname()
+         if array.is_const_lookup_fun():
+             fo = gen_const_l2_function(agi, nt_name,
+                                 target_opname, ild_t_member)
+             l2_func_list.append(fo)
+         else:
+-            for arg_nt_seq,arg_arr in arg_nt_dict.items():
++            for arg_nt_seq,arg_arr in sorted(arg_nt_dict.items()):
+                 fo = gen_scalable_l2_function(agi, nt_name,
+                      target_opname, ild_t_member, arg_arr, list(arg_nt_seq))
+                 l2_func_list.append(fo)
+diff --git a/pysrc/ild_disp.py b/pysrc/ild_disp.py
+index 942c036..cf80e29 100755
+--- a/pysrc/ild_disp.py
++++ b/pysrc/ild_disp.py
+@@ -350,7 +350,8 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     disp_dict = _gen_l3_array_dict(agi, disp_nts, _disp_token)
+ 
+     
+-    nt_arr_list = list(brdisp_dict.values()) + list(disp_dict.values())
++    nt_arr_list = ([v for (k,v) in sorted(brdisp_dict.items())] +
++                   [v for (k,v) in sorted(disp_dict.items())])
+     #create function that calls all initialization functions
+     init_f = ild_nt.gen_init_function(nt_arr_list, 'xed_ild_disp_l3_init')
+     
+@@ -367,7 +368,7 @@ def work(agi, united_lookup,  disp_nts, brdisp_nts, ild_gendir,
+     l2_functions = []
+     eosz_op = ild_eosz.get_target_opname()
+     easz_op = ild_easz.get_target_opname()
+-    for nt_name,array in list(disp_dict.items()) + list(brdisp_dict.items()):
++    for nt_name,array in sorted(disp_dict.items()) + sorted(brdisp_dict.items()):
+         #Some DISP NTs depend on EOSZ, others on EASZ, we need to know
+         #that when we generate L2 functions
+         if eosz_op in array.get_arg_names():
+diff --git a/pysrc/ild_easz.py b/pysrc/ild_easz.py
+index 02cd691..c53b9f2 100755
+--- a/pysrc/ild_easz.py
++++ b/pysrc/ild_easz.py
+@@ -165,9 +165,10 @@ def work(agi, united_lookup, easz_nts, ild_gendir, debug):
+             return
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()),
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values,
+                                        'xed_ild_easz_init')
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _easz_c_fn, 
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _easz_c_fn, 
+                           mbuild.join('include-private', _easz_header_fn),
+                           init_f)
+     getter_fos = []
+diff --git a/pysrc/ild_eosz.py b/pysrc/ild_eosz.py
+index 6643bc3..89d2d89 100755
+--- a/pysrc/ild_eosz.py
++++ b/pysrc/ild_eosz.py
+@@ -200,10 +200,11 @@ def work(agi, united_lookup, eosz_nts, ild_gendir, debug):
+             return None
+         nt_seq_arrays[tuple(nt_seq)] = array
+     #init function calls all single init functions for the created tables
+-    init_f = ild_nt.gen_init_function(list(nt_seq_arrays.values()), 
++    nt_seq_values = [v for (k,v) in sorted(nt_seq_arrays.items())]
++    init_f = ild_nt.gen_init_function(nt_seq_values, 
+                                       'xed_ild_eosz_init')
+     #dump init and lookup functions for EOSZ sequences
+-    ild_nt.dump_lu_arrays(agi, list(nt_seq_arrays.values()), _eosz_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_seq_values, _eosz_c_fn,
+                           mbuild.join('include-private', _eosz_header_fn),
+                           init_f)
+     #generate EOSZ getter functions - they get xed_decoded_inst_t*
+diff --git a/pysrc/ild_imm.py b/pysrc/ild_imm.py
+index 51c413c..0530bae 100755
+--- a/pysrc/ild_imm.py
++++ b/pysrc/ild_imm.py
+@@ -322,12 +322,14 @@ def work(agi, united_lookup, imm_nts, ild_gendir, eosz_dict,
+                                      level='l3')
+         nt_dict[nt_name] = array
+ 
++    nt_dict_values = [v for (k,v) in sorted(nt_dict.items())]
++
+     #create function that calls all initialization functions for L3
+-    init_f = ild_nt.gen_init_function(list(nt_dict.values()),
++    init_f = ild_nt.gen_init_function(nt_dict_values,
+                                       'xed_ild_imm_l3_init')
+     
+     #dump L3 functions
+-    ild_nt.dump_lu_arrays(agi, list(nt_dict.values()), _l3_c_fn,
++    ild_nt.dump_lu_arrays(agi, nt_dict_values, _l3_c_fn,
+                           mbuild.join('include-private',_l3_header_fn),
+                           init_f)
+     
-- 
2.27.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#41574: gnu: Add intel-xev.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

elaexuotee@wilsonb.com writes:

>> Apart from these minor issues, I think it's good to go.  \o/
>> 
>> Can you send an updated patch?  TIA!  :-)
>
> Here you go!

Sorry for the delay, this patch got lost in my horrifying email queue.

Applied now!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#41574] gnu: Add intel-xev.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1.1: Type: text/plain, Size: 117 bytes --]

> Sorry for the delay, this patch got lost in my horrifying email queue.
> 
> Applied now!

No worries. Thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]