[bug#48463] gnu: Add j.

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

Hey Guix,

This packages up the J programming lanuage. The build scripts that upstream
provides are pretty hostile to package maintainers, so we have to jump through
quite a few hoops to get the build working.

I have had this in my personal, private channel for a while, tweaking it over
time, so I believe it works fairly well in practice. Here are the high-level
design decisions I have made:

1) Provide multiple J versions;

It is extremely common for J users to have multiple versions installed in
parallel since there are often major, breaking changes between
(non-patch-level) releases.

In addition, a very large fraction of J users keep the beta installed as well,
so I have elected to provide this too.

2) Bundle together all variants of the interpreter along with a
wrapper script that selects the most featureful one at runtime;

There are 3 versions of the interpreter, supporting different CPU feature-sets:
vanilla, AVX, AVX2. This package elects to build all these variants and
provides a wrapper script that launches the "most AVX-y" version possible at
runtime.

Essentially, this is trying to make up for upstream's lack of a fat binary.

3) Try to mirror the FHS standard for installation locations;

For the most part, J uses idiosyncratic defaults for where it looks for library
files and the like. This package elects to configure things so files sit in
more standard locations---e.g. runtime scripts sit under .../share/j;
configuration files sit under .../etc/j; etc.

4) Point the J package manager to $HOME/.j/<version>.

J maintains its own mini package repository for "addons", which are simply
officially-supported J scripts. This package sets things up so that these
packages get installed to .j/<version>/addons. In addition, other related
directories are gathered under .j/<version>.


Items on the TODO list:

1) Write an importer and create packages for J addons;

This is currently in progress.

2) Create a something like a jsoftware-union procedure to bundle up J with
user-selectable addons;

This is necessary since the interpreter looks for addons at a specified path.
We probably want to mimick something like texlive and provide both an "all
batteries included" package along with one where users can select which addons
they want.

3) Support more older versions of J.

J is open source starting with j801, so it'd be nice to have packages for the
entire j80x series too. Unfortunately, those use an entirely different set of
build scripts from the current j90x series, so it will require munging-work
similar to what we already have.


Cheers!



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 27077 bytes --]

From c231924ab9a72979001640d866db1988f8fd3bde Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Sun, 16 May 2021 19:02:29 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: (GNU_SYSTEM_MODULES, dist_patch_DATA): Add them.
---
 gnu/local.mk                                  |   2 +
 gnu/packages/jsoftware.scm                    | 432 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 ++++
 3 files changed, 514 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index c3b0274945..b706ec4e86 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -305,6 +305,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/javascript.scm			\
   %D%/packages/jemalloc.scm			\
   %D%/packages/jrnl.scm				\
+  %D%/packages/jsoftware.scm			\
   %D%/packages/jose.scm				\
   %D%/packages/julia.scm			\
   %D%/packages/julia-xyz.scm			\
@@ -1270,6 +1271,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/jfsutils-add-sysmacros.patch		\
   %D%/packages/patches/jfsutils-include-systypes.patch		\
   %D%/packages/patches/jsoncpp-fix-inverted-case.patch		\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch	\
   %D%/packages/patches/kdbusaddons-kinit-file-name.patch	\
   %D%/packages/patches/libffi-3.3-powerpc-fixes.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..f6f6099e42
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,432 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix records)
+  #:use-module (guix packages)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix git-download)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix build utils)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 match))
+
+
+(define (jname prefix jtype)
+  "Return a package name for the given prefix and jtype, e.g. `jlib',
+  `jlib-beta', `j', and `j-beta'."
+  (let ((postfix (if (eq? jtype 'release) ""
+                   (string-append "-" (symbol->string jtype)))))
+    (string-append prefix postfix)))
+
+(define (jversion->string version revision)
+  "Return a string representation of a J version and (optional) revision pair."
+  (let ((postfix (if (not revision) ""
+                   (string-append "-" revision))))
+    (string-append version postfix)))
+
+(define (jinfo->git-tag jversion jtype jrevision)
+  "Given version parameters, construct a git tag for upstream releases."
+  (let ((postfix (if (not jrevision) ""
+                   (string-append "-" jrevision))))
+    (string-append "j" jversion "-" (symbol->string jtype) postfix)))
+
+(define-record-type* <jlib-build-configuration>
+  jlib-build-configuration make-jlib-build-configuration
+  jlib-build-configuration?
+  (builder      jlib-build-configuration-builder (default "guix.gnu.org"))
+  (version      jlib-build-configuration-version (default #f))
+  (revision     jlib-build-configuration-revision (default #f))
+  (hash         jlib-build-configuration-hash (default #f))
+  (type         jlib-build-configuration-type (default 'release))
+  (patches      jlib-build-configuration-patches (default '()))
+  (extra-inputs jlib-build-configuration-extra-inputs (default '()))
+  (extra-envars jlib-build-configuration-extra-envars (default '())))
+
+(define make-jlib
+  (match-lambda
+    (($ <jlib-build-configuration>
+        builder jversion revision hash type patches extra-inputs extra-envars)
+     (package
+       (name (jname "jlib" type))
+       (version (jversion->string jversion revision))
+       (source
+        (origin
+          (method git-fetch)
+          (uri (git-reference
+                (url "https://github.com/jsoftware/jsource")
+                (commit (jinfo->git-tag jversion type revision))))
+          (sha256 (base32 hash))
+          (file-name (git-file-name name version))
+          (patches patches)))
+       (properties `((jversion . ,jversion)
+                     (jrevision . ,revision)
+                     (jtype . ,type)))
+       (build-system gnu-build-system)
+       (native-inputs `(("clang-toolchain" ,clang-toolchain)))
+       (inputs (cons* `("libedit" ,libedit)
+                      `("libomp" ,libomp)
+                      extra-inputs))
+       (arguments
+        `(#:modules (((ice-9 ftw) #:select (scandir))
+                     ((ice-9 popen) #:select (open-pipe* close-pipe))
+                     ((ice-9 regex) #:select (match:substring string-match))
+                     ((ice-9 threads) #:select (parallel par-for-each))
+                     ((srfi srfi-26) #:select (cut))
+                     ((srfi srfi-1) #:select (fold))
+                     ,@%gnu-build-system-modules)
+          #:phases
+          ;; Upstream's build system consists of ad-hoc scripts that build
+          ;; build up (very complicated) environment variables to pass to make.
+          ;; The basic build process looks like this:
+          ;;
+          ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+          ;;   2) Set jplatform and j64x environment variables;
+          ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+          ;;
+          ;; However, upstream expects users to run J directly from the source
+          ;; directory; they do not supply a make `install' target.  Thus it
+          ;; takes some massaging to install files in FHS-style directories.
+          (modify-phases %standard-phases
+            ;; In particular, we have to set up
+            ;;
+            ;;   1) jsrc/jversion.h as in a typical build;
+            ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+            ;;   3) make2/build_*.sh to respect standard build conventions;
+            ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+            ;;   5) Hard coded references to addons directory.
+            (replace 'configure
+              (lambda* (#:key target inputs outputs #:allow-other-keys)
+                (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                       (clang (string-append clang-toolchain "/bin/clang"))
+                       (libedit (assoc-ref inputs "libedit"))
+                       (out (assoc-ref outputs "out"))
+                       (share (string-append out "/share/j")))
+                  ;; Set up build constants
+                  (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+                  (substitute* "jsrc/jversion.h"
+                    (("^#define jversion.*$")
+                     (format #f "#define jversion ~s\n" ,version))
+                    (("^#define jtype.*$")
+                     (format #f "#define jtype ~s\n" (symbol->string ',type)))
+                    (("^#define jbuilder.*$")
+                     (format #f "#define jbuilder ~s\n" ,builder)))
+                  ;; Create profilex.ijs overrides to point to the correct
+                  ;; store items.  Note that we set ~install and ~addons
+                  ;; directories to reside under ~user to allow installing
+                  ;; and loading addons.  TODO: Guix-ify J addons as well.
+                  (call-with-output-file "jlibrary/bin/profilex.ijs"
+                    (lambda (port)
+                      (display
+                        (string-join
+                          (list
+                            (format #f "system=. '~a/system'" share)
+                            (format #f "tools=. '~a/tools'" share)
+                            ;; Upstream defaults to spamming $HOME with
+                            ;; unhidden userdata directories.  Set this to be
+                            ;; $HOME/.j/<jtype>/<jversion> instead
+                            "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                            "jversion=. ({.~ i.&'-') jversion"
+                            "jsuffix=. >@{&('';'-beta') jtype -: 'beta'"
+                            "user=. home,'/.j/',jversion,jsuffix"
+                            "addons=. user,'/addons'"
+                            "break=. user,'/break'"
+                            "config=. user,'/config'"
+                            "install=. user,'/install'"
+                            "snap=. user,'/snap'"
+                            "temp=. user,'/temp'"
+                            "\n")
+                          "\n")
+                        port)))
+                  ;; Munge the build scripts into reason:
+                  ;; 1. Short-circuit the fragile compiler detection;
+                  ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+                  ;; 3. Propagate script errors to top level.
+                  (for-each
+                   (lambda (file)
+                     (with-directory-excursion "make2"
+                       (substitute* file
+                         ;; The `compiler' variable doesn't point to the actual
+                         ;; compiler.  It is just a switch to tell the build
+                         ;; scripts whether to use gcc- or clang-specific
+                         ;; flags.
+                         (("^compiler=.*$") "compiler=clang\n")
+                         (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                         (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                          (string-append def "$CFLAGS " rest))
+                         (("^#!.*" shebang)
+                          (string-append shebang "set -o errexit\n")))))
+                     '("build_jconsole.sh" "build_libj.sh"))
+                  ;; The jconsole manually loads libedit with dlopen.  The path
+                  ;; must be absolute to correctly point to our input.
+                  (substitute* "jsrc/jconsole.c"
+                    (("libedit\\.so\\.[0-9]" so-file)
+                     (format #f "~a/lib/~a" libedit so-file)))
+                  ;; The ~addons/dev directory supplies tentative J-script
+                  ;; definitions of new J engine functionality.  Since we point
+                  ;; ~addons under the ~user directory, we move it under
+                  ;; ~system instead, which sits as-is in the output.
+                  (with-directory-excursion "jsrc"
+                    (for-each
+                      (lambda (file)
+                        (substitute* file (("~addons/dev") "~system/dev")))
+                      (scandir "."
+                        (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+                  ;; Implementation of 9!:14 records build time which breaks
+                  ;; build reproducibility.  Note that upstream code depends on
+                  ;; the exact format of these strings, so we need to mimic the
+                  ;; standard.
+                  (substitute* "jsrc/j.c"
+                    (("__DATE__") "\"Jan 01 1970\"")
+                    (("__TIME__") "\"00:00:00\""))
+                  ;; Upstream recommends using clang, with GCC support being
+                  ;; second-class, often resulting in build failures.
+                  (setenv "CC" clang))))
+            ;; The build output depends primarily on the values of the
+            ;; `jplatform' and `j64x' environment variables.  If the target is
+            ;; ARM, then `jplatform' is "raspberry", otherwise it is `linux'.
+            ;; In addition to 32- and 64- bit versions, `j64x' controlls
+            ;; whether AVX or AVX2 variants of libj are built.
+            ;;
+            ;; However, build targets are not fine-grained enough to
+            ;; distinguish between CPU features.  Thus we build and install all
+            ;; variants of libj, expecting jconsole to be called with a wrapper
+            ;; script that detects AVX features and loads the appropriate libj
+            ;; at runtime.
+            (replace 'build
+              (lambda _
+                (setenv "USE_OPENMP" "1")
+                (setenv "USE_THREAD" "1")
+                (for-each (lambda (var-val) (apply setenv var-val))
+                          (quote ,extra-envars))
+                ;; The build scripts assume that PWD is make2.
+                (with-directory-excursion "make2"
+                  (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                         (jplat (string-append "jplatform=" platform))
+                         (target-bit ,(if (target-64bit?) "64" "32"))
+                         (jbit (string-append "j64x=" "j" target-bit))
+                         (jbit-avx (string-append jbit "avx"))
+                         (jbit-avx2 (string-append jbit "avx2")))
+                    (parallel
+                      ;; Since jconsole doesn't depend on AVX features, we just
+                      ;; build it once.
+                      (invoke "env" jplat jbit "./build_jconsole.sh")
+                      (invoke "env" jplat jbit "./build_libj.sh")
+                      (if ,(target-64bit?)
+                        (parallel
+                          (invoke "env" jplat jbit-avx "./build_libj.sh")
+                          (invoke "env" jplat jbit-avx2
+                                  "./build_libj.sh"))))))))
+            ;; The test suite is expected to be run as follows for each variant
+            ;; of libj that we build:
+            ;;
+            ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+            ;;
+            ;; This requires a working jconsole with accessible jlibrary files.
+            ;; We simply place these all under test/bin.
+            (replace 'check
+               (lambda _
+                 (let ((jplatform ,(if (target-arm?) "raspberry" "linux")))
+                   (mkdir-p "test/bin")
+                   (for-each
+                     (lambda (dir)
+                       (let ((source (string-append "jlibrary/" dir))
+                             (dest (string-append "test/bin/" dir)))
+                       (begin
+                         (mkdir-p dest)
+                         (copy-recursively source dest))))
+                     '("system" "tools" "addons"))
+                   ;; The jlibrary/dev directory only sometimes exists, but
+                   ;; needs to be copied into the ~system directory when it
+                   ;; does.
+                   (if (access? "jlibrary/dev" R_OK)
+                     (copy-recursively "jlibrary/dev" "test/bin/system/dev"))
+                   (par-for-each
+                     (lambda (dir)
+                       (let* ((jbin (string-append "bin/" jplatform))
+                              (jbit ,(if (target-64bit?) "j64" "j32"))
+                              (jconsole (string-append jbin "/" jbit
+                                                       "/jconsole"))
+                              (source (string-append jbin "/" dir))
+                              (dest (string-append "test/bin/" dir)))
+                         (begin
+                           (mkdir-p dest)
+                           (copy-recursively source dest)
+                           (install-file "jlibrary/bin/profile.ijs" dest)
+                           (install-file jconsole dest)
+                           (let* ((jc (string-append dest "/jconsole"))
+                                  (tests "test/tsu.ijs")
+                                  (port (open-pipe* OPEN_WRITE jc tests)))
+                             (display "RUN ddall\n" port)
+                             (when (not (zero? (status:exit-val
+                                                 (close-pipe port))))
+                               (error "Some J build tests failed."))))))
+                     (scandir (string-append "bin/" jplatform)
+                              (negate (cut member <> '("." "..")))))
+                   #t)))
+            ;; Now that everything is built, installation is fairly
+            ;; straightforward, following FHS conventions.  The only quirk is
+            ;; that we install jconsole under /libexec to make room for the
+            ;; wrapper replacement under /bin.
+            (replace 'install
+              (lambda* (#:key outputs inputs #:allow-other-keys)
+                (let* ((jplat ,(if (target-arm?) "raspberry" "linux"))
+                       (jbit ,(if (target-64bit?) "j64" "j32"))
+                       (jconsole (string-join
+                                   `("bin" ,jplat ,jbit "jconsole") "/"))
+                       (out (assoc-ref outputs "out"))
+                       (etc (string-append out "/etc/j"))
+                       (lib (string-append out "/lib/j"))
+                       (libexec (string-append out "/libexec"))
+                       (share (string-append out "/share/j"))
+                       (system (string-append share "/system"))
+                       (dev (string-append system "/dev")))
+                  (mkdir-p lib)
+                  (for-each
+                    (lambda (jarch)
+                      (let* ((jbin (string-join `("bin" ,jplat ,jarch) "/"))
+                             (javx-match (string-match "avx.*" jarch))
+                             (javx (if (not javx-match) ""
+                                     (match:substring javx-match)))
+                             (sep (if javx-match "-" ""))
+                             (source (string-append jbin "/libj.so"))
+                             (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                        (copy-file source dest)))
+                    (scandir (string-append "bin/" jplat)
+                             (negate (cut member <> '("." "..")))))
+                  (install-file jconsole libexec)
+                  (copy-recursively "jlibrary/system" system)
+                  (if (access? "jlibrary/dev" R_OK)
+                    (copy-recursively "jlibrary/dev" dev))
+                  (install-file "jlibrary/bin/profile.ijs" etc)
+                  (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+       (home-page "https://www.jsoftware.com/")
+       (synopsis "Ascii-only, array programming language in the APL family")
+       (description
+     "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data. It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+       (license license:gpl3+)))))
+
+
+(define jlib-901
+  (make-jlib
+    (jlib-build-configuration
+      (version "901")
+      (revision "f")
+      (hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p")
+      (patches (search-patches "jsoftware-j901-f-fixes.patch")))))
+
+
+(define jlib-build-configuration-with-sleef
+  (jlib-build-configuration
+    (extra-inputs `(("sleef" ,sleef)))
+    (extra-envars `(("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef")))))
+
+(define jlib-902
+  (make-jlib
+    (jlib-build-configuration
+      (inherit jlib-build-configuration-with-sleef)
+      (version "902")
+      (revision "b")
+      (hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+(define jlib-903-beta
+  (make-jlib
+    (jlib-build-configuration
+      (inherit jlib-build-configuration-with-sleef)
+      (version "903")
+      (revision "h")
+      (type 'beta)
+      (hash "13sw2ffgx6pm699qfi50drlp8y1vmzj4swzx82g6vhyjpjci1w4h"))))
+
+
+(define (make-ijconsole jlib)
+  "Generate a G-exp wrapper script that detects AVX/AVX2 support at runtime and
+  executes jconsole with the appropriate libj.so and profile.ijs."
+  (program-file (string-append "ijconsole-" (package-version jlib))
+    #~(begin
+        (use-modules ((ice-9 rdelim) #:select (read-line))
+                     ((ice-9 regex) #:select (regexp-match? string-match)))
+
+        (define (cpu-feature-line? string)
+          (string-prefix? "flags" string))
+
+        (define (contains-word? word string)
+          (regexp-match?
+            (string-match (string-join `("\\<" ,word "\\>") "")
+                          string)))
+
+        (define (has-cpu-feature? feature)
+          (with-input-from-file "/proc/cpuinfo"
+             (lambda ()
+               (catch 'found
+                 (lambda ()
+                   (let loop ((line (read-line)))
+                     (cond ((eof-object? line) #f)
+                           ((and (cpu-feature-line? line)
+                                 (contains-word? feature line))
+                            (throw 'found))
+                           (else (loop (read-line))))))
+                 (const #t)))))
+
+        (let* ((jconsole (string-append #$jlib "/libexec/jconsole"))
+               (libj (format #f "~a/lib/j/libj-~a.so" #$jlib
+                             (cond ((has-cpu-feature? "avx2") "avx2")
+                                   ((has-cpu-feature? "avx") "avx")
+                                   (else ""))))
+               (jprofile (string-append #$jlib "/etc/j/profile.ijs")))
+          (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                 (cdr (command-line)))))))
+
+(define (make-j jlib)
+  "Generate a J package containing /bin/ijconsole-<jversion> linked against the
+  given jlib library package.  Note that the ijconsole executable is versioned
+  since it is typical for J users to have multiple versions installed on the
+  same system."
+  (let ((jversion (assq-ref (package-properties jlib) 'jversion))
+        (jtype (assq-ref (package-properties jlib) 'jtype)))
+    (package (inherit jlib)
+      (name (jname "j" jtype))
+      (source #f)
+      (inputs `(("jlib" ,jlib)
+                ("ijconsole" ,(make-ijconsole jlib))))
+      (build-system trivial-build-system)
+      (arguments
+       `(#:modules ((guix build utils))
+         #:builder
+         (begin
+           (use-modules (guix build utils))
+           (let* ((ijconsole (assoc-ref %build-inputs "ijconsole"))
+                  (out (assoc-ref %outputs "out"))
+                  (bin (string-append out "/bin")))
+             (mkdir-p bin)
+             (symlink ijconsole
+                      (string-append bin "/ijconsole-" ,jversion)))))))))
+
+(define-public j-901 (make-j jlib-901))
+(define-public j-902 (make-j jlib-902))
+(define-public j-903-beta (make-j jlib-903-beta))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.31.1

[bug#48463] gnu: Add j.

[Leo Prikler]

Hi elaexuotee,

Am Sonntag, den 16.05.2021, 19:54 +0900 schrieb elaexuotee@wilsonb.com:
> Hey Guix,
> 
> This packages up the J programming lanuage. The build scripts that
> upstream provides are pretty hostile to package maintainers, so we
> have to jump through quite a few hoops to get the build working.
Yup, that much is visible from the package description.

> I have had this in my personal, private channel for a while, tweaking
> it over time, so I believe it works fairly well in practice. Here are
> the high-level design decisions I have made:
> 
> 1) Provide multiple J versions;
> 
> It is extremely common for J users to have multiple versions
> installed in parallel since there are often major, breaking changes
> between (non-patch-level) releases.
> 
> In addition, a very large fraction of J users keep the beta installed
> as well, so I have elected to provide this too.
That should be fine, we provide multiple versions for other packages
where applicable as well.  Do note, that we should try to keep the
number limited, though.  You may perhaps want to export make-jlib, so
that the user can build their own.

> 2) Bundle together all variants of the interpreter along with a
> wrapper script that selects the most featureful one at runtime;
> 
> There are 3 versions of the interpreter, supporting different CPU
> feature-sets: vanilla, AVX, AVX2. This package elects to build all
> these variants and provides a wrapper script that launches the "most
> AVX-y" version possible at runtime.
> 
> Essentially, this is trying to make up for upstream's lack of a fat
> binary.
Are fat binaries the accepted HPC way?  I'm not up-to-date to this.

> 3) Try to mirror the FHS standard for installation locations;
> 
> For the most part, J uses idiosyncratic defaults for where it looks
> for library files and the like. This package elects to configure
> things so files sit in more standard locations---e.g. runtime scripts
> sit under .../share/j; configuration files sit under .../etc/j; etc.
> 
> 4) Point the J package manager to $HOME/.j/<version>.
> 
> J maintains its own mini package repository for "addons", which are
> simply officially-supported J scripts. This package sets things up so
> that these packages get installed to .j/<version>/addons. In
> addition, other related directories are gathered under .j/<version>.
Those should be fine.

> Items on the TODO list:
> 
> 1) Write an importer and create packages for J addons;
> 
> This is currently in progress.
> 
> 2) Create a something like a jsoftware-union procedure to bundle up J
> with user-selectable addons;
> 
> This is necessary since the interpreter looks for addons at a
> specified path. We probably want to mimick something like texlive and
> provide both an "all batteries included" package along with one where
> users can select which addons they want.
Both sound like good ideas, although I'm a little less sure about the
texlive one.  Would a meta-package like gnome also be acceptable?

> 3) Support more older versions of J.
> 
> J is open source starting with j801, so it'd be nice to have packages
> for the entire j80x series too. Unfortunately, those use an entirely
> different set of build scripts from the current j90x series, so it
> will require munging-work similar to what we already have.
As above, I'd prefer if it was one procedure and one package pointing
to the latest j80x, assuming all of our added patches can also be
applied to earlier versions.

> +(define-record-type* <jlib-build-configuration>
> +  jlib-build-configuration make-jlib-build-configuration
> +  jlib-build-configuration?
> +  (builder      jlib-build-configuration-builder (default
> "guix.gnu.org"))
> +  (version      jlib-build-configuration-version (default #f))
> +  (revision     jlib-build-configuration-revision (default #f))
> +  (hash         jlib-build-configuration-hash (default #f))
> +  (type         jlib-build-configuration-type (default 'release))
> +  (patches      jlib-build-configuration-patches (default '()))
> +  (extra-inputs jlib-build-configuration-extra-inputs (default '()))
> +  (extra-envars jlib-build-configuration-extra-envars (default
> '())))
> +
> +(define make-jlib
> +  (match-lambda
> +    (($ <jlib-build-configuration>
> +        builder jversion revision hash type patches extra-inputs
> extra-envars)
Please try to use keyword arguments.

> +       (properties `((jversion . ,jversion)
> +                     (jrevision . ,revision)
> +                     (jtype . ,type)))
Are those used anywhere?  Can jversion and jrevision not be parsed from
(package-version jlib)?

Regards,
Leo

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

Thanks for taking a look!

Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> That should be fine, we provide multiple versions for other packages
> where applicable as well.  Do note, that we should try to keep the
> number limited, though.  You may perhaps want to export make-jlib, so
> that the user can build their own.

Cool. I like the idea of exporting the build procedures.

> Are fat binaries the accepted HPC way?  I'm not up-to-date to this.

I believe so? Don't quote me on that though. For this particular package the
overhoad is just ~8MB, so I don't think it's much of an issue either way. It
would probably be more of a hassle to split j into a package for each variant.

> Both sound like good ideas, although I'm a little less sure about the
> texlive one.  Would a meta-package like gnome also be acceptable?

A metapackage is certainly practical. The entire set of J addons currently
weighs in at a whopping 50MB. hehe. However, there are about 125 addons in
total, so it simply feels "more correct" to let the user also choose just the
addons they want, which I think would require something like jsoftware-union.

That said, either way, an "all batteries include" package would be good to
have, and since this is way easier than packaging the addons separately, I'll
definitely work on the metapackage first.

> As above, I'd prefer if it was one procedure and one package pointing
> to the latest j80x, assuming all of our added patches can also be
> applied to earlier versions.

Yeah, that would definitely be ideal.

Unfortunately, it's not so straigthforward. The the "major versions" are the
xxx part of jxxx-y, with large changes between each, e.g. j902 introduced and
non-compatible API change over j901. So, from a pure J user perspective, having
all jxxx versions available is ideal.

However, each version seems to require its own set of build flags and sometimes
version-specific patches (like in the j901 case). We probably don't want to
push those settings out into user manifest specs.

On the other hand, there are already 10 versions from j801 to j903.

> > +(define make-jlib
> > +  (match-lambda
> > +    (($ <jlib-build-configuration>
> > +        builder jversion revision hash type patches extra-inputs
> > extra-envars)
> Please try to use keyword arguments.

Actually, maybe these "build configuration" records could solve the above
"too many versions" problem. We could offer only the latest J (and J beta?)
packages in the repo, but let (gnu packages jsoftware) export a set of "default
configurations" for old versions to be used with the make-j procedure.

Any particular reason to avoid using records though? Currently, its letting us
share configuration options between j902 and j903 that don't work in j901.
Personally, I thought this felt like a nice declarative, scheme-y way to go,
but my Scheme is still very amateurish. Definitely wanting to rectify any
strange ideas I may have.

> > +       (properties `((jversion . ,jversion)
> > +                     (jrevision . ,revision)
> > +                     (jtype . ,type)))
> Are those used anywhere?  Can jversion and jrevision not be parsed from
> (package-version jlib)?

The make-j procedure uses them. We'd have to parse out these from both the
version string and package name with a what's essentially the inverse of the
jname procedure. I found it a lot more readable and less hassle to just
propagate this data directly.

However, if there's a particular reason why using properties is problematic,
I'll try to see what I can do.

[bug#48463] gnu: Add j.

[Leo Prikler]

Hi,

Am Montag, den 24.05.2021, 22:37 +0900 schrieb elaexuotee@wilsonb.com:
> > Are fat binaries the accepted HPC way?  I'm not up-to-date to this.
> 
> I believe so? Don't quote me on that though. For this particular
> package the
> overhoad is just ~8MB, so I don't think it's much of an issue either
> way. It
> would probably be more of a hassle to split j into a package for each
> variant.
In that case it will probably be fine.

> > Both sound like good ideas, although I'm a little less sure about
> > the
> > texlive one.  Would a meta-package like gnome also be acceptable?
> 
> A metapackage is certainly practical. The entire set of J addons
> currently weighs in at a whopping 50MB. hehe. However, there are
> about 125 addons in total, so it simply feels "more correct" to let
> the user also choose just the addons they want, which I think would
> require something like jsoftware-union.
> 
> That said, either way, an "all batteries include" package would be
> good to have, and since this is way easier than packaging the addons
> separately, I'll definitely work on the metapackage first.
stuff-union usually implies, that stuff can't just be put into a
profile and expected to work (like texlive), whereas in other cases
(e.g. the gnome metapackage) people can use its parts on their own
provided that they have the right combination (which typically implies
having icons etc.)

> > As above, I'd prefer if it was one procedure and one package
> > pointing to the latest j80x, assuming all of our added patches can
> > also be applied to earlier versions.
> 
> Yeah, that would definitely be ideal.
> 
> Unfortunately, it's not so straigthforward. The the "major versions"
> are the xxx part of jxxx-y, with large changes between each, e.g.
> j902 introduced and non-compatible API change over j901. So, from a
> pure J user perspective, having all jxxx versions available is ideal.
> 
> However, each version seems to require its own set of build flags and
> sometimes version-specific patches (like in the j901 case). We
> probably don't want to push those settings out into user manifest
> specs.
> 
> On the other hand, there are already 10 versions from j801 to j903.
That's less than the number of Rust versions we need just for
bootstrap, but still a few too many in my opinion.  Are all of those
still used in production or would it be wiser to to put some of them
into Guix-Past [1]?

> > > +(define make-jlib
> > > +  (match-lambda
> > > +    (($ <jlib-build-configuration>
> > > +        builder jversion revision hash type patches extra-inputs
> > > extra-envars)
> > Please try to use keyword arguments.
> 
> Actually, maybe these "build configuration" records could solve the
> above "too many versions" problem. We could offer only the latest J
> (and J beta?) packages in the repo, but let (gnu packages jsoftware)
> export a set of "default configurations" for old versions to be used
> with the make-j procedure.
I don't think there's much to be gained if we provide packages without
packages.

> Any particular reason to avoid using records though? Currently, its
> letting us share configuration options between j902 and j903 that
> don't work in j901.
> Personally, I thought this felt like a nice declarative, scheme-y way
> to go, but my Scheme is still very amateurish. Definitely wanting to
> rectify any strange ideas I may have.
Having keyword lists is also nice and declarative, but more
importantly, it lets you call the function as a normal function without
having to worry about constructing some record in a particular way. 
The configuration you're using doesn't have a specific setting?  Just
override it with your own.  It's as simple as calling (append old-
config extra-config).

> > > +       (properties `((jversion . ,jversion)
> > > +                     (jrevision . ,revision)
> > > +                     (jtype . ,type)))
> > Are those used anywhere?  Can jversion and jrevision not be parsed
> > from (package-version jlib)?
> 
> The make-j procedure uses them. We'd have to parse out these from
> both the version string and package name with a what's essentially
> the inverse of the jname procedure. I found it a lot more readable
> and less hassle to just propagate this data directly.
> 
> However, if there's a particular reason why using properties is
> problematic, I'll try to see what I can do.
I find this way of mistreating the version field very weird.  You're
not adding anything new by doing this and you're not making anything
more secure (I'd argue, that it's less secure, because you could update
the version and forget about the property or vice versa).

First of all, do you even need all this info?  `j' is an objectively
bad name for a package.  `j-beta' is not better in any way, it only
avoids the user installing a potentially unstable "J" package.  (Note,
that we typically use "-next" for that, however).  However, this info
is meaningless when hardcoded into a procedure.  You can just inherit
from the package that's generated and override the name as needed.

Next, should ijconsole not simply be a package like jlib (both properly
prefixed with jsoftware- of course)?  Then you can take whatever
version bits you need from the package version itself.

Regards,
Leo

[1] https://gitlab.inria.fr/guix-hpc/guix-past

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

As always, thanks for the quick turnaround.

Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> stuff-union usually implies, that stuff can't just be put into a
> profile and expected to work (like texlive), whereas in other cases
> (e.g. the gnome metapackage) people can use its parts on their own
> provided that they have the right combination (which typically implies
> having icons etc.)

Oh, okay.  For some reason, I was just imagining "metapackage" to mean "all
batteries included", but your description makes a lot more sense. In this
case, having addons without the J interpreter is pretty useless, so IIUC a
jsoftware-union makes more sense than a metapackage in this case?

Just to be clear, J looks for addons at a path burned into
$store/...-jlib-$version/etc/j/profilex.ijs.  Currently, that points to
$HOME/.j/$jversion/addons, but we'll need to change that to a single store
output path that contains the union of all desired addons, hence why I was
thinking the *-union approach is needed.

For now, instead of messing with the union, would it make sense to just create
a (non-public?) jsoftware-addons package that contains all addons and point
jlib at those?

> That's less than the number of Rust versions we need just for
> bootstrap, but still a few too many in my opinion.  Are all of those
> still used in production or would it be wiser to to put some of them
> into Guix-Past [1]?

Oh cool. Guix Past might be a better solution! Thanks for sharing.

I don't have any solid data, since "in production" J is mostly dominated by the
financial sector from what I hear, but just going by my impression from lurking
on the J mailing list, the j80x series (and even earlier non-free versions)
does still have a user base.

Personally, I just want to make these older versions available to the
community but am agnostic about the *how* so will defer to whatever you (and
others) think is best.

> Having keyword lists is also nice and declarative, but more
> importantly, it lets you call the function as a normal function without
> having to worry about constructing some record in a particular way. 
> The configuration you're using doesn't have a specific setting?  Just
> override it with your own.  It's as simple as calling (append old-
> config extra-config).

Oh! (append old-config extra-config) is exactly the kind of thing I was trying
to achieve; however, I'm not sure I quite grok the usage you are describing.
Something like this?

(define* (proc #:key foo bar) ...)
(define old-config '(#:foo 0))
(define extra-config '(#:bar 1))
(apply proc (append old-config extra-config))

> I find this way of mistreating the version field very weird.  You're
> not adding anything new by doing this and you're not making anything
> more secure (I'd argue, that it's less secure, because you could update
> the version and forget about the property or vice versa).

I agree the version stuff is a bit crappy.  It's something that got munged
badly over time and deserves more thought.  Here are the issues I'm trying to
solve:

  1) The source URLs look like ../j<version>-<type>-<patch> or
     ../j<version>-<type>, where <type> is either "beta" or "release", and
     depending on whether a patch-level exists;
  2) The type string ("release" or "beta") is a build setting; and
  3) ijconsole installs to something like ../bin/ijconsole-902, without
     including the patch-level, like ../bin/ijconsole-902-b.
  4) Users should probably be able to easily install the latest J release as
     well as J beta concurrently without caring about exactly which version
     these are.

Those imply that <version>, <type>, and <patch> are indeed handled separately,
no matter concrete method utilized.

My thinking was that 4 means we want a separate *-beta package, meaning that
the version field should probably just look like "<version>-<patch>".
Additionally, relating back to the above discussion about multiple versions

   5) It would be nice to be able to install the latest j901 without having to
      know that this corresponds to patch level f, i.e. currently j@901-f.

> First of all, do you even need all this info?  `j' is an objectively bad name
> for a package.  `j-beta' is not better in any way, it only avoids the user
> installing a potentially unstable "J" package.  

Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about package
name, though.  Suggestions welcome.

> (Note, that we typically use "-next" for that, however).

Thanks for pointing out this convention.

> However, this info is meaningless when hardcoded into a procedure.  You can
> just inherit from the package that's generated and override the name as
> needed.

I see what you mean in a general sense but not quite how to apply it in this
case. I am all ears if you have a simpler, cleaner solution that addresses the
constraints listed above!

> Next, should ijconsole not simply be a package like jlib (both properly
> prefixed with jsoftware- of course)?

This is effectively what make-j does, no? ijconsole needs to know the path to
jlib, so make-j points it at the correct one and wraps everything up into a
package.


Anyway, thanks again for taking a look!

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Hi,

Am Dienstag, den 25.05.2021, 12:57 +0900 schrieb
elaexuotee@wilsonb.com:
> As always, thanks for the quick turnaround.
> 
> Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> > stuff-union usually implies, that stuff can't just be put into a
> > profile and expected to work (like texlive), whereas in other cases
> > (e.g. the gnome metapackage) people can use its parts on their own
> > provided that they have the right combination (which typically
> > implies having icons etc.)
> 
> Oh, okay.  For some reason, I was just imagining "metapackage" to
> mean "all batteries included", but your description makes a lot more
> sense. In this case, having addons without the J interpreter is
> pretty useless, so IIUC a jsoftware-union makes more sense than a
> metapackage in this case?
> 
> Just to be clear, J looks for addons at a path burned into
> $store/...-jlib-$version/etc/j/profilex.ijs.  Currently, that points
> to $HOME/.j/$jversion/addons, but we'll need to change that to a
> single store output path that contains the union of all desired
> addons, hence why I was thinking the *-union approach is needed.
> 
> For now, instead of messing with the union, would it make sense to
> just create a (non-public?) jsoftware-addons package that contains
> all addons and point jlib at those?
Are those addons data packages or can they be compiled without needing
jlib?  If they're pure data, then you can pack them up and refer them
as you wanted to, if not, you first have to create jlib-minimal without
them, then package up everything and finally do jlib.

Oh, and one thing that was previously ignored is that using XDG
standards for things we can't simply put in the store (like
configuration if it needs that) is preferable to having a ~/.j
directory.

> > That's less than the number of Rust versions we need just for
> > bootstrap, but still a few too many in my opinion.  Are all of
> > those still used in production or would it be wiser to to put some
> > of them into Guix-Past [1]?
> 
> Oh cool. Guix Past might be a better solution! Thanks for sharing.
> 
> I don't have any solid data, since "in production" J is mostly
> dominated by the financial sector from what I hear, but just going by
> my impression from lurking on the J mailing list, the j80x series
> (and even earlier non-free versions) does still have a user base.
> 
> Personally, I just want to make these older versions available to the
> community but am agnostic about the *how* so will defer to whatever
> you (and others) think is best.
People are also still running ancient versions of Debian.  Python 2 has
officially reached end of life, yet it is used as well.  At some point
we do have to declare software that people are still using as old :)

> > Having keyword lists is also nice and declarative, but more
> > importantly, it lets you call the function as a normal function
> > without having to worry about constructing some record in a
> > particular way.  The configuration you're using doesn't have a
> > specific setting?  Just override it with your own.  It's as simple
> > as calling (append old-config extra-config).
> 
> Oh! (append old-config extra-config) is exactly the kind of thing I
> was trying
> to achieve; however, I'm not sure I quite grok the usage you are
> describing.
> Something like this?
> 
> (define* (proc #:key foo bar) ...)
> (define old-config '(#:foo 0))
> (define extra-config '(#:bar 1))
> (apply proc (append old-config extra-config))
Yup.

> > I find this way of mistreating the version field very
> > weird.  You're not adding anything new by doing this and you're not
> > making anything more secure (I'd argue, that it's less secure,
> > because you could update the version and forget about the property
> > or vice versa).
> 
> I agree the version stuff is a bit crappy.  It's something that got
> munged badly over time and deserves more thought.  Here are the
> issues I'm trying to solve:
> 
>   1) The source URLs look like ../j<version>-<type>-<patch> or
>      ../j<version>-<type>, where <type> is either "beta" or
> "release", and
>      depending on whether a patch-level exists;
>   2) The type string ("release" or "beta") is a build setting; and
>   3) ijconsole installs to something like ../bin/ijconsole-902,
> without
>      including the patch-level, like ../bin/ijconsole-902-b.
>   4) Users should probably be able to easily install the latest J
> release as
>      well as J beta concurrently without caring about exactly which
> version
>      these are.
> 
> Those imply that <version>, <type>, and <patch> are indeed handled
> separately, no matter concrete method utilized.
Those four issues do call for separate keyword arguments, but they need
not necessarily be reflected in the version string.  You can have two
packages with the same version, but different hashes in Guix.  

> My thinking was that 4 means we want a separate *-beta package,
> meaning that the version field should probably just look like
> "<version>-<patch>".
> Additionally, relating back to the above discussion about multiple
> versions
Sounds good, though <version>.<patch> would be equally acceptable. 
Your call.

>    5) It would be nice to be able to install the latest j901 without
> having to
>       know that this corresponds to patch level f, i.e. currently 
> j@901-f.
I think `guix build j@901' ought to resolve this automatically.

> > First of all, do you even need all this info?  `j' is an
> > objectively bad name
> > for a package.  `j-beta' is not better in any way, it only avoids
> > the user installing a potentially unstable "J" package.  
> 
> Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about
> package name, though.  Suggestions welcome.
Fair enough, though this remains a problem for single letter programs
in which others would likely want to claim the letter as well.  R has
the benefit of being old and well-known enough, it would probably be
rlang otherwise (not to be confused with erlang).

> > (Note, that we typically use "-next" for that, however).
> 
> Thanks for pointing out this convention.
> 
> > However, this info is meaningless when hardcoded into a
> > procedure.  You can just inherit from the package that's generated
> > and override the name as needed.
> 
> I see what you mean in a general sense but not quite how to apply it
> in this case. I am all ears if you have a simpler, cleaner solution
> that addresses the constraints listed above!
I think the cleaner solution here is to not store those properties in
the package, but still having them as parameters to the make-j function
call.

> > Next, should ijconsole not simply be a package like jlib (both
> > properly prefixed with jsoftware- of course)?
> 
> This is effectively what make-j does, no? ijconsole needs to know the
> path to jlib, so make-j points it at the correct one and wraps
> everything up into a package.
I don't really understand why those needs to be done in two steps
however.  ijconsole could already contain the fully functioning j
program.  Or partially functioning if we account for addons.

Cheers!

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 5519 bytes --]

Here's a long-needed update.

> > For now, instead of messing with the union, would it make sense to
> > just create a (non-public?) jsoftware-addons package that contains
> > all addons and point jlib at those?
>
> Are those addons data packages or can they be compiled without needing
> jlib?  If they're pure data, then you can pack them up and refer them
> as you wanted to, if not, you first have to create jlib-minimal without
> them, then package up everything and finally do jlib.

They're pure data, which is nice.
The J package manager also lets users install arbitrary addons from GitHub, so
packaging up the official addons would lock the `addons/' directory under the
non-writable store, meaning that we probably also want an importer.

I am working on that, but instead of keeping this package in lingo much longer,
maybe we can release sonner and consider the addon package code a separate
issue?

> Oh, and one thing that was previously ignored is that using XDG
> standards for things we can't simply put in the store (like
> configuration if it needs that) is preferable to having a ~/.j
> directory.

Excellent point. Fixed.

> > > That's less than the number of Rust versions we need just for
> > > bootstrap, but still a few too many in my opinion.  Are all of
> > > those still used in production or would it be wiser to to put some
> > > of them into Guix-Past [1]?
>
> People are also still running ancient versions of Debian.  Python 2 has
> officially reached end of life, yet it is used as well.  At some point
> we do have to declare software that people are still using as old :)

Sounds sane.

I will probably eventually work on getting the j80x series in Guix and the
really ancient ones in Guix Past. However, that will be a future task for
myself.

> > > Having keyword lists is also nice and declarative, but more
> > > importantly, it lets you call the function as a normal function
> > > without having to worry about constructing some record in a
> > > particular way.  The configuration you're using doesn't have a
> > > specific setting?  Just override it with your own.  It's as simple
> > > as calling (append old-config extra-config).
> > ...
> Yup.

Now using keyword lists.

> > > I find this way of mistreating the version field very
> > > weird.  You're not adding anything new by doing this and you're not
> > > making anything more secure (I'd argue, that it's less secure,
> > > because you could update the version and forget about the property
> > > or vice versa).

The code overall is a lot more straightforward now. How does the versioning
stuff look now?

> > My thinking was that 4 means we want a separate *-beta package,
> > meaning that the version field should probably just look like
> > "<version>-<patch>".
> > Additionally, relating back to the above discussion about multiple
> > versions
> Sounds good, though <version>.<patch> would be equally acceptable. 
> Your call.
>
> >    5) It would be nice to be able to install the latest j901 without
> > having to
> >       know that this corresponds to patch level f, i.e. currently 
> > j@901-f.
> I think `guix build j@901' ought to resolve this automatically.

It turns out that the j@901 specification only gets recognized with versions
strings of the form <version>.<patch>, so going with that.

> > > First of all, do you even need all this info?  `j' is an
> > > objectively bad name
> > > for a package.  `j-beta' is not better in any way, it only avoids
> > > the user installing a potentially unstable "J" package.  
> > 
> > Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about
> > package name, though.  Suggestions welcome.
> Fair enough, though this remains a problem for single letter programs
> in which others would likely want to claim the letter as well.  R has
> the benefit of being old and well-known enough, it would probably be
> rlang otherwise (not to be confused with erlang).

J has been around from the 1990's :D
That said, I went ahead and named it `jsoftware-j', since despite age, J is
still pretty obscure, and there's not much of a good reason to monopolize a
single letter name.

> > I see what you mean in a general sense but not quite how to apply it
> > in this case. I am all ears if you have a simpler, cleaner solution
> > that addresses the constraints listed above!
> I think the cleaner solution here is to not store those properties in
> the package, but still having them as parameters to the make-j function
> call.

Happily, the new package re-organization was able to get rid of these
properties.

> > > Next, should ijconsole not simply be a package like jlib (both
> > > properly prefixed with jsoftware- of course)?
> > 
> > This is effectively what make-j does, no? ijconsole needs to know the
> > path to jlib, so make-j points it at the correct one and wraps
> > everything up into a package.
> I don't really understand why those needs to be done in two steps
> however.  ijconsole could already contain the fully functioning j
> program.  Or partially functioning if we account for addons.

As you can see, there's no need for a separate ijconsole package any more.
Instead, `ijconsole' is just defined as a program-file that becomes an
input to jsoftware-j.


It is also worth pointing out the convenience procedure
`jsoftware-ijconsole-symlink'. Basically, it p


Anyway, let me know what you think!



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26575 bytes --]

From 98be43134f62132a98a97f7601e97c3ae9ec63c6 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/jsoftware.scm                    | 455 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 +++
 3 files changed, 536 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1706663bde..0cbb8899ee 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1290,6 +1290,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..472a6c30d4
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,455 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module ((guix build utils))
+  #:use-module ((guix build-system gnu))
+  #:use-module ((guix build-system trivial))
+  #:use-module ((guix git-download))
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module ((guix packages))
+  #:use-module ((guix utils))
+  #:use-module ((gnu packages))
+  #:use-module ((gnu packages libedit))
+  #:use-module ((gnu packages llvm))
+  #:use-module ((gnu packages maths))
+  #:use-module ((guix gexp))
+  #:use-module ((ice-9 ftw))
+  #:use-module ((ice-9 match))
+  #:use-module ((ice-9 regex))
+  #:use-module ((ice-9 rdelim))
+  #:use-module ((srfi srfi-26)))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix jtype)
+  "Return a package name for the given prefix and jtype, e.g. `jlib',
+  `jlib-beta', `j', and `j-beta'."
+  (let ((postfix (if (eq? jtype 'release) ""
+                   (string-append "-" (symbol->string jtype)))))
+    (string-append prefix postfix)))
+
+(define (jversion->string version revision)
+  "Return a string representation of a J version and (optional) revision pair."
+  (let ((postfix (if (not revision) ""
+                   (string-append "." revision))))
+    (string-append version postfix)))
+
+(define (jinfo->git-tag jversion jtype jrevision)
+  "Given version parameters, construct a git tag for upstream releases."
+  (let ((postfix (if (not jrevision) ""
+                   (string-append "-" jrevision))))
+    (string-append "j" jversion "-" (symbol->string jtype) postfix)))
+
+(define (ijconsole)
+  "Generate a G-exp script that detects AVX/AVX2 support at runtime and
+  executes jconsole with the appropriate libj.so and profile.ijs."
+  (program-file "ijconsole"
+    #~(begin
+        (use-modules ((ice-9 rdelim) #:select (read-line))
+                     ((ice-9 regex) #:select (regexp-match? string-match)))
+
+        ;; Assume that this script will be installed under bin/.
+        (define %basedir (dirname (dirname (current-filename))))
+
+        (define (cpu-feature-line? string)
+          (string-prefix? "flags" string))
+
+        (define (contains-word? word string)
+          (regexp-match?
+            (string-match (string-join `("\\<" ,word "\\>") "")
+                          string)))
+
+        (define (has-cpu-feature? feature)
+          (with-input-from-file "/proc/cpuinfo"
+             (lambda ()
+               (catch 'found
+                 (lambda ()
+                   (let loop ((line (read-line)))
+                     (cond ((eof-object? line) #f)
+                           ((and (cpu-feature-line? line)
+                                 (contains-word? feature line))
+                            (throw 'found))
+                           (else (loop (read-line))))))
+                 (const #t)))))
+
+        (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+               (libj (format #f "~a/lib/j/libj-~a.so" %basedir
+                             (cond ((has-cpu-feature? "avx2") "avx2")
+                                   ((has-cpu-feature? "avx") "avx")
+                                   (else ""))))
+               (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+          (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                 (cdr (command-line)))))))
+
+(define* (make-j #:key
+                 (builder "guix.gnu.org")
+                 vername
+                 revision
+                 hash
+                 (type 'release)
+                 commit
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '()))
+ (package
+   (name (jname "jsoftware-j" type))
+   (version (jversion->string vername revision))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or commit (jinfo->git-tag vername type revision)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
+   (inputs (cons* `("libedit" ,libedit)
+                  `("libomp" ,libomp)
+                  `("ijconsole" ,(ijconsole))
+                  extra-inputs))
+   (arguments
+    `(#:tests? #f
+      #:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n" (symbol->string ',type)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Create profilex.ijs overrides to point to the correct store
+              ;; items.  Note that we set ~install and ~addons directories to
+              ;; reside under ~user to allow installing and loading addons.
+              ;; TODO: Guix-ify J addons as well.
+              (call-with-output-file "jlibrary/bin/profilex.ijs"
+                (lambda (port)
+                  (display
+                    (string-join
+                      (list
+                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/') BINPATH"
+                        "system=. share,'/system'"
+                        "tools=. share,'/tools'"
+                        ;; Upstream defaults to spamming $HOME with unhidden
+                        ;; userdata directories.  Set this to be
+                        ;; $HOME/.j/<jtype>/<jversion> instead
+                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                        "jversion=. ({.~ i.&'-') jversion"
+                        "jsuffix=. >@{&('';'-beta') jtype -: 'beta'"
+                        "user=. home,'/.config/j/',jversion,jsuffix"
+                        "addons=. user,'/addons'"
+                        "break=. user,'/break'"
+                        "config=. user,'/config'"
+                        "install=. user,'/install'"
+                        "snap=. user,'/snap'"
+                        "temp=. user,'/temp'"
+                        "\n")
+                      "\n")
+                    port)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (jplat (string-append "jplatform=" platform))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (jbit (string-append "j64x=" "j" target-bit))
+                     (jbit-avx (string-append jbit "avx"))
+                     (jbit-avx2 (string-append jbit "avx2")))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (invoke "env" jplat jbit "./build_jconsole.sh")
+                  (invoke "env" jplat jbit "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (invoke "env" jplat jbit-avx "./build_libj.sh")
+                      (invoke "env" jplat jbit-avx2
+                              "./build_libj.sh"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((jplatform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (access? dev-dir R_OK)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((jbin (string-append "bin/" jplatform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append jbin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append jbin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jc (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jc tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" jplatform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((jplat ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (ijconsole (assoc-ref inputs "ijconsole"))
+                   (interp (string-join `("bin" ,jplat ,jbit "jconsole") "/"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file ijconsole (string-append bin "/ijconsole-" ,vername))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,jplat ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" jplat)
+                         (negate (cut member <> '("." "..")))))
+              (install-file interp libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+)))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:vername "901"
+    #:revision "f"
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define jlib-build-configuration-with-sleef
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      '(#:vername "902"
+        #:revision "b"
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      '(#:vername "903"
+        #:revision "a"
+        #:commit "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+;; Keep until j904-beta is released.
+(define-public jsoftware-j-903-beta
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      `(#:vername "903"
+        #:revision "w"
+        #:type ,'beta
+        #:hash "0kd63mrkaq0bs42gw0wrlb14fymhigznx1lrb698dgv6fzlfglim"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (interp (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" interp))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

elaexuotee--- via Guix-patches via schreef op wo 12-01-2022 om 18:47 [+0900]:
+                  (if ,(target-64bit?)
+                    (parallel
+                      (invoke "env" jplat jbit-avx "./build_libj.sh")
+                      (invoke "env" jplat jbit-avx2
+                              "./build_libj.sh"))))))))

I don't think avx and avx2 are available on aarch64.
Maybe you can use ,(target-x86-64?) instead?

> +    `(#:tests? #f
>          [...]
> +        (replace 'check
> +          (lambda* (#:key tests? #:allow-other-keys)
> +            (when tests? [...]))))

I think #:tests? #false should be removed, otherwise this
check phase would never do anything.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#48463] gnu: Add j.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

Hi,
> +                ;; The jlibrary/dev directory only sometimes exists,
> but needs
> +                ;; to be copied into the ~system directory when it
> does.
> +                (for-each
> +                  (lambda (dev-dir)
> +                    (if (access? dev-dir R_OK)
> +                      (copy-recursively dev-dir
> "test/bin/system/dev")))


Are you testing for file permissions, or for the existence of the file?
If the latter, I'd recommend using 'file-exists?' instead.

> +                  (if (access? source-dev R_OK)


Likewise.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

> > +    `(#:tests? #f
> >          [...]
> > +        (replace 'check
> > +          (lambda* (#:key tests? #:allow-other-keys)
> > +            (when tests? [...]))))
> 
> I think #:tests? #false should be removed, otherwise this
> check phase would never do anything.

Nice catch! Thanks. Disabled tests when tweaking builds and forgot to remove.

> > +                ;; The jlibrary/dev directory only sometimes exists,
> > but needs
> > +                ;; to be copied into the ~system directory when it
> > does.
> > +                (for-each
> > +                  (lambda (dev-dir)
> > +                    (if (access? dev-dir R_OK)
> > +                      (copy-recursively dev-dir
> > "test/bin/system/dev")))
> 
> 
> Are you testing for file permissions, or for the existence of the file?
> If the latter, I'd recommend using 'file-exists?' instead.
> 
> > +                  (if (access? source-dev R_OK)

Just existence. Thanks for the pointer. Changed.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26555 bytes --]

From 648efabc2d9e7b9a28d22d47ea53451bc5b46cf3 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/jsoftware.scm                    | 454 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 +++
 3 files changed, 535 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1706663bde..0cbb8899ee 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1290,6 +1290,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..97b561ec27
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,454 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module ((guix build utils))
+  #:use-module ((guix build-system gnu))
+  #:use-module ((guix build-system trivial))
+  #:use-module ((guix git-download))
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module ((guix packages))
+  #:use-module ((guix utils))
+  #:use-module ((gnu packages))
+  #:use-module ((gnu packages libedit))
+  #:use-module ((gnu packages llvm))
+  #:use-module ((gnu packages maths))
+  #:use-module ((guix gexp))
+  #:use-module ((ice-9 ftw))
+  #:use-module ((ice-9 match))
+  #:use-module ((ice-9 regex))
+  #:use-module ((ice-9 rdelim))
+  #:use-module ((srfi srfi-26)))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix jtype)
+  "Return a package name for the given prefix and jtype, e.g. `jlib',
+  `jlib-beta', `j', and `j-beta'."
+  (let ((postfix (if (eq? jtype 'release) ""
+                   (string-append "-" (symbol->string jtype)))))
+    (string-append prefix postfix)))
+
+(define (jversion->string version revision)
+  "Return a string representation of a J version and (optional) revision pair."
+  (let ((postfix (if (not revision) ""
+                   (string-append "." revision))))
+    (string-append version postfix)))
+
+(define (jinfo->git-tag jversion jtype jrevision)
+  "Given version parameters, construct a git tag for upstream releases."
+  (let ((postfix (if (not jrevision) ""
+                   (string-append "-" jrevision))))
+    (string-append "j" jversion "-" (symbol->string jtype) postfix)))
+
+(define (ijconsole)
+  "Generate a G-exp script that detects AVX/AVX2 support at runtime and
+  executes jconsole with the appropriate libj.so and profile.ijs."
+  (program-file "ijconsole"
+    #~(begin
+        (use-modules ((ice-9 rdelim) #:select (read-line))
+                     ((ice-9 regex) #:select (regexp-match? string-match)))
+
+        ;; Assume that this script will be installed under bin/.
+        (define %basedir (dirname (dirname (current-filename))))
+
+        (define (cpu-feature-line? string)
+          (string-prefix? "flags" string))
+
+        (define (contains-word? word string)
+          (regexp-match?
+            (string-match (string-join `("\\<" ,word "\\>") "")
+                          string)))
+
+        (define (has-cpu-feature? feature)
+          (with-input-from-file "/proc/cpuinfo"
+             (lambda ()
+               (catch 'found
+                 (lambda ()
+                   (let loop ((line (read-line)))
+                     (cond ((eof-object? line) #f)
+                           ((and (cpu-feature-line? line)
+                                 (contains-word? feature line))
+                            (throw 'found))
+                           (else (loop (read-line))))))
+                 (const #t)))))
+
+        (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+               (libj (format #f "~a/lib/j/libj-~a.so" %basedir
+                             (cond ((has-cpu-feature? "avx2") "avx2")
+                                   ((has-cpu-feature? "avx") "avx")
+                                   (else ""))))
+               (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+          (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                 (cdr (command-line)))))))
+
+(define* (make-j #:key
+                 (builder "guix.gnu.org")
+                 vername
+                 revision
+                 hash
+                 (type 'release)
+                 commit
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '()))
+ (package
+   (name (jname "jsoftware-j" type))
+   (version (jversion->string vername revision))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or commit (jinfo->git-tag vername type revision)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
+   (inputs (cons* `("libedit" ,libedit)
+                  `("libomp" ,libomp)
+                  `("ijconsole" ,(ijconsole))
+                  extra-inputs))
+   (arguments
+    `(#:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n" (symbol->string ',type)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Create profilex.ijs overrides to point to the correct store
+              ;; items.  Note that we set ~install and ~addons directories to
+              ;; reside under ~user to allow installing and loading addons.
+              ;; TODO: Guix-ify J addons as well.
+              (call-with-output-file "jlibrary/bin/profilex.ijs"
+                (lambda (port)
+                  (display
+                    (string-join
+                      (list
+                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/') BINPATH"
+                        "system=. share,'/system'"
+                        "tools=. share,'/tools'"
+                        ;; Upstream defaults to spamming $HOME with unhidden
+                        ;; userdata directories.  Set this to be
+                        ;; $HOME/.j/<jtype>/<jversion> instead
+                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                        "jversion=. ({.~ i.&'-') jversion"
+                        "jsuffix=. >@{&('';'-beta') jtype -: 'beta'"
+                        "user=. home,'/.config/j/',jversion,jsuffix"
+                        "addons=. user,'/addons'"
+                        "break=. user,'/break'"
+                        "config=. user,'/config'"
+                        "install=. user,'/install'"
+                        "snap=. user,'/snap'"
+                        "temp=. user,'/temp'"
+                        "\n")
+                      "\n")
+                    port)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (jplat (string-append "jplatform=" platform))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (jbit (string-append "j64x=" "j" target-bit))
+                     (jbit-avx (string-append jbit "avx"))
+                     (jbit-avx2 (string-append jbit "avx2")))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (invoke "env" jplat jbit "./build_jconsole.sh")
+                  (invoke "env" jplat jbit "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (invoke "env" jplat jbit-avx "./build_libj.sh")
+                      (invoke "env" jplat jbit-avx2
+                              "./build_libj.sh"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((jplatform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (file-exists? dev-dir)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((jbin (string-append "bin/" jplatform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append jbin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append jbin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jc (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jc tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" jplatform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((jplat ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (ijconsole (assoc-ref inputs "ijconsole"))
+                   (interp (string-join `("bin" ,jplat ,jbit "jconsole") "/"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file ijconsole (string-append bin "/ijconsole-" ,vername))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,jplat ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" jplat)
+                         (negate (cut member <> '("." "..")))))
+              (install-file interp libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+)))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:vername "901"
+    #:revision "f"
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define jlib-build-configuration-with-sleef
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      '(#:vername "902"
+        #:revision "b"
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      '(#:vername "903"
+        #:revision "a"
+        #:commit "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+;; Keep until j904-beta is released.
+(define-public jsoftware-j-903-beta
+  (apply make-j
+    (append jlib-build-configuration-with-sleef
+      `(#:vername "903"
+        #:revision "w"
+        #:type ,'beta
+        #:hash "0kd63mrkaq0bs42gw0wrlb14fymhigznx1lrb698dgv6fzlfglim"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (interp (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" interp))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Hi,

> +(define-module (gnu packages jsoftware)
> +  #:use-module ((guix build utils))
> + [...]

Double bracketing is pointless, use it only when needed.

> +(define* (make-j #:key
> +                 (builder "guix.gnu.org")
> +                 vername
> +                 revision
> +                 hash
> +                 (type 'release)
> +                 commit
> +                 (patches '())
> +                 (extra-inputs '())
> +                 (extra-envars '()))
> + (package
> +   (name (jname "jsoftware-j" type))
> +   (version (jversion->string vername revision))
> +   (source
> +    (origin
> +      (method git-fetch)
> +      (uri (git-reference
> +            (url "https://github.com/jsoftware/jsource")
> +            (commit (or commit (jinfo->git-tag vername type
> revision))))
Vername sounds a little weird, make that version-base or something
clearer.  Also, the argument #:commit is used in an unclear fashion --
if you were to pass an actual commit hash to it, it'd still be treated
as a release and not be using git-version. 

On a related note
> +(define (jversion->string version revision)
> +  "Return a string representation of a J version and (optional)
> revision pair."
> +  (let ((postfix (if (not revision) ""
> +                   (string-append "." revision))))
> +    (string-append version postfix)))
should also take commit and revision should probably be dashed.  In
that way, when packaging commits between releases we can use
"jrevision.guix-revision" as the complete revision.

In short, I'd add a #:tag argument to override the tag and treat commit
like a let-bound commit.

> +    `(#:modules (((ice-9 ftw) #:select (scandir))
> +                 ((ice-9 popen) #:select (open-pipe* close-pipe))
> +                 ((ice-9 regex) #:select (match:substring string-
> match))
> +                 ((ice-9 threads) #:select (parallel par-for-each))
> +                 ((srfi srfi-26) #:select (cut))
> +                 ((srfi srfi-1) #:select (fold))
> +                 ,@%gnu-build-system-modules)
It's nice that you annotated all those, but note that it probably
wouldn't have been needed.  If you notice this list getting longer and
longer as you update, consider dropping the #:selects.

> +        (replace 'build
> +          (lambda _
> +            (setenv "USE_OPENMP" "1")
> +            (setenv "USE_THREAD" "1")
> +            (for-each (lambda (var-val) (apply setenv var-val))
> +                      (quote ,extra-envars))
> +            ;; The build scripts assume that PWD is make2.
> +            (with-directory-excursion "make2"
> +              (let* ((platform ,(if (target-arm?) "raspberry"
> "linux"))
> +                     (jplat (string-append "jplatform=" platform))
> +                     (target-bit ,(if (target-64bit?) "64" "32"))
> +                     (jbit (string-append "j64x=" "j" target-bit))
> +                     (jbit-avx (string-append jbit "avx"))
> +                     (jbit-avx2 (string-append jbit "avx2")))
> +                (parallel
> +                  ;; Since jconsole doesn't depend on AVX features,
> we just
> +                  ;; build it once.
> +                  (invoke "env" jplat jbit "./build_jconsole.sh")
> +                  (invoke "env" jplat jbit "./build_libj.sh")
> +                  (if ,(target-64bit?)
> +                    (parallel
> +                      (invoke "env" jplat jbit-avx
> "./build_libj.sh")
> +                      (invoke "env" jplat jbit-avx2
> +                              "./build_libj.sh"))))))))
Maxime already made a comment w.r.t. 32bit AVX here, but I think this
would be a prime example to use the CPU tuning that was recently
landed.  

Most of the above (except the semantics of the make-j keyword
arguments) are not blockers in my opinion.

Cheers

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 4776 bytes --]

> > +(define-module (gnu packages jsoftware)
> > +  #:use-module ((guix build utils))
> > + [...]
> 
> Double bracketing is pointless, use it only when needed.

Ah, nice catch. I had a bunch of #:select keys and forgot to kill the parens
when removing.

> > +(define* (make-j #:key
> > +                 (builder "guix.gnu.org")
> > +                 vername
> > +                 revision
> > +                 hash
> > +                 (type 'release)
> > +                 commit
> > +                 (patches '())
> > +                 (extra-inputs '())
> > +                 (extra-envars '()))
> > + (package
> > +   (name (jname "jsoftware-j" type))
> > +   (version (jversion->string vername revision))
> > +   (source
> > +    (origin
> > +      (method git-fetch)
> > +      (uri (git-reference
> > +            (url "https://github.com/jsoftware/jsource")
> > +            (commit (or commit (jinfo->git-tag vername type
> > revision))))
> Vername sounds a little weird, make that version-base or something
> clearer.  Also, the argument #:commit is used in an unclear fashion --
> if you were to pass an actual commit hash to it, it'd still be treated
> as a release and not be using git-version. 

Cool. I had a similar sense, but our ideas are a lot sharper than the ones I
had. This actually prompted me to do some code cleanup, leveraging now-me who
has a bit more Guile experience than past-me. At the very least, variable names
should be more descriptive and consistent, overall.

> On a related note
> > +(define (jversion->string version revision)
> > +  "Return a string representation of a J version and (optional)
> > revision pair."
> > +  (let ((postfix (if (not revision) ""
> > +                   (string-append "." revision))))
> > +    (string-append version postfix)))
> should also take commit and revision should probably be dashed.  In
> that way, when packaging commits between releases we can use
> "jrevision.guix-revision" as the complete revision.
> 
> In short, I'd add a #:tag argument to override the tag and treat commit
> like a let-bound commit.

Done.

> > +    `(#:modules (((ice-9 ftw) #:select (scandir))
> > +                 ((ice-9 popen) #:select (open-pipe* close-pipe))
> > +                 ((ice-9 regex) #:select (match:substring string-
> > match))
> > +                 ((ice-9 threads) #:select (parallel par-for-each))
> > +                 ((srfi srfi-26) #:select (cut))
> > +                 ((srfi srfi-1) #:select (fold))
> > +                 ,@%gnu-build-system-modules)
> It's nice that you annotated all those, but note that it probably
> wouldn't have been needed.  If you notice this list getting longer and
> longer as you update, consider dropping the #:selects.
> 
> > +        (replace 'build
> > +          (lambda _
> > +            (setenv "USE_OPENMP" "1")
> > +            (setenv "USE_THREAD" "1")
> > +            (for-each (lambda (var-val) (apply setenv var-val))
> > +                      (quote ,extra-envars))
> > +            ;; The build scripts assume that PWD is make2.
> > +            (with-directory-excursion "make2"
> > +              (let* ((platform ,(if (target-arm?) "raspberry"
> > "linux"))
> > +                     (jplat (string-append "jplatform=" platform))
> > +                     (target-bit ,(if (target-64bit?) "64" "32"))
> > +                     (jbit (string-append "j64x=" "j" target-bit))
> > +                     (jbit-avx (string-append jbit "avx"))
> > +                     (jbit-avx2 (string-append jbit "avx2")))
> > +                (parallel
> > +                  ;; Since jconsole doesn't depend on AVX features,
> > we just
> > +                  ;; build it once.
> > +                  (invoke "env" jplat jbit "./build_jconsole.sh")
> > +                  (invoke "env" jplat jbit "./build_libj.sh")
> > +                  (if ,(target-64bit?)
> > +                    (parallel
> > +                      (invoke "env" jplat jbit-avx
> > "./build_libj.sh")
> > +                      (invoke "env" jplat jbit-avx2
> > +                              "./build_libj.sh"))))))))
> Maxime already made a comment w.r.t. 32bit AVX here, but I think this
> would be a prime example to use the CPU tuning that was recently
> landed.  

Good idea. Upstream's build scripts condition a *lot* of behaviour on the
j64avx environment variable, so it might not be straightforward, but I will put
this on the to-do list of future improvements.

(Note, the code block quoted here got much simplified in the current patch.)

> Most of the above (except the semantics of the make-j keyword
> arguments) are not blockers in my opinion.

Cheers!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26819 bytes --]

From aa3015b070d2314e61a18072610bafb62cc41794 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/jsoftware.scm                    | 457 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 +++
 3 files changed, 538 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1706663bde..0cbb8899ee 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1290,6 +1290,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..8c593b5b23
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,457 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-26))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+(define* (jversion->string major #:optional revision commit)
+  "Return a version string formatted like MAJOR.REVISION-COMMIT or
+  MAJOR.REVISION, MAJOR.COMMIT, or just MAJOR according to existing
+  parameters."
+  (match `(,revision ,commit)
+    ((#f #f) major)
+    ((_  #f) (string-append major "." revision))
+    ((#f _)  (string-append major "." commit))
+    ((_ _)   (string-append major "." revision "-" commit))))
+
+(define* (jrelease-string release-type #:optional version-revision)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if version-revision
+     (string-append release-type "-" version-revision)
+     release-type)))
+
+(define* (jinfo->git-tag version-major release-type #:optional version-revision)
+  "Given version parameters, construct a git tag for upstream releases."
+  (string-append "j" version-major (jrelease-string release-type version-revision)))
+
+(define (ijconsole)
+  "Generate a G-exp script that detects AVX/AVX2 support at runtime and
+  executes jconsole with the appropriate libj.so and profile.ijs."
+  (program-file "ijconsole"
+    #~(begin
+        (use-modules ((ice-9 rdelim) #:select (read-line))
+                     ((ice-9 regex) #:select (regexp-match? string-match)))
+
+        ;; Assume that this script will be installed under bin/.
+        (define %basedir (dirname (dirname (current-filename))))
+
+        (define (cpu-feature-line? string)
+          (string-prefix? "flags" string))
+
+        (define (contains-word? word string)
+          (regexp-match?
+            (string-match (string-join `("\\<" ,word "\\>") "")
+                          string)))
+
+        (define (has-cpu-feature? feature)
+          (with-input-from-file "/proc/cpuinfo"
+             (lambda ()
+               (catch 'found
+                 (lambda ()
+                   (let loop ((line (read-line)))
+                     (cond ((eof-object? line) #f)
+                           ((and (cpu-feature-line? line)
+                                 (contains-word? feature line))
+                            (throw 'found))
+                           (else (loop (read-line))))))
+                 (const #t)))))
+
+        (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+               (libj (format #f "~a/lib/j/libj-~a.so" %basedir
+                             (cond ((has-cpu-feature? "avx2") "avx2")
+                                   ((has-cpu-feature? "avx") "avx")
+                                   (else ""))))
+               (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+          (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                 (cdr (command-line)))))))
+
+(define* (make-j #:key
+                 version-major
+                 version-revision
+                 hash
+                 tag
+                 (release-type 'release)
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+ (package
+   (name (jname "jsoftware-j" release-type))
+   (version (jversion->string version-major version-revision))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or tag (jinfo->git-tag version-major
+                                            release-type
+                                            version-revision)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
+   (inputs (cons* `("libedit" ,libedit)
+                  `("libomp" ,libomp)
+                  `("ijconsole" ,(ijconsole))
+                  extra-inputs))
+   (arguments
+    `(#:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version-major))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n"
+                         ,(jrelease-string release-type version-revision)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Create profilex.ijs overrides to point to the correct store
+              ;; items.  Note that we set ~install and ~addons directories to
+              ;; reside under ~user to allow installing and loading addons.
+              ;; TODO: Guix-ify J addons as well.
+              (call-with-output-file "jlibrary/bin/profilex.ijs"
+                (lambda (port)
+                  (display
+                    (string-join
+                      (list
+                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/') BINPATH"
+                        "system=. share,'/system'"
+                        "tools=. share,'/tools'"
+                        ;; Upstream defaults to spamming $HOME with unhidden
+                        ;; userdata directories.  Set this to be
+                        ;; $HOME/.config/j/<jversion> instead
+                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                        "jversion=. ({.~ i.&'-') jversion"
+                        "user=. home,'/.config/j/',jversion"
+                        "addons=. user,'/addons'"
+                        "break=. user,'/break'"
+                        "config=. user,'/config'"
+                        "install=. user,'/install'"
+                        "snap=. user,'/snap'"
+                        "temp=. user,'/temp'"
+                        "\n")
+                      "\n")
+                    port)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (run (lambda* (script #:key (avx ""))
+                            (invoke "env"
+                                    (string-append "jplatform=" platform)
+                                    (string-append "j64x=j" target-bit avx)
+                                    script))))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (run "./build_jconsole.sh")
+                  (run "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (run "./build_libj.sh" #:avx "avx")
+                      (run "./build_libj.sh" #:avx "avx2"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (file-exists? dev-dir)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((bin (string-append "bin/" platform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append bin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append bin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jconsole (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jconsole tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (ijconsole (assoc-ref inputs "ijconsole"))
+                   (jconsole (string-append "bin/" platform "/"
+                                            jbit "/jconsole"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file ijconsole (string-append bin "/ijconsole-"
+                                                  ,version-major))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" platform)
+                         (negate (cut member <> '("." "..")))))
+              (install-file jconsole libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+)))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:version-major "901"
+    #:version-revision "f"
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      '(#:version-major "902"
+        #:version-revision "b"
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      '(#:version-major "903"
+        #:version-revision "a"
+        #:tag "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]

Hi,

Am Donnerstag, dem 13.01.2022 um 16:51 +0900 schrieb
elaexuotee@wilsonb.com:
> > 
> > > +(define* (make-j #:key
> > > +                 (builder "guix.gnu.org")
> > > +                 vername
> > > +                 revision
> > > +                 hash
> > > +                 (type 'release)
> > > +                 commit
> > > +                 (patches '())
> > > +                 (extra-inputs '())
> > > +                 (extra-envars '()))
> > > + (package
> > > +   (name (jname "jsoftware-j" type))
> > > +   (version (jversion->string vername revision))
> > > +   (source
> > > +    (origin
> > > +      (method git-fetch)
> > > +      (uri (git-reference
> > > +            (url "https://github.com/jsoftware/jsource")
> > > +            (commit (or commit (jinfo->git-tag vername type
> > > revision))))
> > Vername sounds a little weird, make that version-base or something
> > clearer.  Also, the argument #:commit is used in an unclear fashion
> > --
> > if you were to pass an actual commit hash to it, it'd still be
> > treated
> > as a release and not be using git-version. 
> 
> Cool. I had a similar sense, but our ideas are a lot sharper than the
> ones I had. This actually prompted me to do some code cleanup,
> leveraging now-me who has a bit more Guile experience than past-me.
> At the very least, variable names should be more descriptive and
> consistent, overall.
Note, that the current version still ignores commit.  I've attached a
proof of concept patch, which fixes that, plus some other minor things,
but it now uses a pair for version.  Also it's untested, so treat with
care.


> Good idea. Upstream's build scripts condition a *lot* of behaviour on
> the j64avx environment variable, so it might not be straightforward,
> but I will put this on the to-do list of future improvements.
> 
> (Note, the code block quoted here got much simplified in the current
> patch.)
Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which also
provides you the flags that you need.

Cheers


[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 27231 bytes --]

From 93bd47856565af1407b662acfae8857821c571c8 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk [GNU_SYSTEM_MODULES]: Register jsoftware.
[dist_patch_DATA]: Add jsoftware-j901-f-fixes.patch.
---
 gnu/local.mk                                  |   2 +
 gnu/packages/jsoftware.scm                    | 459 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 +++
 3 files changed, 541 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index ef73f9237a..e69a69027a 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -325,6 +325,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/jemalloc.scm			\
   %D%/packages/jrnl.scm				\
   %D%/packages/jose.scm				\
+  %D%/packages/jsoftware.scm				\
   %D%/packages/julia.scm			\
   %D%/packages/julia-jll.scm			\
   %D%/packages/julia-xyz.scm			\
@@ -1290,6 +1291,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..51964dd5ab
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,459 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-71))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+(define* (jversion->string major #:optional minor revision commit)
+  "Return a version string formatted like MAJOR, MAJOR.MINOR, or
+MAJOR[.MINOR]-REVISION-COMMIT depending on the given parameters."
+  (match (list minor revision commit)
+    ((#f #f #f) major)
+    ((minor #f #f) (string-append major "." minor))
+    ((minor revision commit)
+     (string-append (if minor (string-append major "." minor) major)
+                    "-" revision "-" commit))))
+
+(define* (jrelease-string release-type #:optional minor)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if minor
+        (string-append release-type "-" minor)
+        release-type)))
+
+(define* (jinfo->git-tag major release-type #:optional minor)
+  "Given version parameters, construct a git tag for upstream releases."
+  (string-append "j" major (jrelease-string release-type minor)))
+
+(define (ijconsole)
+  "Generate a G-exp script that detects AVX/AVX2 support at runtime and
+executes jconsole with the appropriate libj.so and profile.ijs."
+  (program-file "ijconsole"
+    #~(begin
+        (use-modules ((ice-9 rdelim) #:select (read-line))
+                     ((ice-9 regex) #:select (regexp-match? string-match)))
+
+        ;; Assume that this script will be installed under bin/.
+        (define %basedir (dirname (dirname (current-filename))))
+
+        (define (cpu-feature-line? string)
+          (string-prefix? "flags" string))
+
+        (define (contains-word? word string)
+          (regexp-match?
+            (string-match (string-join `("\\<" ,word "\\>") "")
+                          string)))
+
+        (define (has-cpu-feature? feature)
+          (with-input-from-file "/proc/cpuinfo"
+             (lambda ()
+               (catch 'found
+                 (lambda ()
+                   (let loop ((line (read-line)))
+                     (cond ((eof-object? line) #f)
+                           ((and (cpu-feature-line? line)
+                                 (contains-word? feature line))
+                            (throw 'found))
+                           (else (loop (read-line))))))
+                 (const #t)))))
+
+        (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+               (libj (format #f "~a/lib/j/libj-~a.so" %basedir
+                             (cond ((has-cpu-feature? "avx2") "avx2")
+                                   ((has-cpu-feature? "avx") "avx")
+                                   (else ""))))
+               (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+          (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                 (cdr (command-line)))))))
+
+(define* (make-j #:key
+                 version
+                 revision
+                 hash
+                 commit
+                 tag
+                 (release-type 'release)
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+  (let ((major minor (if (pair? version)
+                         (car+cdr version)
+                         (values version ""))))
+    (package
+      (name (jname "jsoftware-j" release-type))
+      (version (jversion->string major minor revision commit))
+      (source
+       (origin
+         (method git-fetch)
+         (uri (git-reference
+               (url "https://github.com/jsoftware/jsource")
+               (commit (or
+                        commit
+                        tag (jinfo->git-tag major
+                                            release-type
+                                            minor)))))
+         (sha256 (base32 hash))
+         (file-name (git-file-name name version))
+         (patches patches)))
+      (build-system gnu-build-system)
+      (native-inputs
+       (list clang-toolchain))
+      (inputs (cons* libedit libomp (ijconsole) extra-inputs))
+      (arguments
+       `(#:modules (((ice-9 ftw) #:select (scandir))
+                    ((ice-9 popen) #:select (open-pipe* close-pipe))
+                    ((ice-9 regex) #:select (match:substring string-match))
+                    ((ice-9 threads) #:select (parallel par-for-each))
+                    ((srfi srfi-26) #:select (cut))
+                    ((srfi srfi-1) #:select (fold))
+                    ,@%gnu-build-system-modules)
+         #:phases
+         ;; Upstream's build system consists of ad-hoc scripts that build build up
+         ;; (very complicated) environment variables to pass to make.  The basic
+         ;; build process looks like this:
+         ;;
+         ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+         ;;   2) Set jplatform and j64x environment variables;
+         ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+         ;;
+         ;; However, upstream expects users to run J directly from the source
+         ;; directory; they do not supply a make `install' target.  Thus it takes
+         ;; some massaging to install files in FHS-style directories.
+         (modify-phases %standard-phases
+           ;; In particular, we have to set up
+           ;;
+           ;;   1) jsrc/jversion.h as in a typical build;
+           ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+           ;;   3) make2/build_*.sh to respect standard build conventions;
+           ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+           ;;   5) Hard coded references to addons directory.
+           (replace 'configure
+             (lambda* (#:key target inputs outputs #:allow-other-keys)
+               (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                      (clang (string-append clang-toolchain "/bin/clang"))
+                      (libedit (assoc-ref inputs "libedit"))
+                      (out (assoc-ref outputs "out")))
+                 ;; Set up build constants
+                 (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+                 (substitute* "jsrc/jversion.h"
+                   (("^#define jversion.*$")
+                    (format #f "#define jversion ~s\n" ,major))
+                   (("^#define jtype.*$")
+                    (format #f "#define jtype ~s\n"
+                            ,(jrelease-string release-type minor)))
+                   (("^#define jbuilder.*$")
+                    (format #f "#define jbuilder ~s\n" ,builder)))
+                 ;; Create profilex.ijs overrides to point to the correct store
+                 ;; items.  Note that we set ~install and ~addons directories to
+                 ;; reside under ~user to allow installing and loading addons.
+                 ;; TODO: Guix-ify J addons as well.
+                 (call-with-output-file "jlibrary/bin/profilex.ijs"
+                   (lambda (port)
+                     (display
+                      (string-join
+                       (list
+                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/') BINPATH"
+                        "system=. share,'/system'"
+                        "tools=. share,'/tools'"
+                        ;; Upstream defaults to spamming $HOME with unhidden
+                        ;; userdata directories.  Set this to be
+                        ;; $HOME/.config/j/<jversion> instead
+                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                        "jversion=. ({.~ i.&'-') jversion"
+                        "user=. home,'/.config/j/',jversion"
+                        "addons=. user,'/addons'"
+                        "break=. user,'/break'"
+                        "config=. user,'/config'"
+                        "install=. user,'/install'"
+                        "snap=. user,'/snap'"
+                        "temp=. user,'/temp'"
+                        "\n")
+                       "\n")
+                      port)))
+                 ;; Munge the build scripts into reason:
+                 ;; 1. Short-circuit the fragile compiler detection;
+                 ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+                 ;; 3. Propagate script errors to top level.
+                 (for-each
+                  (lambda (file)
+                    (with-directory-excursion "make2"
+                      (substitute* file
+                        ;; The `compiler' variable doesn't point to the actual
+                        ;; compiler.  It is just a switch to tell the build
+                        ;; scripts whether to use gcc- or clang-specific flags.
+                        (("^compiler=.*$") "compiler=clang\n")
+                        (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                        (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                         (string-append def "$CFLAGS " rest))
+                        (("^#!.*" shebang)
+                         (string-append shebang "set -o errexit\n")))))
+                  '("build_jconsole.sh" "build_libj.sh"))
+                 ;; The jconsole manually loads libedit with dlopen.  The path
+                 ;; must be absolute to correctly point to our input.
+                 (substitute* "jsrc/jconsole.c"
+                   (("libedit\\.so\\.[0-9]" so-file)
+                    (format #f "~a/lib/~a" libedit so-file)))
+                 ;; The ~addons/dev directory supplies tentative J-script
+                 ;; definitions of new J engine functionality.  Since we point
+                 ;; ~addons under the ~user directory, we move it under ~system
+                 ;; instead, which sits as-is in the output.
+                 (with-directory-excursion "jsrc"
+                   (for-each
+                    (lambda (file)
+                      (substitute* file (("~addons/dev") "~system/dev")))
+                    (scandir "."
+                             (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+                 ;; Implementation of 9!:14 records build time which breaks build
+                 ;; reproducibility.  Note that upstream code depends on the exact
+                 ;; format of these strings, so we need to mimic the standard.
+                 (substitute* "jsrc/j.c"
+                   (("__DATE__") "\"Jan 01 1970\"")
+                   (("__TIME__") "\"00:00:00\""))
+                 ;; Upstream recommends using clang, with GCC support being
+                 ;; second-class, often resulting in build failures.
+                 (setenv "CC" clang))))
+
+           ;; The build output depends primarily on the values of the `jplatform'
+           ;; and `j64x' environment variables.  If the target is ARM, then
+           ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+           ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+           ;; variants of libj are built.
+           ;;
+           ;; However, build targets are not fine-grained enough to distinguish
+           ;; between CPU features.  Thus we build and install all variants of
+           ;; libj, expecting jconsole to be called with a wrapper script that
+           ;; detects AVX features and loads the appropriate libj at runtime.
+           (replace 'build
+             (lambda _
+               (setenv "USE_OPENMP" "1")
+               (setenv "USE_THREAD" "1")
+               (for-each (lambda (var-val) (apply setenv var-val))
+                         (quote ,extra-envars))
+               ;; The build scripts assume that PWD is make2.
+               (with-directory-excursion "make2"
+                 (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                        (target-bit ,(if (target-64bit?) "64" "32"))
+                        (run (lambda* (script #:key (avx ""))
+                               (invoke "env"
+                                       (string-append "jplatform=" platform)
+                                       (string-append "j64x=j" target-bit avx)
+                                       script))))
+                   (parallel
+                    ;; Since jconsole doesn't depend on AVX features, we just
+                    ;; build it once.
+                    (run "./build_jconsole.sh")
+                    (run "./build_libj.sh")
+                    (if ,(target-64bit?)
+                        (parallel
+                         (run "./build_libj.sh" #:avx "avx")
+                         (run "./build_libj.sh" #:avx "avx2"))))))))
+           ;; The test suite is expected to be run as follows for each variant of
+           ;; libj that we build:
+           ;;
+           ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+           ;;
+           ;; This requires a working jconsole with accessible jlibrary files.  We
+           ;; simply place these all under test/bin.
+           (replace 'check
+             (lambda* (#:key tests? #:allow-other-keys)
+               (when tests?
+                 (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                   (mkdir-p "test/bin")
+                   (for-each
+                    (lambda (dir)
+                      (let ((source (string-append "jlibrary/" dir))
+                            (dest (string-append "test/bin/" dir)))
+                        (begin
+                          (mkdir-p dest)
+                          (copy-recursively source dest))))
+                    '("system" "tools" "addons"))
+                   ;; The jlibrary/dev directory only sometimes exists, but needs
+                   ;; to be copied into the ~system directory when it does.
+                   (for-each
+                    (lambda (dev-dir)
+                      (if (file-exists? dev-dir)
+                          (copy-recursively dev-dir "test/bin/system/dev")))
+                    '("jlibrary/dev" "jlibrary/addons/dev"))
+                   (par-for-each
+                    (lambda (dir)
+                      (let* ((bin (string-append "bin/" platform))
+                             (jbit ,(if (target-64bit?) "j64" "j32"))
+                             (jconsole (string-append bin "/" jbit
+                                                      "/jconsole"))
+                             (source (string-append bin "/" dir))
+                             (dest (string-append "test/bin/" dir)))
+                        (begin
+                          (mkdir-p dest)
+                          (copy-recursively source dest)
+                          (install-file "jlibrary/bin/profile.ijs" dest)
+                          (install-file jconsole dest)
+                          (let* ((jconsole (string-append dest "/jconsole"))
+                                 (tests "test/tsu.ijs")
+                                 (port (open-pipe* OPEN_WRITE jconsole tests)))
+                            (display "RUN ddall\n" port)
+                            (when (not (zero? (status:exit-val
+                                               (close-pipe port))))
+                              (error "Some J build tests failed."))))))
+                    (scandir (string-append "bin/" platform)
+                             (negate (cut member <> '("." "..")))))
+                   #t))))
+           ;; Now that everything is built, installation is fairly
+           ;; straightforward, following FHS conventions.  The only quirk is that
+           ;; we install jconsole under /libexec to make room for the wrapper
+           ;; replacement under /bin.
+           (replace 'install
+             (lambda* (#:key outputs inputs #:allow-other-keys)
+               (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                      (jbit ,(if (target-64bit?) "j64" "j32"))
+                      (ijconsole (assoc-ref inputs "ijconsole"))
+                      (jconsole (string-append "bin/" platform "/"
+                                               jbit "/jconsole"))
+                      (out (assoc-ref outputs "out"))
+                      (bin (string-append out "/bin"))
+                      (etc (string-append out "/etc/j"))
+                      (lib (string-append out "/lib/j"))
+                      (libexec (string-append out "/libexec/j"))
+                      (share (string-append out "/share/j"))
+                      (system (string-append share "/system"))
+                      (dev (string-append system "/dev")))
+                 (mkdir-p bin)
+                 (copy-file ijconsole (string-append bin "/ijconsole-"
+                                                     ,major))
+                 (mkdir-p lib)
+                 (for-each
+                  (lambda (jarch)
+                    (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                           (javx-match (string-match "avx.*" jarch))
+                           (javx (if (not javx-match) ""
+                                     (match:substring javx-match)))
+                           (sep (if javx-match "-" ""))
+                           (source (string-append jbin "/libj.so"))
+                           (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                      (copy-file source dest)))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                 (install-file jconsole libexec)
+                 (copy-recursively "jlibrary/system" system)
+                 (for-each
+                  (lambda (source-dev)
+                    (if (access? source-dev R_OK)
+                        (copy-recursively source-dev dev)))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                 (install-file "jlibrary/bin/profile.ijs" etc)
+                 (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+      (home-page "https://www.jsoftware.com/")
+      (synopsis "Ascii-only, array programming language in the APL family")
+      (description
+       "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+      (license license:gpl3+))))
+
+
+(define-public jsoftware-j-901
+  (make-j
+   #:version '("901" . "f")
+   #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+   #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  `(#:extra-inputs (,sleef)
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+         #:version '("902" . "b")
+         #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"
+         j-build-configuration-with-sleef))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+         #:version '("903" . "a")
+         #:tag "903-release-a"
+         #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"
+         j-build-configuration-with-sleef))
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 1439 bytes --]

Thanks for staying on top of this patch review.

Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:
> Hi,
> 
> Note, that the current version still ignores commit.  I've attached a
> proof of concept patch, which fixes that, plus some other minor things,
> but it now uses a pair for version.  Also it's untested, so treat with
> care.

Oh! I see. You're wanting the version string to have optional REVISION and
COMMIT parts, like `git-version'. That makes a lot of sense.

There was a small issue with `jversion->string' in your patch, which produces
MAJOR-REVISION-COMMIT if no MINOR value is provided. I want to make sure that
users can always have specifications like "jsoftware-j@MAJOR" in their profile
manifest, but guix doesn't treat hyphen as a version separator.

I just changed the logic to always keep a dot after MAJOR, and then use hyphens
after that.

> > Good idea. Upstream's build scripts condition a *lot* of behaviour on
> > the j64avx environment variable, so it might not be straightforward,
> > but I will put this on the to-do list of future improvements.
> > 
> > (Note, the code block quoted here got much simplified in the current
> > patch.)
> Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which also
> provides you the flags that you need.

Oh nice. A new, shiny module! This is exactly what I was looking for when
originally writing the script. Updated.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26901 bytes --]

From 13450733d99eafc3bc3fe84629224e653dd85753 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/jsoftware.scm                    | 452 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 ++++
 3 files changed, 533 insertions(+)
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index f72d2680af..b90295278a 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1293,6 +1293,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..6f5aeeb1cb
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,452 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-71))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+;; We want a version string where packages specifications like pkg@MAJOR work.
+;; This requires that the first version part separator be dot.  Subsequent
+;; separators are hyphen, mirror `git-version' etc.
+(define* (jversion->string major #:optional minor revision commit)
+  "Return a version string formatted like MAJOR.MINOR-REVISION-COMMIT.  Only
+  MAJOR is required, but ensure that the first separator is dot and subsequent
+  are hypen."
+  (let* ((commit (and commit (string-take commit 7)))
+         (sub-parts (filter (cut (compose not eq?) #f <>)
+                            (list minor revision commit))))
+    (match sub-parts
+      (() major)
+      (_  (string-append major "." (string-join sub-parts "-"))))))
+
+(define* (jrelease-string release-type #:optional version-minor)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if version-minor
+     (string-append release-type "-" version-minor)
+     release-type)))
+
+(define* (jinfo->git-tag version-major release-type #:optional version-minor)
+  "Given version parameters, construct a git tag for upstream releases."
+  (string-append "j" version-major (jrelease-string release-type version-minor)))
+
+(define (ijconsole)
+  "Generate a G-exp script that detects AVX/AVX2 support at runtime and
+  executes jconsole with the appropriate libj.so and profile.ijs."
+  (with-imported-modules '((guix cpu)
+                           (guix memoization)
+                           (guix profiling)
+                           (guix sets)
+                           (srfi srfi-26))
+    (program-file "ijconsole"
+      #~(begin
+          (use-modules ((guix cpu)     #:select (cpu-flags current-cpu))
+                       ((guix sets)    #:select (set-contains?))
+                       ((srfi srfi-26) #:select (cute)))
+
+          ;; Assume that this script will be installed under bin/.
+          (define %basedir (dirname (dirname (current-filename))))
+
+          (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+                 (cpu-has-flag?
+                   (cute set-contains? (cpu-flags (current-cpu)) <>))
+                 (libj (format #f "~a/lib/j/libj~a.so" %basedir
+                               (cond ((cpu-has-flag? "avx2") "-avx2")
+                                     ((cpu-has-flag? "avx") "-avx")
+                                     (else ""))))
+                 (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+            (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                   (cdr (command-line))))))))
+
+(define* (make-j #:key
+                 version
+                 revision
+                 hash
+                 tag
+                 commit
+                 (release-type 'release)
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+ (let* ((version-major version-minor (if (pair? version)
+                                       (car+cdr version)
+                                       (values version #f))))
+ (package
+   (name (jname "jsoftware-j" release-type))
+   (version (jversion->string version-major version-minor revision commit))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or commit tag
+                        (jinfo->git-tag version-major
+                                        release-type
+                                        version-minor)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
+   (inputs (cons* `("libedit" ,libedit)
+                  `("libomp" ,libomp)
+                  `("ijconsole" ,(ijconsole))
+                  extra-inputs))
+   (arguments
+    `(#:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version-major))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n"
+                         ,(jrelease-string release-type version-minor)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Create profilex.ijs overrides to point to the correct store
+              ;; items.  Note that we set ~install and ~addons directories to
+              ;; reside under ~user to allow installing and loading addons.
+              ;; TODO: Guix-ify J addons as well.
+              (call-with-output-file "jlibrary/bin/profilex.ijs"
+                (lambda (port)
+                  (display
+                    (string-join
+                      (list
+                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/') BINPATH"
+                        "system=. share,'/system'"
+                        "tools=. share,'/tools'"
+                        ;; Upstream defaults to spamming $HOME with unhidden
+                        ;; userdata directories.  Set this to be
+                        ;; $HOME/.config/j/<jversion> instead
+                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''"
+                        "jversion=. ({.~ i.&'-') jversion"
+                        "user=. home,'/.config/j/',jversion"
+                        "addons=. user,'/addons'"
+                        "break=. user,'/break'"
+                        "config=. user,'/config'"
+                        "install=. user,'/install'"
+                        "snap=. user,'/snap'"
+                        "temp=. user,'/temp'"
+                        "\n")
+                      "\n")
+                    port)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (run (lambda* (script #:key (avx ""))
+                            (invoke "env"
+                                    (string-append "jplatform=" platform)
+                                    (string-append "j64x=j" target-bit avx)
+                                    script))))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (run "./build_jconsole.sh")
+                  (run "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (run "./build_libj.sh" #:avx "avx")
+                      (run "./build_libj.sh" #:avx "avx2"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (file-exists? dev-dir)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((bin (string-append "bin/" platform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append bin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append bin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jconsole (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jconsole tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (ijconsole (assoc-ref inputs "ijconsole"))
+                   (jconsole (string-append "bin/" platform "/"
+                                            jbit "/jconsole"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file ijconsole (string-append bin "/ijconsole-"
+                                                  ,version-major))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" platform)
+                         (negate (cut member <> '("." "..")))))
+              (install-file jconsole libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (install-file "jlibrary/bin/profilex.ijs" etc)))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+))))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:version '("901" . "f")
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("902" . "b")
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("903" . "a")
+        #:tag "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Hi,

Am Samstag, dem 15.01.2022 um 23:05 +0900 schrieb
elaexuotee@wilsonb.com:
> Thanks for staying on top of this patch review.
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:
> > Hi,
> > 
> > Note, that the current version still ignores commit.  I've attached
> > a proof of concept patch, which fixes that, plus some other minor
> > things, but it now uses a pair for version.  Also it's untested, so
> > treat with care.
> 
> Oh! I see. You're wanting the version string to have optional
> REVISION and COMMIT parts, like `git-version'. That makes a lot of
> sense.
> 
> There was a small issue with `jversion->string' in your patch, which
> produces MAJOR-REVISION-COMMIT if no MINOR value is provided. I want
> to make sure that users can always have specifications like
> "jsoftware-j@MAJOR" in their profile manifest, but guix doesn't treat
> hyphen as a version separator.
> 
> I just changed the logic to always keep a dot after MAJOR, and then
> use hyphens after that.
I'm not quite sure if I agree with that decision, but you're right that
@MAJOR ought to be supported.  How about enforcing that MINOR exists if
REVISION and COMMIT are used and setting it to "0" if there hasn't been
an "a" beta or release yet?

> > 
> > > Good idea. Upstream's build scripts condition a *lot* of
> > > behaviour on he j64avx environment variable, so it might not be
> > > straightforward, but I will put this on the to-do list of future
> > > improvements.
> > > 
> > > (Note, the code block quoted here got much simplified in the
> > > current patch.)
> > Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which
> > also provides you the flags that you need.
> 
> Oh nice. A new, shiny module! This is exactly what I was looking for
> when originally writing the script. Updated.
LGTM.

> +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> +   (inputs (cons* `("libedit" ,libedit)
> +                  `("libomp" ,libomp)
> +                  `("ijconsole" ,(ijconsole))
> +                  extra-inputs))
My variant already had these translated to the new style.  If you're
not using old style on purpose – and I don't think you do – you might
want to make your life easier by dropping these labels and going with
plain (list)s.

> +              (call-with-output-file "jlibrary/bin/profilex.ijs"
> +                (lambda (port)
> +                  (display
> +                    (string-join
> +                      (list
> +                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/')
> BINPATH"
> +                        "system=. share,'/system'"
> +                        "tools=. share,'/tools'"
> +                        ;; Upstream defaults to spamming $HOME with
> unhidden
> +                        ;; userdata directories.  Set this to be
> +                        ;; $HOME/.config/j/<jversion> instead
> +                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/'
> 9!:14''"
> +                        "jversion=. ({.~ i.&'-') jversion"
> +                        "user=. home,'/.config/j/',jversion"
> +                        "addons=. user,'/addons'"
> +                        "break=. user,'/break'"
> +                        "config=. user,'/config'"
> +                        "install=. user,'/install'"
> +                        "snap=. user,'/snap'"
> +                        "temp=. user,'/temp'"
> +                        "\n")
> +                      "\n")
> +                    port)))
You might want to use an aux-file for that.  Name it
jsoftware/profile.ijs if it's just a plain file (which I assume).  I
recently pushed a commit towards renpy which replaces a large format
block by fetching an auxiliary file and substitute*, which you can take
as reference if you're unsure.  Note that renpy still uses old-style
inputs, so the assoc-ref makes sense in that case; however, you should
be able to also express this pattern in terms of search-input-file
somehow (or otherwise express it as gexp).

Cheers

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

Good day.

> > I just changed the logic to always keep a dot after MAJOR, and then
> > use hyphens after that.
> I'm not quite sure if I agree with that decision, but you're right that
> @MAJOR ought to be supported.  How about enforcing that MINOR exists if
> REVISION and COMMIT are used and setting it to "0" if there hasn't been
> an "a" beta or release yet?

Interesting idea. How about just always forcing a MINOR part, setting to "0" if
upstream doesn't have one?

> > +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> > +   (inputs (cons* `("libedit" ,libedit)
> > +                  `("libomp" ,libomp)
> > +                  `("ijconsole" ,(ijconsole))
> > +                  extra-inputs))
> My variant already had these translated to the new style.  If you're
> not using old style on purpose – and I don't think you do – you might
> want to make your life easier by dropping these labels and going with
> plain (list)s.

Yeah, I had trouble getting that to work nicely with the ijconsole input. See
below for more details.

> You might want to use an aux-file for that.  Name it
> jsoftware/profile.ijs if it's just a plain file (which I assume).  I
> recently pushed a commit towards renpy which replaces a large format
> block by fetching an auxiliary file and substitute*, which you can take
> as reference if you're unsure.  Note that renpy still uses old-style
> inputs, so the assoc-ref makes sense in that case; however, you should
> be able to also express this pattern in terms of search-input-file
> somehow (or otherwise express it as gexp).

Oooh. Neat. That makes adding and testing changes their much nicer.

Note, I wasn't able to find a nice way to get this working together with the
new inputs syntax, since `search-input-files' only searches under input paths
which are directories.

The ijconsole and profilex input are regular files, so it raises a search-error.
One workaronud is to put ijconsole and profilex together under a file-union
input, but I found that almost worse than just using old-style syntax.

If you have a better idea, I am all ears.


Cheers!



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26490 bytes --]

From 49d44631c736bb45fe5f32ee446e7af506f5145c Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/aux-files/jsoftware/profilex.ijs: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/aux-files/jsoftware/profilex.ijs |  14 +
 gnu/packages/jsoftware.scm                    | 428 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 ++++
 4 files changed, 523 insertions(+)
 create mode 100644 gnu/packages/aux-files/jsoftware/profilex.ijs
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 3335d368df..fdbcb364c7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1293,6 +1293,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/aux-files/jsoftware/profilex.ijs b/gnu/packages/aux-files/jsoftware/profilex.ijs
new file mode 100644
index 0000000000..30e0d229e2
--- /dev/null
+++ b/gnu/packages/aux-files/jsoftware/profilex.ijs
@@ -0,0 +1,14 @@
+'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''
+basedir=. ({.~ _2 { I.@:=&'/') BINPATH
+
+share=.  basedir,'/share/j'
+system=. share,'/system'
+tools=.  share,'/tools'
+
+user=.    home,'/.config/j/',jversion
+addons=.  user,'/addons'
+break=.   user,'/break'
+config=.  user,'/config'
+install=. user,'/install'
+snap=.    user,'/snap'
+temp=.    user,'/temp'
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..a28e786123
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,428 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-71))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+;; We want a version string where packages specifications like pkg@MAJOR work.
+;; This requires that the first version part separator be dot.  Subsequent
+;; separators are hyphen, mirror `git-version' etc.
+(define* (jversion->string major #:optional minor revision commit)
+  "Return a version string formatted like MAJOR.MINOR-REVISION-COMMIT.  Only
+  MAJOR is required, and MINOR defaults to ``0'' if not supplied."
+  (let* ((commit (and commit (string-take commit 7)))
+         (minor (or minor "0"))
+         (sub-parts (filter (cut (compose not eq?) #f <>)
+                            (list minor revision commit))))
+    (string-append major "." (string-join sub-parts "-"))))
+
+(define* (jrelease-string release-type #:optional version-minor)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if version-minor
+     (string-append release-type "-" version-minor)
+     release-type)))
+
+(define* (jinfo->git-tag version-major release-type #:optional version-minor)
+  "Given version parameters, construct a git tag for upstream releases."
+  (string-append "j" version-major (jrelease-string release-type version-minor)))
+
+;; G-exp script that detects AVX/AVX2 support at runtime and executes jconsole
+;; with the appropriate libj.so and profile.ijs."
+(define ijconsole
+  (with-imported-modules '((guix cpu)
+                           (guix memoization)
+                           (guix profiling)
+                           (guix sets)
+                           (srfi srfi-26))
+    (program-file "ijconsole"
+      #~(begin
+          (use-modules ((guix cpu)     #:select (cpu-flags current-cpu))
+                       ((guix sets)    #:select (set-contains?))
+                       ((srfi srfi-26) #:select (cute)))
+
+          ;; Assume that this script will be installed under bin/.
+          (define %basedir (dirname (dirname (current-filename))))
+
+          (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+                 (cpu-has-flag?
+                   (cute set-contains? (cpu-flags (current-cpu)) <>))
+                 (libj (format #f "~a/lib/j/libj~a.so" %basedir
+                               (cond ((cpu-has-flag? "avx2") "-avx2")
+                                     ((cpu-has-flag? "avx") "-avx")
+                                     (else ""))))
+                 (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+            (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                   (cdr (command-line))))))))
+
+(define* (make-j #:key
+                 version
+                 revision
+                 hash
+                 tag
+                 commit
+                 (release-type 'release)
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+ (let* ((version-major version-minor (if (pair? version)
+                                       (car+cdr version)
+                                       (values version #f))))
+ (package
+   (name (jname "jsoftware-j" release-type))
+   (version (jversion->string version-major version-minor revision commit))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or commit tag
+                        (jinfo->git-tag version-major
+                                        release-type
+                                        version-minor)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs (list clang-toolchain))
+   (inputs (cons* `("libedit" ,libedit)
+                  `("libomp" ,libomp)
+                  `("ijconsole" ,ijconsole)
+                   ;; profilex.ijs overrides ~install and ~addons
+                   ;; directories to reside under the user-writable ~user.
+                   ;; This allows local-install of addons via pacman.  TODO:
+                   ;; Guix-ify J addons as well.
+                  `("profilex" ,(search-auxiliary-file
+                                  "jsoftware/profilex.ijs"))
+                  extra-inputs))
+   (arguments
+    `(#:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version-major))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n"
+                         ,(jrelease-string release-type version-minor)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (run (lambda* (script #:key (avx ""))
+                            (invoke "env"
+                                    (string-append "jplatform=" platform)
+                                    (string-append "j64x=j" target-bit avx)
+                                    script))))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (run "./build_jconsole.sh")
+                  (run "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (run "./build_libj.sh" #:avx "avx")
+                      (run "./build_libj.sh" #:avx "avx2"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (file-exists? dev-dir)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((bin (string-append "bin/" platform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append bin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append bin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jconsole (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jconsole tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file (assoc-ref inputs "ijconsole")
+                         (string-append bin "/ijconsole-" ,version-major))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" platform)
+                         (negate (cut member <> '("." "..")))))
+              (install-file (string-append "bin/" platform "/" jbit "/jconsole")
+                            libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (copy-file (assoc-ref inputs "profilex")
+                         (string-append etc "/profilex.ijs"))))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+))))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:version '("901" . "f")
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("902" . "b")
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("903" . "a")
+        #:tag "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Am Sonntag, dem 16.01.2022 um 14:29 +0900 schrieb
elaexuotee@wilsonb.com:
> Good day.
> 
> > 
> > > I just changed the logic to always keep a dot after MAJOR, and
> > > then use hyphens after that.
> > I'm not quite sure if I agree with that decision, but you're right
> > that @MAJOR ought to be supported.  How about enforcing that MINOR
> > exists if REVISION and COMMIT are used and setting it to "0" if
> > there hasn't been an "a" beta or release yet?
> 
> Interesting idea. How about just always forcing a MINOR part, setting
> to "0" if upstream doesn't have one?
That'd declare regular releases as MAJOR.0 in the version field, which
I'm not sure if we want that.  In the case of random commits I'm less
reserved, as they don't correspond to releases anyway.

> > 
> > > +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> > > +   (inputs (cons* `("libedit" ,libedit)
> > > +                  `("libomp" ,libomp)
> > > +                  `("ijconsole" ,(ijconsole))
> > > +                  extra-inputs))
> > My variant already had these translated to the new style.  If
> > you're not using old style on purpose – and I don't think you do –
> > you might want to make your life easier by dropping these labels
> > and going with plain (list)s.
> 
> Yeah, I had trouble getting that to work nicely with the ijconsole
> input. See below for more details.
> 
> > You might want to use an aux-file for that.  Name it
> > jsoftware/profile.ijs if it's just a plain file (which I assume). 
> > I recently pushed a commit towards renpy which replaces a large
> > format block by fetching an auxiliary file and substitute*, which
> > you can take as reference if you're unsure.  Note that renpy still
> > uses old-style inputs, so the assoc-ref makes sense in that case;
> > however, you should be able to also express this pattern in terms
> > of search-input-file somehow (or otherwise express it as gexp).
> 
> Oooh. Neat. That makes adding and testing changes their much nicer.
> 
> Note, I wasn't able to find a nice way to get this working together
> with the new inputs syntax, since `search-input-files' only searches
> under input paths which are directories.
> 
> The ijconsole and profilex input are regular files, so it raises a
> search-error.  One workaronud is to put ijconsole and profilex
> together under a file-union input, but I found that almost worse than
> just using old-style syntax.
> 
> If you have a better idea, I am all ears.
You could define that file-union right after ijconsole.  If you want to
golf even more, you could define ijconsole inside that file-union, i.e.
  (define jsoftware-aux-files
    (file-union "jsoftware-aux-files"
      `(("profile.ijs" ,(search-aux-file ...)
        ("ijconsole" ,(program-file ...))))

I'm not quite sure if you want to use jsoftware-aux-files directly as
input or whether it's wiser to stuff it into another union like 
(file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
search-input-file will probably do the right thing regardless.
The new style should also still work with assoc-ref, it'd just be
weirder to look at.  Lastly, you could code up a (search-file-input)
just in case; I'm not sure if we have one already.

Cheers

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

> > Interesting idea. How about just always forcing a MINOR part, setting
> > to "0" if upstream doesn't have one?
> That'd declare regular releases as MAJOR.0 in the version field, which
> I'm not sure if we want that.  In the case of random commits I'm less
> reserved, as they don't correspond to releases anyway.

I see your point. In fact, upstream releases start with MINOR part "a" and
"count up" through the letters over the course of a year. It's a pretty simple
scheme. For example, J 901 started at "j901-release-a" and ended at
"j901-release-f".

When I originally wrote this package, I noticed that the release tag for the
first J 902 was "j902-release", and so treaded MINOR as potentially optional.
However, after checking upstream's forums, this looks to just be a git tag
mishap.

How about we just go ahead and treat MINOR as mandatory as well?


> > If you have a better idea, I am all ears.
> You could define that file-union right after ijconsole.  If you want to
> golf even more, you could define ijconsole inside that file-union, i.e.
>   (define jsoftware-aux-files
>     (file-union "jsoftware-aux-files"
>       `(("profile.ijs" ,(search-aux-file ...)
>         ("ijconsole" ,(program-file ...))))
> 
> I'm not quite sure if you want to use jsoftware-aux-files directly as
> input or whether it's wiser to stuff it into another union like 
> (file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
> search-input-file will probably do the right thing regardless.
> The new style should also still work with assoc-ref, it'd just be
> weirder to look at.  Lastly, you could code up a (search-file-input)
> just in case; I'm not sure if we have one already.

The file-union seems like a cludgy workaround to me. What we really want is an
easy, direct way to get handles on the input files. Heck, program-file objects
already have a name property; why can't we use that? Attached patches are a
proof-of-concept.

That said, if this is going to turn into a big rabbit hole, can we just munge
the J package inputs into whatever you think is best?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-packages-Expand-range-of-objects-add-input-label-can.patch --]
[-- Type: text/x-patch, Size: 2711 bytes --]

From 3b8e7fa8fbd58e7e164e3730c708419f612b8549 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Sun, 16 Jan 2022 23:54:51 +0900
Subject: [PATCH 1/2] packages: Expand range of objects 'add-input-label' can
 label
To: guix-patches@gnu.org

* guix/packages.scm (%auxiliary-files-subpath-dir): New variable.
(add-input-label): Support labels from the name property of
objects that have one.  Also, name auxiliary files from their
subpath.
---
 guix/packages.scm | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index 9d5b23eb8a..4feea8ad5f 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -7,6 +7,7 @@
 ;;; Copyright © 2019 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2020, 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -569,6 +570,10 @@ (define-record-type* <package>
                        (default (current-definition-location))
                        (innate)))
 
+;; Note: This morally imports from gnu/packages.scm, but since they import us,
+;; we define here instead.
+(define %auxiliary-files-subdir-path "/gnu/packages/aux-files")
+
 (define (add-input-label input)
   "Add an input label to INPUT."
   (match input
@@ -576,7 +581,24 @@ (define (add-input-label input)
      (list (package-name package) package))
     (((? package? package) output)                ;XXX: ugly?
      (list (package-name package) package output))
-    ((? gexp-input?)       ;XXX: misplaced because 'native?' field is ignored?
+    ((? local-file? local-file)
+     (list (local-file-name local-file) local-file))
+    ((? plain-file? plain-file)
+     (list (plain-file-name plain-file) plain-file))
+    ((? computed-file? computed-file)
+     (list (computed-file-name computed-file) computed-file))
+    ((? program-file? program-file)
+     (list (program-file-name program-file) program-file))
+    ((? scheme-file? scheme-file)
+     (list (scheme-file-name scheme-file) scheme-file))
+    ((? string? path)
+     (let* ((regex (string-append %auxiliary-files-subdir-path "/(.*)"))
+            (match (string-match regex input)))
+       `(,(if match
+               (match:substring match 1)
+               "_")
+          ,input)))
+    ((? gexp-input?)      ;XXX: misplaced because 'native?' field is ignored?
      (let ((obj    (gexp-input-thing input))
            (output (gexp-input-output input)))
        `(,(if (package? obj)
-- 
2.34.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26330 bytes --]

From d32fd0cb243f6b51a9b2c178279a19015b621df2 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH 2/2] gnu: Add j.
To: guix-patches@gnu.org

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/aux-files/jsoftware/profilex.ijs: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk: Register it.
---
 gnu/local.mk                                  |   1 +
 gnu/packages/aux-files/jsoftware/profilex.ijs |  14 +
 gnu/packages/jsoftware.scm                    | 424 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 ++++
 4 files changed, 519 insertions(+)
 create mode 100644 gnu/packages/aux-files/jsoftware/profilex.ijs
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 3335d368df..fdbcb364c7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1293,6 +1293,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/aux-files/jsoftware/profilex.ijs b/gnu/packages/aux-files/jsoftware/profilex.ijs
new file mode 100644
index 0000000000..30e0d229e2
--- /dev/null
+++ b/gnu/packages/aux-files/jsoftware/profilex.ijs
@@ -0,0 +1,14 @@
+'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''
+basedir=. ({.~ _2 { I.@:=&'/') BINPATH
+
+share=.  basedir,'/share/j'
+system=. share,'/system'
+tools=.  share,'/tools'
+
+user=.    home,'/.config/j/',jversion
+addons=.  user,'/addons'
+break=.   user,'/break'
+config=.  user,'/config'
+install=. user,'/install'
+snap=.    user,'/snap'
+temp=.    user,'/temp'
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..e907f03d24
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,424 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-71))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+;; We want a version string where packages specifications like pkg@MAJOR work.
+;; This requires that the first version part separator be dot.  Subsequent
+;; separators are hyphen, mirror `git-version' etc.
+(define* (jversion->string major minor #:optional revision commit)
+  "Return a version string formatted like MAJOR.MINOR-REVISION-COMMIT.  Only
+  MAJOR is required, and MINOR defaults to ``0'' if not supplied."
+  (let* ((commit (and commit (string-take commit 7)))
+         (sub-parts (filter (cut (compose not eq?) #f <>)
+                            (list minor revision commit))))
+    (string-append major "." (string-join sub-parts "-"))))
+
+(define* (jrelease-string release-type #:optional version-minor)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if version-minor
+     (string-append release-type "-" version-minor)
+     release-type)))
+
+(define* (jinfo->git-tag version-major release-type #:optional version-minor)
+  "Given version parameters, construct a git tag for upstream releases."
+  (string-append "j" version-major (jrelease-string release-type version-minor)))
+
+;; G-exp script that detects AVX/AVX2 support at runtime and executes jconsole
+;; with the appropriate libj.so and profile.ijs."
+(define ijconsole
+  (with-imported-modules '((guix cpu)
+                           (guix memoization)
+                           (guix profiling)
+                           (guix sets)
+                           (srfi srfi-26))
+    (program-file "ijconsole"
+      #~(begin
+          (use-modules ((guix cpu)     #:select (cpu-flags current-cpu))
+                       ((guix sets)    #:select (set-contains?))
+                       ((srfi srfi-26) #:select (cute)))
+
+          ;; Assume that this script will be installed under bin/.
+          (define %basedir (dirname (dirname (current-filename))))
+
+          (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+                 (cpu-has-flag?
+                   (cute set-contains? (cpu-flags (current-cpu)) <>))
+                 (libj (format #f "~a/lib/j/libj~a.so" %basedir
+                               (cond ((cpu-has-flag? "avx2") "-avx2")
+                                     ((cpu-has-flag? "avx") "-avx")
+                                     (else ""))))
+                 (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+            (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                   (cdr (command-line))))))))
+
+(define* (make-j #:key
+                 version
+                 revision
+                 hash
+                 tag
+                 commit
+                 (release-type 'release)
+                 (patches '())
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+ (let* ((version-major version-minor (if (pair? version)
+                                       (car+cdr version)
+                                       (values version #f))))
+ (package
+   (name (jname "jsoftware-j" release-type))
+   (version (jversion->string version-major version-minor revision commit))
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://github.com/jsoftware/jsource")
+            (commit (or commit tag
+                        (jinfo->git-tag version-major
+                                        release-type
+                                        version-minor)))))
+      (sha256 (base32 hash))
+      (file-name (git-file-name name version))
+      (patches patches)))
+   (build-system gnu-build-system)
+   (native-inputs (list clang-toolchain))
+   (inputs (cons* libedit libomp ijconsole
+                   ;; profilex.ijs overrides ~install and ~addons
+                   ;; directories to reside under the user-writable ~user.
+                   ;; This allows local-install of addons via pacman.  TODO:
+                   ;; Guix-ify J addons as well.
+                  (search-auxiliary-file "jsoftware/profilex.ijs")
+                  extra-inputs))
+   (arguments
+    `(#:modules (((ice-9 ftw) #:select (scandir))
+                 ((ice-9 popen) #:select (open-pipe* close-pipe))
+                 ((ice-9 regex) #:select (match:substring string-match))
+                 ((ice-9 threads) #:select (parallel par-for-each))
+                 ((srfi srfi-26) #:select (cut))
+                 ((srfi srfi-1) #:select (fold))
+                 ,@%gnu-build-system-modules)
+      #:phases
+      ;; Upstream's build system consists of ad-hoc scripts that build build up
+      ;; (very complicated) environment variables to pass to make.  The basic
+      ;; build process looks like this:
+      ;;
+      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+      ;;   2) Set jplatform and j64x environment variables;
+      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+      ;;
+      ;; However, upstream expects users to run J directly from the source
+      ;; directory; they do not supply a make `install' target.  Thus it takes
+      ;; some massaging to install files in FHS-style directories.
+      (modify-phases %standard-phases
+        ;; In particular, we have to set up
+        ;;
+        ;;   1) jsrc/jversion.h as in a typical build;
+        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+        ;;   3) make2/build_*.sh to respect standard build conventions;
+        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+        ;;   5) Hard coded references to addons directory.
+        (replace 'configure
+          (lambda* (#:key target inputs outputs #:allow-other-keys)
+            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                   (clang (string-append clang-toolchain "/bin/clang"))
+                   (libedit (assoc-ref inputs "libedit"))
+                   (out (assoc-ref outputs "out")))
+              ;; Set up build constants
+              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+              (substitute* "jsrc/jversion.h"
+                (("^#define jversion.*$")
+                 (format #f "#define jversion ~s\n" ,version-major))
+                (("^#define jtype.*$")
+                 (format #f "#define jtype ~s\n"
+                         ,(jrelease-string release-type version-minor)))
+                (("^#define jbuilder.*$")
+                 (format #f "#define jbuilder ~s\n" ,builder)))
+              ;; Munge the build scripts into reason:
+              ;; 1. Short-circuit the fragile compiler detection;
+              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+              ;; 3. Propagate script errors to top level.
+              (for-each
+               (lambda (file)
+                 (with-directory-excursion "make2"
+                   (substitute* file
+                     ;; The `compiler' variable doesn't point to the actual
+                     ;; compiler.  It is just a switch to tell the build
+                     ;; scripts whether to use gcc- or clang-specific flags.
+                     (("^compiler=.*$") "compiler=clang\n")
+                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                      (string-append def "$CFLAGS " rest))
+                     (("^#!.*" shebang)
+                      (string-append shebang "set -o errexit\n")))))
+                 '("build_jconsole.sh" "build_libj.sh"))
+              ;; The jconsole manually loads libedit with dlopen.  The path
+              ;; must be absolute to correctly point to our input.
+              (substitute* "jsrc/jconsole.c"
+                (("libedit\\.so\\.[0-9]" so-file)
+                 (format #f "~a/lib/~a" libedit so-file)))
+              ;; The ~addons/dev directory supplies tentative J-script
+              ;; definitions of new J engine functionality.  Since we point
+              ;; ~addons under the ~user directory, we move it under ~system
+              ;; instead, which sits as-is in the output.
+              (with-directory-excursion "jsrc"
+                (for-each
+                  (lambda (file)
+                    (substitute* file (("~addons/dev") "~system/dev")))
+                  (scandir "."
+                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+              ;; Implementation of 9!:14 records build time which breaks build
+              ;; reproducibility.  Note that upstream code depends on the exact
+              ;; format of these strings, so we need to mimic the standard.
+              (substitute* "jsrc/j.c"
+                (("__DATE__") "\"Jan 01 1970\"")
+                (("__TIME__") "\"00:00:00\""))
+              ;; Upstream recommends using clang, with GCC support being
+              ;; second-class, often resulting in build failures.
+              (setenv "CC" clang))))
+
+        ;; The build output depends primarily on the values of the `jplatform'
+        ;; and `j64x' environment variables.  If the target is ARM, then
+        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
+        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
+        ;; variants of libj are built.
+        ;;
+        ;; However, build targets are not fine-grained enough to distinguish
+        ;; between CPU features.  Thus we build and install all variants of
+        ;; libj, expecting jconsole to be called with a wrapper script that
+        ;; detects AVX features and loads the appropriate libj at runtime.
+        (replace 'build
+          (lambda _
+            (setenv "USE_OPENMP" "1")
+            (setenv "USE_THREAD" "1")
+            (for-each (lambda (var-val) (apply setenv var-val))
+                      (quote ,extra-envars))
+            ;; The build scripts assume that PWD is make2.
+            (with-directory-excursion "make2"
+              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                     (target-bit ,(if (target-64bit?) "64" "32"))
+                     (run (lambda* (script #:key (avx ""))
+                            (invoke "env"
+                                    (string-append "jplatform=" platform)
+                                    (string-append "j64x=j" target-bit avx)
+                                    script))))
+                (parallel
+                  ;; Since jconsole doesn't depend on AVX features, we just
+                  ;; build it once.
+                  (run "./build_jconsole.sh")
+                  (run "./build_libj.sh")
+                  (if ,(target-64bit?)
+                    (parallel
+                      (run "./build_libj.sh" #:avx "avx")
+                      (run "./build_libj.sh" #:avx "avx2"))))))))
+        ;; The test suite is expected to be run as follows for each variant of
+        ;; libj that we build:
+        ;;
+        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+        ;;
+        ;; This requires a working jconsole with accessible jlibrary files.  We
+        ;; simply place these all under test/bin.
+        (replace 'check
+          (lambda* (#:key tests? #:allow-other-keys)
+            (when tests?
+              (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                (mkdir-p "test/bin")
+                (for-each
+                  (lambda (dir)
+                    (let ((source (string-append "jlibrary/" dir))
+                          (dest (string-append "test/bin/" dir)))
+                    (begin
+                      (mkdir-p dest)
+                      (copy-recursively source dest))))
+                  '("system" "tools" "addons"))
+                ;; The jlibrary/dev directory only sometimes exists, but needs
+                ;; to be copied into the ~system directory when it does.
+                (for-each
+                  (lambda (dev-dir)
+                    (if (file-exists? dev-dir)
+                      (copy-recursively dev-dir "test/bin/system/dev")))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                (par-for-each
+                  (lambda (dir)
+                    (let* ((bin (string-append "bin/" platform))
+                           (jbit ,(if (target-64bit?) "j64" "j32"))
+                           (jconsole (string-append bin "/" jbit
+                                                    "/jconsole"))
+                           (source (string-append bin "/" dir))
+                           (dest (string-append "test/bin/" dir)))
+                      (begin
+                        (mkdir-p dest)
+                        (copy-recursively source dest)
+                        (install-file "jlibrary/bin/profile.ijs" dest)
+                        (install-file jconsole dest)
+                        (let* ((jconsole (string-append dest "/jconsole"))
+                               (tests "test/tsu.ijs")
+                               (port (open-pipe* OPEN_WRITE jconsole tests)))
+                          (display "RUN ddall\n" port)
+                          (when (not (zero? (status:exit-val
+                                              (close-pipe port))))
+                            (error "Some J build tests failed."))))))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                #t))))
+        ;; Now that everything is built, installation is fairly
+        ;; straightforward, following FHS conventions.  The only quirk is that
+        ;; we install jconsole under /libexec to make room for the wrapper
+        ;; replacement under /bin.
+        (replace 'install
+          (lambda* (#:key outputs inputs #:allow-other-keys)
+            (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                   (jbit ,(if (target-64bit?) "j64" "j32"))
+                   (out (assoc-ref outputs "out"))
+                   (bin (string-append out "/bin"))
+                   (etc (string-append out "/etc/j"))
+                   (lib (string-append out "/lib/j"))
+                   (libexec (string-append out "/libexec/j"))
+                   (share (string-append out "/share/j"))
+                   (system (string-append share "/system"))
+                   (dev (string-append system "/dev")))
+              (mkdir-p bin)
+              (copy-file (assoc-ref inputs "ijconsole")
+                         (string-append bin "/ijconsole-" ,version-major))
+              (mkdir-p lib)
+              (for-each
+                (lambda (jarch)
+                  (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                         (javx-match (string-match "avx.*" jarch))
+                         (javx (if (not javx-match) ""
+                                 (match:substring javx-match)))
+                         (sep (if javx-match "-" ""))
+                         (source (string-append jbin "/libj.so"))
+                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                    (copy-file source dest)))
+                (scandir (string-append "bin/" platform)
+                         (negate (cut member <> '("." "..")))))
+              (install-file (string-append "bin/" platform "/" jbit "/jconsole")
+                            libexec)
+              (copy-recursively "jlibrary/system" system)
+              (for-each
+                (lambda (source-dev)
+                  (if (access? source-dev R_OK)
+                    (copy-recursively source-dev dev)))
+                '("jlibrary/dev" "jlibrary/addons/dev"))
+              (install-file "jlibrary/bin/profile.ijs" etc)
+              (copy-file (assoc-ref inputs "jsoftware/profilex.ijs")
+                         (string-append etc "/profilex.ijs"))))))))
+   (home-page "https://www.jsoftware.com/")
+   (synopsis "Ascii-only, array programming language in the APL family")
+   (description
+    "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+   (license license:gpl3+))))
+
+
+(define-public jsoftware-j-901
+  (make-j
+    #:version '("901" . "f")
+    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  `(#:extra-inputs (,sleef)
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("902" . "b")
+        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))
+
+
+(define-public jsoftware-j-903
+  (apply make-j
+    (append j-build-configuration-with-sleef
+      `(#:version ,'("903" . "a")
+        #:tag "903-release-a"
+        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))
+
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Hi,

Am Montag, dem 17.01.2022 um 00:19 +0900 schrieb
elaexuotee@wilsonb.com:
> > > Interesting idea. How about just always forcing a MINOR part,
> > > setting to "0" if upstream doesn't have one?
> > That'd declare regular releases as MAJOR.0 in the version field,
> > which I'm not sure if we want that.  In the case of random commits
> > I'm less reserved, as they don't correspond to releases anyway.
> 
> I see your point. In fact, upstream releases start with MINOR part "a"
> and "count up" through the letters over the course of a year. It's a
> pretty simple scheme. For example, J 901 started at "j901-release-a"
> and ended at "j901-release-f".
> 
> When I originally wrote this package, I noticed that the release tag
> for the first J 902 was "j902-release", and so treaded MINOR as
> potentially optional.
> However, after checking upstream's forums, this looks to just be a git
> tag mishap.
> 
> How about we just go ahead and treat MINOR as mandatory as well?
In other words minor should always have a value >= "a" and 902 was just
missing it?  Fair enough, in that case making version mandatory sounds
like the way to go.

> > > If you have a better idea, I am all ears.
> > You could define that file-union right after ijconsole.  If you want
> > to
> > golf even more, you could define ijconsole inside that file-union,
> > i.e.
> >   (define jsoftware-aux-files
> >     (file-union "jsoftware-aux-files"
> >       `(("profile.ijs" ,(search-aux-file ...)
> >         ("ijconsole" ,(program-file ...))))
> > 
> > I'm not quite sure if you want to use jsoftware-aux-files directly as
> > input or whether it's wiser to stuff it into another union like 
> > (file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
> > search-input-file will probably do the right thing regardless.
> > The new style should also still work with assoc-ref, it'd just be
> > weirder to look at.  Lastly, you could code up a (search-file-input)
> > just in case; I'm not sure if we have one already.
> 
> The file-union seems like a cludgy workaround to me. What we really
> want is an easy, direct way to get handles on the input files. Heck,
> program-file objects already have a name property; why can't we use
> that? Attached patches are a proof-of-concept.
That proof of concept does look nice, but for one we're trying to move
away from labels, and for the other, it's on a scale that I don't want
to decide as part of a package addition.  If you feel it has value
outside of the proposed usage for j, discussing it under a different
number or perhaps on guix-devel might be worth it.

> That said, if this is going to turn into a big rabbit hole, can we just
> munge the J package inputs into whatever you think is best?
As said in my previous mail, that'd be
> >   (define jsoftware-aux-files
> >     (file-union "jsoftware-aux-files"
> >       `(("profile.ijs" ,(search-aux-file ...)
> >         ("ijconsole" ,(program-file ...))))
In my personal opinion, you can then simply add jsoftware-aux-files as
input and (search-input-file "ijconsole") instead of the current assoc-
ref.  WDYT?

Don't worry, I don't plan to drag this out too long, but I also don't
planning on pushing this today.  There are definitely some stylistic
choices that I want to make under the considerations here (basically,
we can just merge major and minor into a single base that'd be "903.a",
for example), but it's almost bedtime and I want to relax a little
before going to sleep.

Cheers

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 3978 bytes --]

Again, thanks for the consistent quick turn-around, lately.

> > How about we just go ahead and treat MINOR as mandatory as well?
> In other words minor should always have a value >= "a" and 902 was just
> missing it?  Fair enough, in that case making version mandatory sounds
> like the way to go.

Exactly.

Thanks to your close reading of the patch and your opinionated suggestions, we
were able to catch this

> > The file-union seems like a cludgy workaround to me. What we really
> > want is an easy, direct way to get handles on the input files. Heck,
> > program-file objects already have a name property; why can't we use
> > that? Attached patches are a proof-of-concept.
> That proof of concept does look nice, but for one we're trying to move
> away from labels, and for the other, it's on a scale that I don't want
> to decide as part of a package addition.  If you feel it has value
> outside of the proposed usage for j, discussing it under a different
> number or perhaps on guix-devel might be worth it.

Of course. It would be kind of ridiculous to merge some random, only vaguely
related patch deep in the internals of the system, as part of a simple package
addendum.

We're not writing bills for the Senate here!

Whether or not the patch is valuable, I could learn a lot from any ensuing
discussion, so I might take up your idea to post separately.

> > That said, if this is going to turn into a big rabbit hole, can we just
> > munge the J package inputs into whatever you think is best?
> As said in my previous mail, that'd be
> > >   (define jsoftware-aux-files
> > >     (file-union "jsoftware-aux-files"
> > >       `(("profile.ijs" ,(search-aux-file ...)
> > >         ("ijconsole" ,(program-file ...))))
> In my personal opinion, you can then simply add jsoftware-aux-files as
> input and (search-input-file "ijconsole") instead of the current assoc-
> ref.  WDYT?

Sounds clear to me!

However, for some reason, right now if 'search-auxiliary-file' is inside a
file-union, I'm getting ENOENT on the file somewhere:

    Backtrace:
               2 (primitive-load "/gnu/store/fk7mr923n47r7wj7xqlfmh80jc5?")
    In ice-9/eval.scm:
        619:8  1 (_ #f)
    In unknown file:
               0 (stat "/home/x/devel/org.gnu.savannah/guix/gnu/package?" ?)

    ERROR: In procedure stat:
    In procedure stat: No such file or directory: "/home/x/devel/org.gnu.savannah/guix/gnu/packages/aux-files/jsoftware/profilex.ijs"
    builder for `/gnu/store/4zhrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv' failed with exit code 1
    build of /gnu/store/4zhrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv failed
    View build log at '/var/log/guix/drvs/4z/hrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv.bz2'.
    cannot build derivation `/gnu/store/ax3nwc5xybqcirxadm4ynz99jsq3l3j7-jsoftware-j-903.a.drv': 1 dependencies couldn't be built
    guix build: error: build of `/gnu/store/ax3nwc5xybqcirxadm4ynz99jsq3l3j7-jsoftware-j-903.a.drv' failed

Running `stat <path>` from the command line on the offending path succeeds as
expected, and moving the 'search-auxiliary-file' out of the file-union and
into the package inputs lets the build proceed.

I'm stumped. Attached is the package definition, for reference.


> Don't worry, I don't plan to drag this out too long, but I also don't
> planning on pushing this today.  There are definitely some stylistic
> choices that I want to make under the considerations here (basically,
> we can just merge major and minor into a single base that'd be "903.a",
> for example), but it's almost bedtime and I want to relax a little
> before going to sleep.

Sure. I'm mostly just way out of my depth here, getting sent on deep
rabbit-holes every time you respond give me some feedback to chew on. :P

Definitely down for some stylistic improvements. Let me know!

Hope you got some nice relaxation time in!



[-- Attachment #2: jsoftware.scm --]
[-- Type: text/plain, Size: 20876 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (gnu packages jsoftware)
  #:use-module (guix build utils)
  #:use-module (guix build-system gnu)
  #:use-module (guix build-system trivial)
  #:use-module (guix git-download)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix utils)
  #:use-module (gnu packages)
  #:use-module (gnu packages libedit)
  #:use-module (gnu packages llvm)
  #:use-module (gnu packages maths)
  #:use-module (guix gexp)
  #:use-module (ice-9 ftw)
  #:use-module (ice-9 match)
  #:use-module (ice-9 regex)
  #:use-module (ice-9 rdelim)
  #:use-module (srfi srfi-1)
  #:use-module (srfi srfi-26)
  #:use-module (srfi srfi-71))


;;; TODO: Make importer and packages for J addons:
;;; http://www.jsoftware.com/jal/

;;; TODO: Package up j80x series


(define (jname prefix release-type)
  "Return a package name for J, including RELEASE-TYPE only if not 'release."
  (match release-type
    ('release prefix)
    (_        (string-append prefix "-" (symbol->string release-type)))))

;; We want a version string where packages specifications like pkg@MAJOR work.
;; This requires that the first version part separator be dot.  Subsequent
;; separators are hyphen, mirror `git-version' etc.
(define* (jversion->string major minor #:optional revision commit)
  "Return a version string formatted like MAJOR.MINOR-REVISION-COMMIT.  Only
  MAJOR is required, and MINOR defaults to ``0'' if not supplied."
  (let* ((commit (and commit (string-take commit 7)))
         (sub-parts (filter (cut (compose not eq?) #f <>)
                            (list minor revision commit))))
    (string-append major "." (string-join sub-parts "-"))))

(define* (jrelease-string release-type #:optional version-minor)
  "Construct J release identifier string."
  (let ((release-type (symbol->string release-type)))
    (if version-minor
     (string-append release-type "-" version-minor)
     release-type)))

(define* (jinfo->git-tag version-major release-type #:optional version-minor)
  "Given version parameters, construct a git tag for upstream releases."
  (string-append "j" version-major (jrelease-string release-type version-minor)))

(define jsoftware-aux-files
  (file-union "jsoftware-aux-files"
     ;; profilex.ijs overrides ~install and ~addons directories to reside under
     ;; the user-writable ~user.  This allows local-install of addons via
     ;; pacman.
     ;; TODO: Guix-ify J addons as well.
   `(("profilex.ijs" ,(search-auxiliary-file "jsoftware/profilex.ijs"))
     ;; Gexp script that detects AVX/AVX2 support at runtime and executes
     ;; jconsole with the appropriate libj.so and profile.ijs."
     ("ijconsole"
      ,(with-imported-modules '((guix cpu)
                                (guix memoization)
                                (guix profiling)
                                (guix sets)
                                (srfi srfi-26))
         (program-file "ijconsole"
           #~(begin
               (use-modules ((guix cpu)     #:select (cpu-flags current-cpu))
                            ((guix sets)    #:select (set-contains?))
                            ((srfi srfi-26) #:select (cute)))

               ;; Assume that this script will be installed under bin/.
               (define %basedir (dirname (dirname (current-filename))))

               (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
                      (cpu-has-flag?
                        (cute set-contains? (cpu-flags (current-cpu)) <>))
                      (libj (format #f "~a/lib/j/libj~a.so" %basedir
                                    (cond ((cpu-has-flag? "avx2") "-avx2")
                                          ((cpu-has-flag? "avx") "-avx")
                                          (else ""))))
                      (jprofile (string-append %basedir "/etc/j/profile.ijs")))
                 (apply execl jconsole "ijconsole"
                        "-lib" libj "-jprofile" jprofile
                        (cdr (command-line)))))))))))

(define* (make-j #:key
                 version
                 revision
                 hash
                 tag
                 commit
                 (release-type 'release)
                 (patches '())
                 (extra-inputs '())
                 (extra-envars '())
                 (builder "guix.gnu.org"))
 (let* ((version-major version-minor (if (pair? version)
                                       (car+cdr version)
                                       (values version #f))))
 (package
   (name (jname "jsoftware-j" release-type))
   (version (jversion->string version-major version-minor revision commit))
   (source
    (origin
      (method git-fetch)
      (uri (git-reference
            (url "https://github.com/jsoftware/jsource")
            (commit (or commit tag
                        (jinfo->git-tag version-major
                                        release-type
                                        version-minor)))))
      (sha256 (base32 hash))
      (file-name (git-file-name name version))
      (patches patches)))
   (build-system gnu-build-system)
   (native-inputs (list clang-toolchain))
   (inputs (cons* libedit libomp jsoftware-aux-files extra-inputs))
   (arguments
    `(#:tests? #f
      #:modules (((ice-9 ftw) #:select (scandir))
                 ((ice-9 popen) #:select (open-pipe* close-pipe))
                 ((ice-9 regex) #:select (match:substring string-match))
                 ((ice-9 threads) #:select (parallel par-for-each))
                 ((srfi srfi-26) #:select (cut))
                 ((srfi srfi-1) #:select (fold))
                 ,@%gnu-build-system-modules)
      #:phases
      ;; Upstream's build system consists of ad-hoc scripts that build build up
      ;; (very complicated) environment variables to pass to make.  The basic
      ;; build process looks like this:
      ;;
      ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
      ;;   2) Set jplatform and j64x environment variables;
      ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
      ;;
      ;; However, upstream expects users to run J directly from the source
      ;; directory; they do not supply a make `install' target.  Thus it takes
      ;; some massaging to install files in FHS-style directories.
      (modify-phases %standard-phases
        ;; In particular, we have to set up
        ;;
        ;;   1) jsrc/jversion.h as in a typical build;
        ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
        ;;   3) make2/build_*.sh to respect standard build conventions;
        ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
        ;;   5) Hard coded references to addons directory.
        (replace 'configure
          (lambda* (#:key target inputs outputs #:allow-other-keys)
            (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
                   (clang (string-append clang-toolchain "/bin/clang"))
                   (libedit (assoc-ref inputs "libedit"))
                   (out (assoc-ref outputs "out")))
              ;; Set up build constants
              (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
              (substitute* "jsrc/jversion.h"
                (("^#define jversion.*$")
                 (format #f "#define jversion ~s\n" ,version-major))
                (("^#define jtype.*$")
                 (format #f "#define jtype ~s\n"
                         ,(jrelease-string release-type version-minor)))
                (("^#define jbuilder.*$")
                 (format #f "#define jbuilder ~s\n" ,builder)))
              ;; Munge the build scripts into reason:
              ;; 1. Short-circuit the fragile compiler detection;
              ;; 2. Make sure to include our CFLAGS and LFLAGS; and
              ;; 3. Propagate script errors to top level.
              (for-each
               (lambda (file)
                 (with-directory-excursion "make2"
                   (substitute* file
                     ;; The `compiler' variable doesn't point to the actual
                     ;; compiler.  It is just a switch to tell the build
                     ;; scripts whether to use gcc- or clang-specific flags.
                     (("^compiler=.*$") "compiler=clang\n")
                     (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
                     (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
                      (string-append def "$CFLAGS " rest))
                     (("^#!.*" shebang)
                      (string-append shebang "set -o errexit\n")))))
                 '("build_jconsole.sh" "build_libj.sh"))
              ;; The jconsole manually loads libedit with dlopen.  The path
              ;; must be absolute to correctly point to our input.
              (substitute* "jsrc/jconsole.c"
                (("libedit\\.so\\.[0-9]" so-file)
                 (format #f "~a/lib/~a" libedit so-file)))
              ;; The ~addons/dev directory supplies tentative J-script
              ;; definitions of new J engine functionality.  Since we point
              ;; ~addons under the ~user directory, we move it under ~system
              ;; instead, which sits as-is in the output.
              (with-directory-excursion "jsrc"
                (for-each
                  (lambda (file)
                    (substitute* file (("~addons/dev") "~system/dev")))
                  (scandir "."
                    (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
              ;; Implementation of 9!:14 records build time which breaks build
              ;; reproducibility.  Note that upstream code depends on the exact
              ;; format of these strings, so we need to mimic the standard.
              (substitute* "jsrc/j.c"
                (("__DATE__") "\"Jan 01 1970\"")
                (("__TIME__") "\"00:00:00\""))
              ;; Upstream recommends using clang, with GCC support being
              ;; second-class, often resulting in build failures.
              (setenv "CC" clang))))

        ;; The build output depends primarily on the values of the `jplatform'
        ;; and `j64x' environment variables.  If the target is ARM, then
        ;; `jplatform' is "raspberry", otherwise it is `linux'.  In addition to
        ;; 32- and 64- bit versions, `j64x' controlls whether AVX or AVX2
        ;; variants of libj are built.
        ;;
        ;; However, build targets are not fine-grained enough to distinguish
        ;; between CPU features.  Thus we build and install all variants of
        ;; libj, expecting jconsole to be called with a wrapper script that
        ;; detects AVX features and loads the appropriate libj at runtime.
        (replace 'build
          (lambda _
            (setenv "USE_OPENMP" "1")
            (setenv "USE_THREAD" "1")
            (for-each (lambda (var-val) (apply setenv var-val))
                      (quote ,extra-envars))
            ;; The build scripts assume that PWD is make2.
            (with-directory-excursion "make2"
              (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
                     (target-bit ,(if (target-64bit?) "64" "32"))
                     (run (lambda* (script #:key (avx ""))
                            (invoke "env"
                                    (string-append "jplatform=" platform)
                                    (string-append "j64x=j" target-bit avx)
                                    script))))
                (parallel
                  ;; Since jconsole doesn't depend on AVX features, we just
                  ;; build it once.
                  (run "./build_jconsole.sh")
                  (run "./build_libj.sh")
                  (if ,(target-64bit?)
                    (parallel
                      (run "./build_libj.sh" #:avx "avx")
                      (run "./build_libj.sh" #:avx "avx2"))))))))
        ;; The test suite is expected to be run as follows for each variant of
        ;; libj that we build:
        ;;
        ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
        ;;
        ;; This requires a working jconsole with accessible jlibrary files.  We
        ;; simply place these all under test/bin.
        (replace 'check
          (lambda* (#:key tests? #:allow-other-keys)
            (when tests?
              (let ((platform ,(if (target-arm?) "raspberry" "linux")))
                (mkdir-p "test/bin")
                (for-each
                  (lambda (dir)
                    (let ((source (string-append "jlibrary/" dir))
                          (dest (string-append "test/bin/" dir)))
                    (begin
                      (mkdir-p dest)
                      (copy-recursively source dest))))
                  '("system" "tools" "addons"))
                ;; The jlibrary/dev directory only sometimes exists, but needs
                ;; to be copied into the ~system directory when it does.
                (for-each
                  (lambda (dev-dir)
                    (if (file-exists? dev-dir)
                      (copy-recursively dev-dir "test/bin/system/dev")))
                  '("jlibrary/dev" "jlibrary/addons/dev"))
                (par-for-each
                  (lambda (dir)
                    (let* ((bin (string-append "bin/" platform))
                           (jbit ,(if (target-64bit?) "j64" "j32"))
                           (jconsole (string-append bin "/" jbit
                                                    "/jconsole"))
                           (source (string-append bin "/" dir))
                           (dest (string-append "test/bin/" dir)))
                      (begin
                        (mkdir-p dest)
                        (copy-recursively source dest)
                        (install-file "jlibrary/bin/profile.ijs" dest)
                        (install-file jconsole dest)
                        (let* ((jconsole (string-append dest "/jconsole"))
                               (tests "test/tsu.ijs")
                               (port (open-pipe* OPEN_WRITE jconsole tests)))
                          (display "RUN ddall\n" port)
                          (when (not (zero? (status:exit-val
                                              (close-pipe port))))
                            (error "Some J build tests failed."))))))
                  (scandir (string-append "bin/" platform)
                           (negate (cut member <> '("." "..")))))
                #t))))
        ;; Now that everything is built, installation is fairly
        ;; straightforward, following FHS conventions.  The only quirk is that
        ;; we install jconsole under /libexec to make room for the wrapper
        ;; replacement under /bin.
        (replace 'install
          (lambda* (#:key outputs inputs #:allow-other-keys)
            (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
                   (jbit ,(if (target-64bit?) "j64" "j32"))
                   (out (assoc-ref outputs "out"))
                   (bin (string-append out "/bin"))
                   (etc (string-append out "/etc/j"))
                   (lib (string-append out "/lib/j"))
                   (libexec (string-append out "/libexec/j"))
                   (share (string-append out "/share/j"))
                   (system (string-append share "/system"))
                   (dev (string-append system "/dev")))
              (mkdir-p bin)
              (copy-file (search-input-file inputs "ijconsole")
                         (string-append bin "/ijconsole-" ,version-major))
              (mkdir-p lib)
              (for-each
                (lambda (jarch)
                  (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
                         (javx-match (string-match "avx.*" jarch))
                         (javx (if (not javx-match) ""
                                 (match:substring javx-match)))
                         (sep (if javx-match "-" ""))
                         (source (string-append jbin "/libj.so"))
                         (dest (format #f "~a/libj~a~a.so" lib sep javx)))
                    (copy-file source dest)))
                (scandir (string-append "bin/" platform)
                         (negate (cut member <> '("." "..")))))
              (install-file (string-append "bin/" platform "/" jbit "/jconsole")
                            libexec)
              (copy-recursively "jlibrary/system" system)
              (for-each
                (lambda (source-dev)
                  (if (access? source-dev R_OK)
                    (copy-recursively source-dev dev)))
                '("jlibrary/dev" "jlibrary/addons/dev"))
              (install-file "jlibrary/bin/profile.ijs" etc)
              (install-file (search-input-file inputs "profilex.ijs")
                            etc)))))))
   (home-page "https://www.jsoftware.com/")
   (synopsis "Ascii-only, array programming language in the APL family")
   (description
    "J is a high-level, general-purpose programming language that is
particularly suited to the mathematical, statistical, and logical analysis of
data.  It is a powerful tool for developing algorithms and exploring problems
that are not already well understood.")
   (license license:gpl3+))))


(define-public jsoftware-j-901
  (make-j
    #:version '("901" . "f")
    #:hash "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
    #:patches (search-patches "jsoftware-j901-f-fixes.patch")))


(define j-build-configuration-with-sleef
  `(#:extra-inputs (,sleef)
    #:extra-envars (("USE_SLEEF_SRC" "0")
                    ("LDFLAGS" "-lsleef"))))

(define-public jsoftware-j-902
  (apply make-j
    (append j-build-configuration-with-sleef
      `(#:version ,'("902" . "b")
        #:hash "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"))))


(define-public jsoftware-j-903
  (apply make-j
    (append j-build-configuration-with-sleef
      `(#:version ,'("903" . "a")
        #:tag "903-release-a"
        #:hash "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"))))


(define-public (jsoftware-ijconsole-symlink jpkg)
  "Provide bin/ijconsole symlink that points to pkg's bin/ijconsole-<jversion>"
  (package
    (name "jsoftware-ijconsole")
    (version (package-version jpkg))
    (source #f)
    (build-system trivial-build-system)
    (propagated-inputs `(("jpkg" ,jpkg)))
    (arguments
      `(#:modules ((guix build utils)
                   (srfi srfi-26))
        #:builder
        (begin
          (use-modules ((guix build utils) #:select (mkdir-p))
                       ((ice-9 regex) #:select (string-match))
                       ((ice-9 ftw) #:select (scandir))
                       ((srfi srfi-26) #:select (cut)))
          (let* ((out (assoc-ref %outputs "out"))
                 (jpkg (assoc-ref %build-inputs "jpkg"))
                 (ijconsole (car (scandir (string-append jpkg "/bin")
                                       (cut string-match "ijconsole-.*" <>))))
                 (source (string-append jpkg "/bin/" ijconsole))
                 (dest (string-append out "/bin/ijconsole")))
            (mkdir-p (dirname dest))
            (symlink source dest)))))
  (home-page (package-home-page jpkg))
  (synopsis "Provide `ijconsole' symlink to default interpreter version")
  (description
  "The interpreter provided by the J package has a filename like
ijconsole-<version>, which provides support for having multiple, concurrent
versions installed.  This package provides a version-agnostic `ijconsole'
symlink to interpreter version indicated and build time.")
  (license license:gpl3+)))

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

Hi,

I tried to get J ready for packaging, but thanks to my CPU not
supporting AVX2 and some helpful discussion in IRC, I was able to avoid
an error here.

The (guix cpu) code is not meant to be used at runtime as the comment
to ijconsole implies it wants it to be used.  Rather, you should use it
at compile time to bake in the correct tuning parameters, and it should
ideally also support "cross-tuning" (though in that case you probably
need to disable the tests because your CPU won't make sense of the
code).  ijconsole not only fails that, but it also fails at an even
more basic task in Guix' launcher scripts, which is actually knowing
the thing you launch.  (We hardcode everything, period.)

Now I could just disable everything AVX-related in J and push something
that works on x86 and amd64, but since you do claim that AVX is
important to J, there are also other options one could take here.  One
is to implement tuning for this package the correct way, though since
it reinvents build systems, that's probably going to be a hard one. 
The other would be to define package variants (e.g. jsoftware-j-with-
avx) and use those -- there ought to be an old blog post on Guix HPC
detailing the rationale behind doing that.

WDYT?  Is AVX worth the trouble?

[-- Attachment #2: 0001-gnu-Add-j.patch --]
[-- Type: text/x-patch, Size: 26834 bytes --]

From 6b15deda2ecc09b73545e3feccfac7f3f695c9e8 Mon Sep 17 00:00:00 2001
From: "B. Wilson" <elaexuotee@wilsonb.com>
Date: Wed, 12 Jan 2022 18:44:36 +0900
Subject: [PATCH] gnu: Add j.

* gnu/packages/jsoftware.scm: New file.
* gnu/packages/patches/jsoftware-j901-f-fixes.patch: New file.
* gnu/local.mk [GNU_SYSTEM_MODULES]: Add jsoftware.scm.
[dist_patch_DATA]: Add jsoftware-j901-f-fixes.patch.
* gnu/packages/aux-files/jsoftware/profilex.ijs: New file.
*Makefile.am [AUX_FILES]: Add it here.

---
 Makefile.am                                   |   1 +
 gnu/local.mk                                  |   2 +
 gnu/packages/aux-files/jsoftware/profilex.ijs |  14 +
 gnu/packages/jsoftware.scm                    | 421 ++++++++++++++++++
 .../patches/jsoftware-j901-f-fixes.patch      |  80 ++++
 5 files changed, 518 insertions(+)
 create mode 100644 gnu/packages/aux-files/jsoftware/profilex.ijs
 create mode 100644 gnu/packages/jsoftware.scm
 create mode 100644 gnu/packages/patches/jsoftware-j901-f-fixes.patch

diff --git a/Makefile.am b/Makefile.am
index a10aeb817b..1efd8a9b26 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -379,6 +379,7 @@ AUX_FILES =						\
   gnu/packages/aux-files/chromium/master-preferences.json		\
   gnu/packages/aux-files/emacs/guix-emacs.el		\
   gnu/packages/aux-files/guix.vim			\
+  gnu/packages/aux-files/jsoftware/profilex.ijs			\
   gnu/packages/aux-files/linux-libre/5.15-arm.conf	\
   gnu/packages/aux-files/linux-libre/5.15-arm64.conf	\
   gnu/packages/aux-files/linux-libre/5.15-i686.conf	\
diff --git a/gnu/local.mk b/gnu/local.mk
index 7e044d4a2b..eb1309c9d7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -325,6 +325,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/jemalloc.scm			\
   %D%/packages/jrnl.scm				\
   %D%/packages/jose.scm				\
+  %D%/packages/jsoftware.scm				\
   %D%/packages/julia.scm			\
   %D%/packages/julia-jll.scm			\
   %D%/packages/julia-xyz.scm			\
@@ -1295,6 +1296,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/irrlicht-use-system-libs.patch		\
   %D%/packages/patches/isc-dhcp-gcc-compat.patch		\
   %D%/packages/patches/isl-0.11.1-aarch64-support.patch	\
+  %D%/packages/patches/jsoftware-j901-f-fixes.patch		\
   %D%/packages/patches/json-c-0.13-CVE-2020-12762.patch	\
   %D%/packages/patches/json-c-0.12-CVE-2020-12762.patch	\
   %D%/packages/patches/jsoncpp-pkg-config-version.patch		\
diff --git a/gnu/packages/aux-files/jsoftware/profilex.ijs b/gnu/packages/aux-files/jsoftware/profilex.ijs
new file mode 100644
index 0000000000..30e0d229e2
--- /dev/null
+++ b/gnu/packages/aux-files/jsoftware/profilex.ijs
@@ -0,0 +1,14 @@
+'jtype jversion'=. (3&{,{.) <;._2 ,&'/' 9!:14''
+basedir=. ({.~ _2 { I.@:=&'/') BINPATH
+
+share=.  basedir,'/share/j'
+system=. share,'/system'
+tools=.  share,'/tools'
+
+user=.    home,'/.config/j/',jversion
+addons=.  user,'/addons'
+break=.   user,'/break'
+config=.  user,'/config'
+install=. user,'/install'
+snap=.    user,'/snap'
+temp=.    user,'/temp'
diff --git a/gnu/packages/jsoftware.scm b/gnu/packages/jsoftware.scm
new file mode 100644
index 0000000000..c7d5c4d7b5
--- /dev/null
+++ b/gnu/packages/jsoftware.scm
@@ -0,0 +1,421 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 B. Wilson <elaexuotee@wilsonb.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages jsoftware)
+  #:use-module (guix build utils)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system trivial)
+  #:use-module (guix git-download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages libedit)
+  #:use-module (gnu packages llvm)
+  #:use-module (gnu packages maths)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 rdelim)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-71))
+
+
+;;; TODO: Make importer and packages for J addons:
+;;; http://www.jsoftware.com/jal/
+
+;;; TODO: Package up j80x series
+
+
+(define (jname prefix release-type)
+  "Return a package name for J, including RELEASE-TYPE only if not 'release."
+  (match release-type
+    ('release prefix)
+    (_        (string-append prefix "-" (symbol->string release-type)))))
+
+(define (version-major+minor* version)
+  "Like version-major+minor, but returning two string values."
+  (let ((parts (string-split version #\.)))
+    (values (first parts) (second parts))))
+
+(define* (jrelease-string release-type #:optional version-minor)
+  "Construct J release identifier string."
+  (let ((release-type (symbol->string release-type)))
+    (if version-minor
+        (string-append release-type "-" version-minor)
+        release-type)))
+
+(define* (jinfo->git-tag version release-type)
+  "Given version parameters, construct a git tag for upstream releases."
+  (let ((major minor (version-major+minor* version)))
+    (string-append "j" major "-" (jrelease-string release-type minor))))
+
+;; G-exp script that detects AVX/AVX2 support at runtime and executes jconsole
+;; with the appropriate libj.so and profile.ijs."
+;; NOTE: This should be baked in at compile time into `jsoftware-j'.
+(define ijconsole
+  (with-imported-modules '((guix cpu)
+                           (guix memoization)
+                           (guix profiling)
+                           (guix sets)
+                           (srfi srfi-26))
+    (program-file "ijconsole"
+      #~(begin
+          (use-modules ((guix cpu)     #:select (cpu-flags current-cpu))
+                       ((guix sets)    #:select (set-contains?))
+                       ((srfi srfi-26) #:select (cute)))
+
+          ;; Assume that this script will be installed under bin/.
+          (define %basedir (dirname (dirname (current-filename))))
+
+          (let* ((jconsole (string-append %basedir "/libexec/j/jconsole"))
+                 (cpu-has-flag?
+                   (cute set-contains? (cpu-flags (current-cpu)) <>))
+                 (libj (format #f "~a/lib/j/libj~a.so" %basedir
+                               ""#;
+                               (cond ((cpu-has-flag? "avx2") "-avx2")
+                                     ((cpu-has-flag? "avx") "-avx")
+                                     (else ""))))
+                 (jprofile (string-append %basedir "/etc/j/profile.ijs")))
+            (apply execl jconsole "ijconsole" "-lib" libj "-jprofile" jprofile
+                   (cdr (command-line))))))))
+
+(define* (make-j base-version hash
+                 #:key
+                 revision
+                 commit
+                 tag
+                 (release-type 'release)
+                 (patches '())
+                 (modules '())
+                 (snippet #f)
+                 (extra-inputs '())
+                 (extra-envars '())
+                 (builder "guix.gnu.org"))
+  (let ((major minor (version-major+minor* base-version)))
+    (package
+      (name (jname "jsoftware-j" release-type))
+      (version
+       (if commit (git-version base-version revision commit) base-version))
+      (source
+       (origin
+         (method git-fetch)
+         (uri (git-reference
+               (url "https://github.com/jsoftware/jsource")
+               (commit (or commit tag
+                           (jinfo->git-tag base-version release-type)))))
+         (sha256 (base32 hash))
+         (file-name (git-file-name name version))
+         (patches patches)
+         (modules modules)
+         (snippet snippet)))
+      (build-system gnu-build-system)
+      (native-inputs (list clang-toolchain))
+      (inputs
+       ;; ijconsole and profile.ijs still require labels
+       `(("ijconsole" ,ijconsole)
+         ("profilex.ijs" ,(search-auxiliary-file "jsoftware/profilex.ijs"))
+         ("libedit" ,libedit)
+         ("libomp" ,libomp)
+         ,@extra-inputs))
+      (arguments
+       `(#:modules (((ice-9 ftw) #:select (scandir))
+                    ((ice-9 popen) #:select (open-pipe* close-pipe))
+                    ((ice-9 regex) #:select (match:substring string-match))
+                    ((ice-9 threads) #:select (parallel par-for-each))
+                    ((srfi srfi-26) #:select (cut))
+                    ((srfi srfi-1) #:select (fold))
+                    ,@%gnu-build-system-modules)
+         #:phases
+         ;; Upstream's build system consists of ad-hoc scripts that build up
+         ;; (very complicated) environment variables to pass to make.
+         ;; The basic build process looks like this:
+         ;;
+         ;;   1) Copy jsrc/jversion-x.h to jsrc/jversion.h and edit values;
+         ;;   2) Set jplatform and j64x environment variables;
+         ;;   3) Run make2/build_jconsole.sh and make2/build_libj.sh;
+         ;;
+         ;; However, upstream expects users to run J directly from the source
+         ;; directory; they do not supply a make `install' target.  Thus it
+         ;; takes some massaging to install files in FHS-style directories.
+         (modify-phases %standard-phases
+           ;; In particular, we have to set up
+           ;;
+           ;;   1) jsrc/jversion.h as in a typical build;
+           ;;   2) jlibrary/bin/profilex.ijs to point to writable directories;
+           ;;   3) make2/build_*.sh to respect standard build conventions;
+           ;;   4) jsrc/jconsole.c to fix libedit dlopen; and
+           ;;   5) Hard coded references to addons directory.
+           (replace 'configure
+             (lambda* (#:key target inputs outputs #:allow-other-keys)
+               (let* ((clang-toolchain (assoc-ref inputs "clang-toolchain"))
+                      (clang (string-append clang-toolchain "/bin/clang"))
+                      (libedit (assoc-ref inputs "libedit"))
+                      (out (assoc-ref outputs "out")))
+                 ;; Set up build constants
+                 (copy-file "jsrc/jversion-x.h" "jsrc/jversion.h")
+                 (substitute* "jsrc/jversion.h"
+                   (("^#define jversion.*$")
+                    (format #f "#define jversion ~s\n" ,major))
+                   (("^#define jtype.*$")
+                    (format #f "#define jtype ~s\n"
+                            ,(jrelease-string release-type minor)))
+                   (("^#define jbuilder.*$")
+                    (format #f "#define jbuilder ~s\n" ,builder)))
+                 ;; Munge the build scripts into reason:
+                 ;; 1. Short-circuit the fragile compiler detection;
+                 ;; 2. Make sure to include our CFLAGS and LFLAGS; and
+                 ;; 3. Propagate script errors to top level.
+                 (for-each
+                  (lambda (file)
+                    (with-directory-excursion "make2"
+                      (substitute* file
+                        ;; The `compiler' variable doesn't point to the actual
+                        ;; compiler.  It is just a switch to tell the build
+                        ;; scripts whether to use gcc- or clang-specific flags.
+                        (("^compiler=.*$") "compiler=clang\n")
+                        (("^LDFLAGS=\"" def) (string-append def "$LDFLAGS "))
+                        (("^(common=\")(\\$USETHREAD.*)$" _ def rest)
+                         (string-append def "$CFLAGS " rest))
+                        (("^#!.*" shebang)
+                         (string-append shebang "set -o errexit\n")))))
+                  '("build_jconsole.sh" "build_libj.sh"))
+                 ;; The jconsole manually loads libedit with dlopen.  The path
+                 ;; must be absolute to correctly point to our input.
+                 (substitute* "jsrc/jconsole.c"
+                   (("libedit\\.so\\.[0-9]" so-file)
+                    (format #f "~a/lib/~a" libedit so-file)))
+                 ;; The ~addons/dev directory supplies tentative J-script
+                 ;; definitions of new J engine functionality.  Since we point
+                 ;; ~addons under the ~user directory, we move it under ~system
+                 ;; instead, which sits as-is in the output.
+                 (with-directory-excursion "jsrc"
+                   (for-each
+                    (lambda (file)
+                      (substitute* file (("~addons/dev") "~system/dev")))
+                    (scandir "."
+                             (lambda (f) (eq? (stat:type (stat f)) 'regular)))))
+                 ;; Implementation of 9!:14 records build time which breaks
+                 ;; reproducibility.  Note that upstream code depends on the
+                 ;; exact format of these strings, so we need to mimic the
+                 ;; standard.
+                 (substitute* "jsrc/j.c"
+                   (("__DATE__") "\"Jan 01 1970\"")
+                   (("__TIME__") "\"00:00:00\""))
+                 ;; Upstream recommends using clang, with GCC support being
+                 ;; second-class, often resulting in build failures.
+                 (setenv "CC" clang))))
+
+           ;; The build output depends primarily on the values of the
+           ;; `jplatform' and `j64x' environment variables.  If the target is
+           ;; ARM, then `jplatform' is "raspberry", otherwise it is `linux'.
+           ;; In addition to 32- and 64- bit versions, `j64x' controlls
+           ;; whether AVX or AVX2 variants of libj are built.
+           ;;
+           ;; However, build targets are not fine-grained enough to distinguish
+           ;; between CPU features.  Thus we build and install all variants of
+           ;; libj, expecting jconsole to be called with a wrapper script that
+           ;; detects AVX features and loads the appropriate libj at runtime.
+           (replace 'build
+             (lambda _
+               (setenv "USE_OPENMP" "1")
+               (setenv "USE_THREAD" "1")
+               (for-each (lambda (var-val) (apply setenv var-val))
+                         (quote ,extra-envars))
+               ;; The build scripts assume that PWD is make2.
+               (with-directory-excursion "make2"
+                 (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                        (target-bit ,(if (target-64bit?) "64" "32"))
+                        (run (lambda* (script #:key (avx ""))
+                               (invoke "env"
+                                       (string-append "jplatform=" platform)
+                                       (string-append "j64x=j" target-bit avx)
+                                       script))))
+                   (parallel
+                    ;; Since jconsole doesn't depend on AVX features, we just
+                    ;; build it once.
+                    (run "./build_jconsole.sh")
+                    (run "./build_libj.sh")
+                    (when #false ;; ,(target-64bit?)
+                      (run "./build_libj.sh" #:avx "avx")
+                      (run "./build_libj.sh" #:avx "avx2")))))))
+           ;; The test suite is expected to be run as follows for each variant
+           ;; of libj that we build:
+           ;;
+           ;;     $ echo 'RUN ddall' | jconsole test/tsu.ijs
+           ;;
+           ;; This requires a working jconsole with accessible jlibrary files.
+           ;; We simply place these all under test/bin.
+           (replace 'check
+             (lambda* (#:key tests? #:allow-other-keys)
+               (when tests?
+                 (let ((platform ,(if (target-arm?) "raspberry" "linux")))
+                   (mkdir-p "test/bin")
+                   (for-each
+                    (lambda (dir)
+                      (let ((source (string-append "jlibrary/" dir))
+                            (dest (string-append "test/bin/" dir)))
+                        (begin
+                          (mkdir-p dest)
+                          (copy-recursively source dest))))
+                    '("system" "tools" "addons"))
+                   ;; The jlibrary/dev directory only exists sometimes, but
+                   ;; when it does, it needs to be in ~system.
+                   (for-each
+                    (lambda (dev-dir)
+                      (if (file-exists? dev-dir)
+                          (copy-recursively dev-dir "test/bin/system/dev")))
+                    '("jlibrary/dev" "jlibrary/addons/dev"))
+                   (par-for-each
+                    (lambda (dir)
+                      (let* ((bin (string-append "bin/" platform))
+                             (jbit ,(if (target-64bit?) "j64" "j32"))
+                             (jconsole (string-append bin "/" jbit
+                                                      "/jconsole"))
+                             (source (string-append bin "/" dir))
+                             (dest (string-append "test/bin/" dir)))
+                        (begin
+                          (mkdir-p dest)
+                          (copy-recursively source dest)
+                          (install-file "jlibrary/bin/profile.ijs" dest)
+                          (install-file jconsole dest)
+                          (let* ((jconsole (string-append dest "/jconsole"))
+                                 (tests "test/tsu.ijs")
+                                 (port (open-pipe* OPEN_WRITE jconsole tests)))
+                            (display "RUN ddall\n" port)
+                            (unless (zero? (status:exit-val (close-pipe port)))
+                              (error "Some J build tests failed."))))))
+                    (scandir (string-append "bin/" platform)
+                             (negate (cut member <> '("." "..")))))
+                   #t))))
+           ;; Now that everything is built, installation is fairly
+           ;; straightforward, following FHS conventions.  The only quirk is
+           ;; that we install jconsole under /libexec to make room for the
+           ;; wrapper replacement under /bin.
+           (replace 'install
+             (lambda* (#:key outputs inputs #:allow-other-keys)
+               (let* ((platform ,(if (target-arm?) "raspberry" "linux"))
+                      (jbit ,(if (target-64bit?) "j64" "j32"))
+                      (out (assoc-ref outputs "out"))
+                      (bin (string-append out "/bin"))
+                      (etc (string-append out "/etc/j"))
+                      (lib (string-append out "/lib/j"))
+                      (libexec (string-append out "/libexec/j"))
+                      (share (string-append out "/share/j"))
+                      (system (string-append share "/system"))
+                      (dev (string-append system "/dev")))
+                 (mkdir-p bin)
+                 (copy-file (assoc-ref inputs "ijconsole")
+                            (string-append bin "/ijconsole-" ,major))
+                 (mkdir-p lib)
+                 (for-each
+                  (lambda (jarch)
+                    (let* ((jbin (string-join `("bin" ,platform ,jarch) "/"))
+                           (javx-match (string-match "avx.*" jarch))
+                           (javx (if (not javx-match) ""
+                                     (match:substring javx-match)))
+                           (sep (if javx-match "-" ""))
+                           (source (string-append jbin "/libj.so"))
+                           (dest (format #f "~a/libj~a~a.so" lib sep javx)))
+                      (copy-file source dest)))
+                  (scandir (string-append "bin/" platform)
+                           (negate (cut member <> '("." "..")))))
+                 (install-file
+                  (string-append "bin/" platform "/" jbit "/jconsole")
+                  libexec)
+                 (copy-recursively "jlibrary/system" system)
+                 (for-each
+                  (lambda (source-dev)
+                    (if (access? source-dev R_OK)
+                        (copy-recursively source-dev dev)))
+                  '("jlibrary/dev" "jlibrary/addons/dev"))
+                 (install-file "jlibrary/bin/profile.ijs" etc)
+                 (copy-file (assoc-ref inputs "profilex.ijs")
+                            (string-append etc "/profilex.ijs"))))))))
+      (home-page "https://www.jsoftware.com/")
+      (synopsis "Ascii-only, array programming language in the APL family")
+      (description
+       "J is a high-level, general-purpose programming language that is
+particularly suited to the mathematical, statistical, and logical analysis of
+data.  It is a powerful tool for developing algorithms and exploring problems
+that are not already well understood.")
+      (license license:gpl3+))))
+
+
+(define-public jsoftware-j-901
+  (make-j "901.f"
+          "1776021m0j1aanzwg60by83n53pw7i6afd5wplfzczwk8bywax4p"
+          #:patches (search-patches "jsoftware-j901-f-fixes.patch")))
+
+
+(define j-build-configuration-with-sleef
+  ;; XXX: label required because of ijconsole and profile.ijs in make-j.
+  ;;      if labels are dropped in make-j, drop them here too.
+  ;; XXX: sleef is still being bundled in j and unbundling it causes build
+  ;;      errors... investigate them and unbundle
+  `(#:extra-inputs (("sleef" ,sleef))
+    #:extra-envars (("USE_SLEEF_SRC" "0")
+                    ("LDFLAGS" "-lsleef"))))
+
+(define-public jsoftware-j-902
+  (apply make-j "902.b"
+         "0j67vgikqflwjqacsdicasvyv1k54s2c8vjgwmf0ix7l41p4xqz0"
+         j-build-configuration-with-sleef))
+
+(define-public jsoftware-j-903
+  (apply make-j "903.a"
+         "1fcfl7q7c2vj4fmnqqc8c6hwgsjm20ff93v8xxfniasss1b2fmc4"
+         #:tag "903-release-a"
+         j-build-configuration-with-sleef))
+
+(define-public (jsoftware-ijconsole-symlink jpkg)
+  "Provide bin/ijconsole symlink that points to pkg's
+bin/ijconsole-<jversion>."
+  (package
+    (name "jsoftware-ijconsole")
+    (version (package-version jpkg))
+    (source #f)
+    (build-system trivial-build-system)
+    (propagated-inputs `(("jpkg" ,jpkg)))
+    (arguments
+      `(#:modules ((guix build utils)
+                   (srfi srfi-26))
+        #:builder
+        (begin
+          (use-modules ((guix build utils) #:select (mkdir-p))
+                       ((ice-9 regex) #:select (string-match))
+                       ((ice-9 ftw) #:select (scandir))
+                       ((srfi srfi-26) #:select (cut)))
+          (let* ((out (assoc-ref %outputs "out"))
+                 (jpkg (assoc-ref %build-inputs "jpkg"))
+                 (ijconsole (car (scandir (string-append jpkg "/bin")
+                                       (cut string-match "ijconsole-.*" <>))))
+                 (source (string-append jpkg "/bin/" ijconsole))
+                 (dest (string-append out "/bin/ijconsole")))
+            (mkdir-p (dirname dest))
+            (symlink source dest)))))
+  (home-page (package-home-page jpkg))
+  (synopsis "Provide `ijconsole' symlink to default interpreter version")
+  (description
+  "The interpreter provided by the J package has a filename like
+ijconsole-<version>, which provides support for having multiple, concurrent
+versions installed.  This package provides a version-agnostic `ijconsole'
+symlink to interpreter version indicated and build time.")
+  (license license:gpl3+)))
diff --git a/gnu/packages/patches/jsoftware-j901-f-fixes.patch b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
new file mode 100644
index 0000000000..0ac7e94de4
--- /dev/null
+++ b/gnu/packages/patches/jsoftware-j901-f-fixes.patch
@@ -0,0 +1,80 @@
+This patch fixes two separate issues with ustream sources:
+
+* Normalize import paths in jsrc/cip.c
+
+Upstream claims to have some build requirements that force them to use strange
+import paths. However, these paths do not exist inside our build chroot.
+
+* Fix unititialized variable warning
+
+Clang 9 issues some warnings which cause the build to fail since upstream
+compiles with -Werror.
+
+
+diff --git a/jsrc/cip.c b/jsrc/cip.c
+index 61da4088..fb3c03b6 100644
+--- a/jsrc/cip.c
++++ b/jsrc/cip.c
+@@ -3,9 +3,9 @@
+ /*                                                                         */
+ /* Conjunctions: Inner Product                                             */
+ 
+-#include "../../jsource/jsrc/j.h"
+-#include "../../jsource/jsrc/vasm.h"
+-#include "../../jsource/jsrc/gemm.h"
++#include "j.h"
++#include "vasm.h"
++#include "gemm.h"
+ 
+ #define MAXAROWS 384  // max rows of a that we can process to stay in L2 cache   a strip is m*CACHEHEIGHT, z strip is m*CACHEWIDTH   this is wired to 128*3 - check if you chage
+ 
+@@ -1057,15 +1057,15 @@ static A jtipbx(J jt,A a,A w,C c,C d){A g=0,x0,x1,z;B*av,*av0,b,*v0,*v1,*zv;C c0
+  switch(c){
+   case CPLUSDOT:
+ #define F |=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CSTARDOT:
+ #define F &=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+   case CNE:
+ #define F ^=
+-#include "../../jsource/jsrc/cip_t.h"
++#include "cip_t.h"
+    break;
+  }
+  R z;
+diff --git a/jsrc/gemm.c b/jsrc/gemm.c
+index 51fe306e..b105dfc1 100644
+--- a/jsrc/gemm.c
++++ b/jsrc/gemm.c
+@@ -318,7 +318,7 @@ dgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -501,7 +501,7 @@ igemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
+@@ -831,7 +831,7 @@ zgemm_nn         (I              m,
+                    _B);
+ 
+ // loop 3
+-            I i;
++            I i=0;
+ #pragma omp parallel for default(none),private(i),shared(j,l,A,C,mb,nc,kc,alpha,_beta,_mc,_B,rs_a,cs_a,rs_c,cs_c)
+             for (i=0; i<mb; ++i) {
+                 I mc;
-- 
2.34.0

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

> I tried to get J ready for packaging, but thanks to my CPU not
> supporting AVX2 and some helpful discussion in IRC, I was able to avoid
> an error here.

The black hole has extended it's grip...

> The (guix cpu) code is not meant to be used at runtime as the comment
> to ijconsole implies it wants it to be used.  Rather, you should use it
> at compile time to bake in the correct tuning parameters, and it should
> ideally also support "cross-tuning" (though in that case you probably
> need to disable the tests because your CPU won't make sense of the
> code).

Hrm... Mind if I ask about the rationale? We could go back to scraping
/proc/cpuinfo, but I suspect that's not addressing the issue you have in mind.

> ijconsole not only fails that, but it also fails at an even more basic task
> in Guix' launcher scripts, which is actually knowing the thing you launch.
> (We hardcode everything, period.)

Okay. I see the logic in that.

The original patch indeed hard-coded paths in the ijconsole script, but that
required having ijconsole be a wrapper package. I thought I was being clever by
eliminating a "needless" wrapper, but it sounds like we can just bring it
back.

> Now I could just disable everything AVX-related in J and push something
> that works on x86 and amd64, but since you do claim that AVX is
> important to J, there are also other options one could take here.  One
> is to implement tuning for this package the correct way, though since
> it reinvents build systems, that's probably going to be a hard one. 
> The other would be to define package variants (e.g. jsoftware-j-with-
> avx) and use those -- there ought to be an old blog post on Guix HPC
> detailing the rationale behind doing that.

Not including AVX would be a hard hit.

Morally, the ijconsole script just wraps the libj*.so libs into a "fat binary".
From what I understand, fat binaries are (were?) an accepted solution in the
HPC world. Heck, the libj*.so total up to a whopping 15Mb, which feels more
like slim that fat, anyway.

Questions floating around in my head:

- What is wrong with detecting CPU features in a wrapper vs in the binary?
- If a wrapper is theoretically okay, how can we fix the current one?
- Is such a "proper fix" massively more subtle and bug-prone than the naive
  solution? Why?

> WDYT?  Is AVX worth the trouble?

Personally, I really do not like the jsoftware-j-with-* variants workaround. If
you say that tuning support is the only other acceptable solution, then I'll
bite the bullet and see what is possible.


P.S. Just throwing this out there, but if you'd like, I'd be willing to do a
mob session together. It seems like we are in compatible timezones.

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Hi,

Am Dienstag, dem 18.01.2022 um 13:01 +0900 schrieb
elaexuotee@wilsonb.com:
> 
> 
> > The (guix cpu) code is not meant to be used at runtime as the
> > comment to ijconsole implies it wants it to be used.  Rather, you
> > should use it at compile time to bake in the correct tuning
> > parameters, and it should ideally also support "cross-tuning"
> > (though in that case you probably need to disable the tests because
> > your CPU won't make sense of the code).
> 
> Hrm... Mind if I ask about the rationale? We could go back to
> scraping /proc/cpuinfo, but I suspect that's not addressing the issue
> you have in mind.
> 
> > ijconsole not only fails that, but it also fails at an even more
> > basic task in Guix' launcher scripts, which is actually knowing the
> > thing you launch. (We hardcode everything, period.)
> 
> Okay. I see the logic in that.
> 
> The original patch indeed hard-coded paths in the ijconsole script,
> but that required having ijconsole be a wrapper package. I thought I
> was being clever by eliminating a "needless" wrapper, but it sounds
> like we can just bring it back.
As far as I can see, you still need a wrapper to provide a symlink
stripped of the version.  I think you can combine the CPU detection
code with that (see below), so you can have a wrapper procedure, which
strips the version LOGICAL OR detects the CPU.

> > Now I could just disable everything AVX-related in J and push
> > something that works on x86 and amd64, but since you do claim that
> > AVX is important to J, there are also other options one could take
> > here.  One is to implement tuning for this package the correct way,
> > though since it reinvents build systems, that's probably going to
> > be a hard one. The other would be to define package variants (e.g.
> > jsoftware-j-with-avx) and use those -- there ought to be an old
> > blog post on Guix HPC detailing the rationale behind doing that.
> 
> Not including AVX would be a hard hit.
> 
> Morally, the ijconsole script just wraps the libj*.so libs into a
> "fat binary".  From what I understand, fat binaries are (were?) an
> accepted solution in the HPC world. Heck, the libj*.so total up to a
> whopping 15Mb, which feels more like slim that fat, anyway.
> 
> Questions floating around in my head:
> 
> - What is wrong with detecting CPU features in a wrapper vs in the
> binary?
> - If a wrapper is theoretically okay, how can we fix the current one?
> - Is such a "proper fix" massively more subtle and bug-prone than the
> naive solution? Why?
FWIW I don't quite think that fat binaries are the issue here, but that
building AVX blows up the check phase in the way we currently do.  It's
a similar issue w.r.t. check? for cross-compiling.  In my opinion only
the base feature set can be checked unless the CPU supports the same
features as the target.

I think if we want to do fat binaries, the solution would be to build
all three variants from make-j and tests only the base variants, and
then have a non-substitutable wrapper package, that takes that as an
input and simply provides a wrapper tuned to the current CPU features.
If you want, you can then rerun the correct tests in the wrapper
package.  The package returned by make-j could itself also provide
three binaries just in case someone doesn't want to generate the
wrapper package, but knows they can run ijconsole-avx2 just fine. 
WDYT?

> > WDYT?  Is AVX worth the trouble?
> 
> Personally, I really do not like the jsoftware-j-with-* variants
> workaround.  If you say that tuning support is the only other
> acceptable solution, then I'll bite the bullet and see what is
> possible.
> 
> P.S. Just throwing this out there, but if you'd like, I'd be willing
> to do a mob session together. It seems like we are in compatible
> timezones.
I'm in UTC+1 and have a day job from 8am to 4pm (plus delta and bus
routes), which normally make me unable to reply from roughly 6:30am to
8pm.  I'm also a little shy when it comes to letting strangers hear my
voice and experiencing healthy vaccine side effects atm.

Cheers

[bug#48463] gnu: Add j.

[elaexuotee--- via Guix-patches via]

> FWIW I don't quite think that fat binaries are the issue here, but that
> building AVX blows up the check phase in the way we currently do.  It's
> a similar issue w.r.t. check? for cross-compiling.  In my opinion only
> the base feature set can be checked unless the CPU supports the same
> features as the target.

Oh, good. Apparently, I mis-interpreted your original concerns.

> I think if we want to do fat binaries, the solution would be to build
> all three variants from make-j and tests only the base variants, and
> then have a non-substitutable wrapper package, that takes that as an
> input and simply provides a wrapper tuned to the current CPU features.
> If you want, you can then rerun the correct tests in the wrapper
> package.  The package returned by make-j could itself also provide
> three binaries just in case someone doesn't want to generate the
> wrapper package, but knows they can run ijconsole-avx2 just fine. 
> WDYT?

Slick. Let me check my understanding against yours with some specifics:

- Name make-j's package 'jsoftware-j-lib' which
  1. Builds all upstream stuff, and
  2. Only runs libj.so (non-avx) tests;

- Create non-substitutable 'jsoftware-j' wrapper package which
  1. Detects SSE extensions at build time and specializes the 'ijconsole'
     script accordingly, and
  2. Runs tests if and only if an avx or avx2 variant.

Does those points jibe with your thoughts on the fat binaries approach?


Glancing around at the CPU tuning stuff, it looks like tunable packages end up
getting a 'cpu-tuning' property which holds the microartecture name passed to
-march. AVX first landed in Sandy Bridge, and AVX2 in Haswell, so assuming
cpu-tuning is accessible at build time, it should be easy enough to condition
the build phase on those.

Regarding the check phase, here's a relevant comment in
guix/transformations.scm(tuned-package):552:

    ;; The machine building this package may or may not be able to run code
    ;; for MICRO-ARCHITECTURE.  Because of that, skip tests; they are run for
    ;; the "baseline" variant anyway.

Which I read to mean that the check phase should explicitly use libj.so,
ignoring libj-avx.so and libj-avx2.so. Running specialized tests in a
non-substitutable wrapper seems potentially better in this particular case.


Thoughts?


> I'm in UTC+1 and have a day job from 8am to 4pm (plus delta and bus
> routes), which normally make me unable to reply from roughly 6:30am to
> 8pm.  I'm also a little shy when it comes to letting strangers hear my
> voice and experiencing healthy vaccine side effects atm.

UTC+9 here. Okay. That's unfortunate, but I'll keep the offer open if you
change your mind.

Hope you get to feeling better soon!

Cheers,
B. Wilson

[bug#48463] gnu: Add j.

[Liliana Marie Prikler]

Am Dienstag, dem 18.01.2022 um 19:40 +0900 schrieb
elaexuotee@wilsonb.com:
> > FWIW I don't quite think that fat binaries are the issue here, but
> > that building AVX blows up the check phase in the way we currently
> > do. It's a similar issue w.r.t. check? for cross-compiling.  In my
> > opinion only the base feature set can be checked unless the CPU
> > supports the same features as the target.
> 
> Oh, good. Apparently, I mis-interpreted your original concerns.
> 
> > I think if we want to do fat binaries, the solution would be to
> > build all three variants from make-j and tests only the base
> > variants, and then have a non-substitutable wrapper package, that
> > takes that as an input and simply provides a wrapper tuned to the
> > current CPU features.
> > If you want, you can then rerun the correct tests in the wrapper
> > package.  The package returned by make-j could itself also provide
> > three binaries just in case someone doesn't want to generate the
> > wrapper package, but knows they can run ijconsole-avx2 just fine. 
> > WDYT?
> 
> Slick. Let me check my understanding against yours with some
> specifics:
> 
> - Name make-j's package 'jsoftware-j-lib' which
>   1. Builds all upstream stuff, and
>   2. Only runs libj.so (non-avx) tests;
> 
> - Create non-substitutable 'jsoftware-j' wrapper package which
>   1. Detects SSE extensions at build time and specializes the
> 'ijconsole'
>      script accordingly, and
>   2. Runs tests if and only if an avx or avx2 variant.
> 
> Does those points jibe with your thoughts on the fat binaries
> approach?
That's roughly what I was suggesting, yes.

> Glancing around at the CPU tuning stuff, it looks like tunable
> packages end up getting a 'cpu-tuning' property which holds the
> microartecture name passed to -march. AVX first landed in Sandy
> Bridge, and AVX2 in Haswell, so assuming cpu-tuning is accessible at
> build time, it should be easy enough to condition the build phase on
> those.
> 
> Regarding the check phase, here's a relevant comment in
> guix/transformations.scm(tuned-package):552:
> 
>     ;; The machine building this package may or may not be able to
> run code
>     ;; for MICRO-ARCHITECTURE.  Because of that, skip tests; they are
> run for
>     ;; the "baseline" variant anyway.
> 
> Which I read to mean that the check phase should explicitly use
> libj.so, ignoring libj-avx.so and libj-avx2.so. Running specialized
> tests in a non-substitutable wrapper seems potentially better in this
> particular case.
> 
> Thoughts?
I can't say that I agree with your reasoning here.  Essentially, what
you (or rather the J authors) are doing here is trying to outsmart the
compiler, while Guix assumes, that -march=WHATEVER ought to both
suffice in terms of performance and still be correct.  I'm leaning more
towards the assumption that Guix makes here.


If you want to enable tests on your local machine, you can still do so
by inheriting the transformed package.  Perhaps we should add a "--
with-tests" transformation as a counterpart to the already existing --
without-tests, which would forcefully enable checks that have been
disabled elsewhere.

Furthermore, tuned-package would still support substitution, you just
need to tell your CI to build for particular micro-architectures.  If
you're part of a sufficiently large dev team that uses J with Guix
tooling, that makes a difference between everyone running checks Monday
in the morning or simply fetching a build that has already succeeded on
Sunday.

Cheers

[bug#48463] gnu: Add j

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

Hi,

There is now a mechanism for tuning packages for particular
subarchitectures without resorting to irreproducible and
unsubstitutable packages, see
<https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/>.
 Might be useful here.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#48463] gnu: Add j.

[jgart via Guix-patches via]

Hi!

Is anyone interested in picking this up again?

-- 
all the best,
jgart