[bug#54507] [PATCH 1/2] gnu: openfst: Add 1.7.3 for kaldi

[bug#54507] [PATCH 1/2] gnu: openfst: Add 1.7.3 for kaldi

[Aurora via Guix-patches via]

Kaldi is currently broken with openfst versions newer than 1.7.3 as it changed
some of its function signatures.

Once Kaldi is patched/fixed, this should be removed.

Reference Issues:
- https://github.com/kaldi-asr/kaldi/issues/4131
- https://github.com/kaldi-asr/kaldi/issues/4393
---
 gnu/packages/machine-learning.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index afac56f..ec70980 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -585,6 +585,19 @@ sample proximities between pairs of cases.")
 optimizing, and searching weighted finite-state transducers (FSTs).")
     (license license:asl2.0)))
 
+;; This is a temporary addition to bypass upstream issues with the kaldi
+;; package.
+(define-public openfst-1.7.3
+  (package (inherit openfst)
+    (version "1.7.3")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "http://www.openfst.org/twiki/pub/FST/"
+                                  "FstDownload/openfst-" version ".tar.gz"))
+              (sha256
+               (base32
+                "038a60w7y8qnbxmcrsim9rafz9mihsny8xv50jpzlr7rl166pp5q"))))))
+
 (define-public shogun
   (package
     (name "shogun")

base-commit: a1632d59529909ab3733772f203e7ea64809b85e
-- 
2.30.2

[bug#54507] [PATCH 2/2] gnu: kaldi: Update to 498b25d, add vector fix patch, bypass tests

[Aurora via Guix-patches via]

The patch is in the process of being upstreamed: https://github.com/kaldi-asr/kaldi/pull/4713

Either kaldi's tests are broken or openblas has reintroduced a bug:
- https://github.com/kaldi-asr/kaldi/pull/4421

More testing & fixes required upstream.
---
 gnu/packages/machine-learning.scm             | 10 ++-
 ...or-casts-by-using-explicit-namespace.patch | 89 +++++++++++++++++++
 2 files changed, 96 insertions(+), 3 deletions(-)
 create mode 100644 gnu/packages/patches/kaldi-Fix-vector-casts-by-using-explicit-namespace.patch

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index ec70980..ba6a439 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1549,8 +1549,9 @@ discrete, and conditional dimensions.")
 
 ;; There have been no proper releases yet.
 (define-public kaldi
-  (let ((commit "d4791c0f3fc1a09c042dac365e120899ee2ad21e")
-        (revision "2"))
+  (let ((commit "498b25db122ec68a96aee154b9d829030adfae4c")
+        (revision "1")
+        (openfst openfst-1.7.3)) ;; Temporary bypass for upstream issues
     (package
       (name "kaldi")
       (version (git-version "0" revision commit))
@@ -1562,7 +1563,8 @@ discrete, and conditional dimensions.")
                 (file-name (git-file-name name version))
                 (sha256
                  (base32
-                  "07k80my6f19mhrkwbzhjsnpf9871wmrwkl0ym468i830w67qyjrz"))))
+                  "06wapq8xkr2xwkj10z70w20437kqjkf6v705z68h78zrgrj9dzj9"))
+                (patches (search-patches "kaldi-Fix-vector-casts-by-using-explicit-namespace.patch"))))
       (build-system gnu-build-system)
       (arguments
        `(#:test-target "test"
@@ -1586,6 +1588,8 @@ discrete, and conditional dimensions.")
                               "gst-plugin/Makefile")
                  (("../../tools/portaudio/install")
                   (assoc-ref inputs "portaudio")))
+               (substitute* "matrix/Makefile" ;; temporary test bypass
+                            (("matrix-lib-test sparse-matrix-test") ""))
 
                ;; This `configure' script doesn't support variables passed as
                ;; arguments, nor does it support "prefix".
diff --git a/gnu/packages/patches/kaldi-Fix-vector-casts-by-using-explicit-namespace.patch b/gnu/packages/patches/kaldi-Fix-vector-casts-by-using-explicit-namespace.patch
new file mode 100644
index 0000000..5bae584
--- /dev/null
+++ b/gnu/packages/patches/kaldi-Fix-vector-casts-by-using-explicit-namespace.patch
@@ -0,0 +1,89 @@
+---
+ src/onlinebin/online-audio-server-decode-faster.cc | 2 +-
+ src/onlinebin/online-gmm-decode-faster.cc          | 4 ++--
+ src/onlinebin/online-server-gmm-decode-faster.cc   | 4 ++--
+ src/onlinebin/online-wav-gmm-decode-faster.cc      | 4 ++--
+ 4 files changed, 7 insertions(+), 7 deletions(-)
+
+diff --git a/src/onlinebin/online-audio-server-decode-faster.cc b/src/onlinebin/online-audio-server-decode-faster.cc
+index ca4bfeb88..e5faf9530 100644
+--- a/src/onlinebin/online-audio-server-decode-faster.cc
++++ b/src/onlinebin/online-audio-server-decode-faster.cc
+@@ -308,7 +308,7 @@ int32 main(int argc, char *argv[]) {
+         } else {
+           std::vector<int32> word_ids;
+           if (decoder.PartialTraceback(&out_fst)) {
+-            GetLinearSymbolSequence(out_fst, static_cast<vector<int32> *>(0),
++            GetLinearSymbolSequence(out_fst, static_cast<std::vector<int32> *>(0),
+                                     &word_ids,
+                                     static_cast<LatticeArc::Weight*>(0));
+             for (size_t i = 0; i < word_ids.size(); i++) {
+diff --git a/src/onlinebin/online-gmm-decode-faster.cc b/src/onlinebin/online-gmm-decode-faster.cc
+index 46904dbc5..74f381658 100644
+--- a/src/onlinebin/online-gmm-decode-faster.cc
++++ b/src/onlinebin/online-gmm-decode-faster.cc
+@@ -166,7 +166,7 @@ int main(int argc, char *argv[]) {
+         std::vector<int32> word_ids;
+         decoder.FinishTraceBack(&out_fst);
+         fst::GetLinearSymbolSequence(out_fst,
+-                                     static_cast<vector<int32> *>(0),
++                                     static_cast<std::vector<int32> *>(0),
+                                      &word_ids,
+                                      static_cast<LatticeArc::Weight*>(0));
+         PrintPartialResult(word_ids, word_syms, partial_res || word_ids.size());
+@@ -181,7 +181,7 @@ int main(int argc, char *argv[]) {
+         std::vector<int32> word_ids;
+         if (decoder.PartialTraceback(&out_fst)) {
+           fst::GetLinearSymbolSequence(out_fst,
+-                                       static_cast<vector<int32> *>(0),
++                                       static_cast<std::vector<int32> *>(0),
+                                        &word_ids,
+                                        static_cast<LatticeArc::Weight*>(0));
+           PrintPartialResult(word_ids, word_syms, false);
+diff --git a/src/onlinebin/online-server-gmm-decode-faster.cc b/src/onlinebin/online-server-gmm-decode-faster.cc
+index 80973bf07..9840c0f88 100644
+--- a/src/onlinebin/online-server-gmm-decode-faster.cc
++++ b/src/onlinebin/online-server-gmm-decode-faster.cc
+@@ -179,7 +179,7 @@ int main(int argc, char *argv[]) {
+       if (dstate & (decoder.kEndFeats | decoder.kEndUtt)) {
+         decoder.FinishTraceBack(&out_fst);
+         fst::GetLinearSymbolSequence(out_fst,
+-                                     static_cast<vector<int32> *>(0),
++                                     static_cast<std::vector<int32> *>(0),
+                                      &word_ids,
+                                      static_cast<LatticeArc::Weight*>(0));
+         SendPartialResult(word_ids, word_syms, partial_res || word_ids.size(),
+@@ -188,7 +188,7 @@ int main(int argc, char *argv[]) {
+       } else {
+         if (decoder.PartialTraceback(&out_fst)) {
+           fst::GetLinearSymbolSequence(out_fst,
+-                                       static_cast<vector<int32> *>(0),
++                                       static_cast<std::vector<int32> *>(0),
+                                        &word_ids,
+                                        static_cast<LatticeArc::Weight*>(0));
+           SendPartialResult(word_ids, word_syms, false,
+diff --git a/src/onlinebin/online-wav-gmm-decode-faster.cc b/src/onlinebin/online-wav-gmm-decode-faster.cc
+index fe7c6d6b9..ca482934d 100644
+--- a/src/onlinebin/online-wav-gmm-decode-faster.cc
++++ b/src/onlinebin/online-wav-gmm-decode-faster.cc
+@@ -196,7 +196,7 @@ int main(int argc, char *argv[]) {
+           std::vector<int32> word_ids;
+           decoder.FinishTraceBack(&out_fst);
+           fst::GetLinearSymbolSequence(out_fst,
+-                                       static_cast<vector<int32> *>(0),
++                                       static_cast<std::vector<int32> *>(0),
+                                        &word_ids,
+                                        static_cast<LatticeArc::Weight*>(0));
+           PrintPartialResult(word_ids, word_syms, partial_res || word_ids.size());
+@@ -220,7 +220,7 @@ int main(int argc, char *argv[]) {
+           std::vector<int32> word_ids;
+           if (decoder.PartialTraceback(&out_fst)) {
+             fst::GetLinearSymbolSequence(out_fst,
+-                                        static_cast<vector<int32> *>(0),
++                                        static_cast<std::vector<int32> *>(0),
+                                         &word_ids,
+                                         static_cast<LatticeArc::Weight*>(0));
+             PrintPartialResult(word_ids, word_syms, false);
+-- 
+2.30.2
+
-- 
2.30.2

[bug#54507] gnu: kaldi: bypass tests

[Aurora via Guix-patches via]

The bypass of tests also relates to issue #49727

[bug#54507] [PATCH v2 1/2] gnu: openfst: Add 1.7.3 for kaldi

[Aurora via Guix-patches via]

Kaldi is currently broken with openfst versions newer than 1.7.3 as it changed
some of its function signatures.

Once Kaldi is patched/fixed, this should be removed.

Reference Issues:
- https://github.com/kaldi-asr/kaldi/issues/4131
- https://github.com/kaldi-asr/kaldi/issues/4393
---
 gnu/packages/machine-learning.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index afac56f..ec70980 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -585,6 +585,19 @@ sample proximities between pairs of cases.")
 optimizing, and searching weighted finite-state transducers (FSTs).")
     (license license:asl2.0)))
 
+;; This is a temporary addition to bypass upstream issues with the kaldi
+;; package.
+(define-public openfst-1.7.3
+  (package (inherit openfst)
+    (version "1.7.3")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "http://www.openfst.org/twiki/pub/FST/"
+                                  "FstDownload/openfst-" version ".tar.gz"))
+              (sha256
+               (base32
+                "038a60w7y8qnbxmcrsim9rafz9mihsny8xv50jpzlr7rl166pp5q"))))))
+
 (define-public shogun
   (package
     (name "shogun")

base-commit: a1632d59529909ab3733772f203e7ea64809b85e
-- 
2.30.2

[bug#54507] [PATCH v2 2/2] gnu: kaldi: Update to dd107fd, bypass tests

[Aurora via Guix-patches via]

Either kaldi's tests are broken or openblas has reintroduced a bug:
- https://github.com/kaldi-asr/kaldi/pull/4421

More testing & fixes required.
---
 gnu/packages/machine-learning.scm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index ec70980..ddc1392 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1549,8 +1549,9 @@ discrete, and conditional dimensions.")
 
 ;; There have been no proper releases yet.
 (define-public kaldi
-  (let ((commit "d4791c0f3fc1a09c042dac365e120899ee2ad21e")
-        (revision "2"))
+  (let ((commit "dd107fd594ac58af962031c1689abfdc10f84452")
+        (revision "0")
+        (openfst openfst-1.7.3)) ;; Temporary bypass for upstream issues
     (package
       (name "kaldi")
       (version (git-version "0" revision commit))
@@ -1562,7 +1563,7 @@ discrete, and conditional dimensions.")
                 (file-name (git-file-name name version))
                 (sha256
                  (base32
-                  "07k80my6f19mhrkwbzhjsnpf9871wmrwkl0ym468i830w67qyjrz"))))
+                  "0iqbzgn7gzmgwvjfzifpbwwidxx887qmlgmsjkg7b1yzyfv00l21"))))
       (build-system gnu-build-system)
       (arguments
        `(#:test-target "test"
@@ -1586,6 +1587,8 @@ discrete, and conditional dimensions.")
                               "gst-plugin/Makefile")
                  (("../../tools/portaudio/install")
                   (assoc-ref inputs "portaudio")))
+               (substitute* "matrix/Makefile" ;; temporary test bypass
+                            (("matrix-lib-test sparse-matrix-test") ""))
 
                ;; This `configure' script doesn't support variables passed as
                ;; arguments, nor does it support "prefix".
-- 
2.30.2

[bug#54507] [PATCH 1/2] gnu: openfst: Add 1.7.3 for kaldi

[Ludovic Courtès]

Hi,

Aurora <rind38@disroot.org> skribis:

> Either kaldi's tests are broken or openblas has reintroduced a bug:
> - https://github.com/kaldi-asr/kaldi/pull/4421
>
> More testing & fixes required.

Can this patch be applied as-is, or are you seeing that it needs more
work?

Thanks,
Ludo’.

[bug#54507] [PATCH 1/2] gnu: openfst: Add 1.7.3 for kaldi

[Aurora via Guix-patches via]

Ludovic Courtès <ludo@gnu.org> writes:

>> Either kaldi's tests are broken or openblas has reintroduced a bug:
>> - https://github.com/kaldi-asr/kaldi/pull/4421
>>
>> More testing & fixes required.
>
> Can this patch be applied as-is, or are you seeing that it needs more
> work?

I think it needs some more work

[bug#54507] [PATCH v3 0/3] Add openfst 1.7.3 with extensions for kaldi

[Aurora via Guix-patches via]

This patchset adds a temporary openfst pin at version 1.7.3 in order to build
kaldi, as kaldi dependencies (namely openfst) have changed significantly in
the meantime and further fixes upstream are required in order to build with
newer versions.

Similarly, some of kaldi's tests are currently broken, which will also need
fixing upstream, for now they are disabled by this patchset.

This patchset also enables the building of an extension used by some other
programs dependent on openfst, although some of those have yet to be packaged.

Aurora (3):
  gnu: openfst: Add 1.7.3 for kaldi
  gnu: kaldi: Update to dd107fd, bypass tests
  gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

 gnu/packages/machine-learning.scm | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)


base-commit: f76898be6ded531e459f106549886afbdc426a78
-- 
2.30.2

[bug#54507] [PATCH v3 1/3] gnu: openfst: Add 1.7.3 for kaldi

[Aurora via Guix-patches via]

Kaldi is currently broken with openfst versions newer than 1.7.3 as it changed
some of its function signatures.

Once Kaldi is patched/fixed, this should be removed.

Reference Issues:
- https://github.com/kaldi-asr/kaldi/issues/4131
- https://github.com/kaldi-asr/kaldi/issues/4393
---
 gnu/packages/machine-learning.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index afac56f..ec70980 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -585,6 +585,19 @@ sample proximities between pairs of cases.")
 optimizing, and searching weighted finite-state transducers (FSTs).")
     (license license:asl2.0)))
 
+;; This is a temporary addition to bypass upstream issues with the kaldi
+;; package.
+(define-public openfst-1.7.3
+  (package (inherit openfst)
+    (version "1.7.3")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "http://www.openfst.org/twiki/pub/FST/"
+                                  "FstDownload/openfst-" version ".tar.gz"))
+              (sha256
+               (base32
+                "038a60w7y8qnbxmcrsim9rafz9mihsny8xv50jpzlr7rl166pp5q"))))))
+
 (define-public shogun
   (package
     (name "shogun")
-- 
2.30.2

[bug#54507] [PATCH v3 2/3] gnu: kaldi: Update to dd107fd, bypass tests

[Aurora via Guix-patches via]

Either kaldi's tests are broken or openblas has reintroduced a bug:
- https://github.com/kaldi-asr/kaldi/pull/4421

More testing & fixes required upstream to re-enable the tests later. For now
they'll just have to be ignored.
---
 gnu/packages/machine-learning.scm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index ec70980..ddc1392 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1549,8 +1549,9 @@ discrete, and conditional dimensions.")
 
 ;; There have been no proper releases yet.
 (define-public kaldi
-  (let ((commit "d4791c0f3fc1a09c042dac365e120899ee2ad21e")
-        (revision "2"))
+  (let ((commit "dd107fd594ac58af962031c1689abfdc10f84452")
+        (revision "0")
+        (openfst openfst-1.7.3)) ;; Temporary bypass for upstream issues
     (package
       (name "kaldi")
       (version (git-version "0" revision commit))
@@ -1562,7 +1563,7 @@ discrete, and conditional dimensions.")
                 (file-name (git-file-name name version))
                 (sha256
                  (base32
-                  "07k80my6f19mhrkwbzhjsnpf9871wmrwkl0ym468i830w67qyjrz"))))
+                  "0iqbzgn7gzmgwvjfzifpbwwidxx887qmlgmsjkg7b1yzyfv00l21"))))
       (build-system gnu-build-system)
       (arguments
        `(#:test-target "test"
@@ -1586,6 +1587,8 @@ discrete, and conditional dimensions.")
                               "gst-plugin/Makefile")
                  (("../../tools/portaudio/install")
                   (assoc-ref inputs "portaudio")))
+               (substitute* "matrix/Makefile" ;; temporary test bypass
+                            (("matrix-lib-test sparse-matrix-test") ""))
 
                ;; This `configure' script doesn't support variables passed as
                ;; arguments, nor does it support "prefix".
-- 
2.30.2

[bug#54507] [PATCH v3 3/3] gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

[Aurora via Guix-patches via]

The extension doesn't build in old versions of openfst such as 1.7.3 without
manually setting CXXFLAGS.

This package definition, including this workaround to build extensions
successfully, will need to be removed once kaldi is fixed upstream and can be
used with up-to-date versions of openfst.
---
 gnu/packages/machine-learning.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index ddc1392..d4a3604 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -596,7 +596,9 @@ optimizing, and searching weighted finite-state transducers (FSTs).")
                                   "FstDownload/openfst-" version ".tar.gz"))
               (sha256
                (base32
-                "038a60w7y8qnbxmcrsim9rafz9mihsny8xv50jpzlr7rl166pp5q"))))))
+                "038a60w7y8qnbxmcrsim9rafz9mihsny8xv50jpzlr7rl166pp5q"))))
+    (arguments '(#:configure-flags '("--enable-ngram-fsts" "CXXFLAGS=-std=c++14")
+                 #:make-flags '("CXXFLAGS=-std=c++14")))))
 
 (define-public shogun
   (package
-- 
2.30.2

[bug#54507] [PATCH v3 3/3] gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

[Aurora via Guix-patches via]

I just finished building this again, just to make sure.

I also added that cover letter to summarize just what the changes and
rationales behind them were.

[bug#54507] [PATCH v3 3/3] gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

[Aurora via Guix-patches via]

Aurora <rind38@disroot.org> writes:

> I just finished building this again, just to make sure.
>
> I also added that cover letter to summarize just what the changes and
> rationales behind them were.

Bumping for issue activity. It build successfully.

[bug#54507] [PATCH v3 3/3] gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

[Aurora via Guix-patches via]

Aurora <rind38@disroot.org> writes:

> Aurora <rind38@disroot.org> writes:
>
>> I just finished building this again, just to make sure.
>>
>> I also added that cover letter to summarize just what the changes and
>> rationales behind them were.
>
> Bumping for issue activity. It build successfully.

Review by a friendly #guix user pointed out the commit messages,
revision number and inline explanations need some work.

bug#54507: [PATCH 1/2] gnu: openfst: Add 1.7.3 for kaldi

[Ludovic Courtès]

Aurora <rind38@disroot.org> skribis:

>   gnu: openfst: Add 1.7.3 for kaldi
>   gnu: kaldi: Update to dd107fd, bypass tests
>   gnu: openfst-1.7.3: Enable ngram-fst extension & C++14 build

Applied to ‘staging’ as well, thanks!