[bug#36071] [PATCH] gnu: Add leela-zero.

[bug#36071] [PATCH] gnu: Add leela-zero.

[Guillaume LE VAILLANT]

* gnu/packages/games.scm (leela-zero): New variable.
* gnu/packages/patches/leela-zero-gtest.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
---
 gnu/local.mk                                |  2 +
 gnu/packages/games.scm                      | 51 +++++++++++++++++++++
 gnu/packages/patches/leela-zero-gtest.patch | 40 ++++++++++++++++
 3 files changed, 93 insertions(+)
 create mode 100644 gnu/packages/patches/leela-zero-gtest.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 55a8fcd361..19cab48aff 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -20,6 +20,7 @@
 # Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 # Copyright © 2018 Stefan Stefanović <stefanx2ovic@gmail.com>
 # Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+# Copyright © 2019 Guillaume Le Vaillant <glv@posteo.net>
 #
 # This file is part of GNU Guix.
 #
@@ -969,6 +970,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/lcms-CVE-2018-16435.patch		\
   %D%/packages/patches/ldc-bootstrap-disable-tests.patch	\
   %D%/packages/patches/ldc-disable-phobos-tests.patch		\
+  %D%/packages/patches/leela-zero-gtest.patch			\
   %D%/packages/patches/liba52-enable-pic.patch			\
   %D%/packages/patches/liba52-link-with-libm.patch		\
   %D%/packages/patches/liba52-set-soname.patch			\
diff --git a/gnu/packages/games.scm b/gnu/packages/games.scm
index 4fdc9b01e6..ee8cc69273 100644
--- a/gnu/packages/games.scm
+++ b/gnu/packages/games.scm
@@ -40,6 +40,7 @@
 ;;; Copyright © 2019 Oleg Pykhalov <go.wigust@gmail.com>
 ;;; Copyright © 2019 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2019 Julien Lepiller <julien@lepiller.eu>
+;;; Copyright © 2019 Guillaume Le Vaillant <glv@posteo.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -128,6 +129,7 @@
   #:use-module (gnu packages netpbm)
   #:use-module (gnu packages networking)
   #:use-module (gnu packages ocaml)
+  #:use-module (gnu packages opencl)
   #:use-module (gnu packages pcre)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages perl-check)
@@ -7336,3 +7338,52 @@ Unfortunately, Hacker is not aware of Drascula's real ambitions: DOMINATING
 the World and demonstrating that he is even more evil than his brother Vlad.")
     ;; Drascula uses a BSD-like license.
     (license (license:non-copyleft "file:///readme.txt"))))
+
+(define-public leela-zero
+  (package
+   (name "leela-zero")
+   (version "0.17")
+   (source
+    (origin
+     (method git-fetch)
+     (uri (git-reference
+           (url "https://github.com/leela-zero/leela-zero.git")
+           (commit (string-append "v" version))))
+     (file-name (git-file-name name version))
+     (sha256
+      (base32
+       "17px5iny8mql5c01bymcli7zfssswkzvb2i8gnsmjcck6i2n8srl"))
+     (patches (search-patches "leela-zero-gtest.patch"))))
+   (build-system cmake-build-system)
+   (native-inputs
+    `(("googletest" ,googletest)))
+   (inputs
+    `(("boost" ,boost)
+      ("ocl-icd" ,ocl-icd)
+      ("openblas" ,openblas)
+      ("opencl-headers" ,opencl-headers)
+      ("qtbase" ,qtbase)
+      ("zlib" ,zlib)))
+   (arguments
+    '(#:configure-flags '("-DUSE_BLAS=YES")
+      #:phases (modify-phases %standard-phases
+                 (add-before 'configure 'fix-tests
+                   (lambda* (#:key outputs #:allow-other-keys)
+                     (let ((home (getcwd)))
+                       (setenv "HOME" home)
+                       (substitute* "src/tests/gtests.cpp"
+                         (("\\.\\./src/tests/0k\\.txt")
+                          (string-append home "/src/tests/0k.txt"))
+                         (("cfg_gtp_mode = true;")
+                          "cfg_gtp_mode = true; cfg_cpu_only = true;")))
+                     #t))
+                 (replace 'check
+                   (lambda _
+                     (zero? (system* "./tests")))))))
+   (home-page "https://github.com/leela-zero/leela-zero")
+   (synopsis "Program playing the game of Go")
+   (description
+    "Leela-zero is a Go engine with no human-provided knowledge, modeled
+after the AlphaGo Zero paper.  The current best network weights file for
+the engine can be downloaded from https://zero.sjeng.org/best-network.")
+   (license license:gpl3+)))
diff --git a/gnu/packages/patches/leela-zero-gtest.patch b/gnu/packages/patches/leela-zero-gtest.patch
new file mode 100644
index 0000000000..7291b4ab72
--- /dev/null
+++ b/gnu/packages/patches/leela-zero-gtest.patch
@@ -0,0 +1,40 @@
+Use shared gtest instead of submodule
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 617e4b2..2857875 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -18,7 +18,6 @@ set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
+ include(GNUInstallDirs)
+ 
+ project(leelaz)
+-add_subdirectory(gtest EXCLUDE_FROM_ALL) # We don't want to install gtest, exclude it from `all`
+ 
+ # Required Packages
+ set(Boost_MIN_VERSION "1.58.0")
+@@ -27,6 +26,7 @@ find_package(Boost 1.58.0 REQUIRED program_options filesystem)
+ find_package(Threads REQUIRED)
+ find_package(ZLIB REQUIRED)
+ find_package(OpenCL REQUIRED)
++find_package(GTest REQUIRED)
+ # We need OpenBLAS for now, because we make some specific
+ # calls. Ideally we'd use OpenBLAS is possible and fall back to
+ # not doing those calls if it's not present.
+@@ -93,6 +93,7 @@ include_directories(${IncludePath})
+ include_directories(${Boost_INCLUDE_DIRS})
+ include_directories(${OpenCL_INCLUDE_DIRS})
+ include_directories(${ZLIB_INCLUDE_DIRS})
++include_directories(${GTEST_INCLUDE_DIRS})
+ 
+ if((UNIX AND NOT APPLE) OR WIN32)
+     include_directories(${BLAS_INCLUDE_DIRS})
+@@ -141,7 +142,8 @@ target_link_libraries(tests ${Boost_LIBRARIES})
+ target_link_libraries(tests ${BLAS_LIBRARIES})
+ target_link_libraries(tests ${OpenCL_LIBRARIES})
+ target_link_libraries(tests ${ZLIB_LIBRARIES})
+-target_link_libraries(tests gtest_main ${CMAKE_THREAD_LIBS_INIT})
++target_link_libraries(tests ${GTEST_BOTH_LIBRARIES})
++target_link_libraries(tests ${CMAKE_THREAD_LIBS_INIT})
+ 
+ include(GetGitRevisionDescription)
+ git_describe(VERSION --tags)
-- 
2.21.0

[bug#36071] [PATCH] gnu: Add leela-zero.

[Ludovic Courtès]

Hi,

Guillaume LE VAILLANT <glv@posteo.net> skribis:

> * gnu/packages/games.scm (leela-zero): New variable.
> * gnu/packages/patches/leela-zero-gtest.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.

[...]

> +                 (replace 'check
> +                   (lambda _
> +                     (zero? (system* "./tests")))))))

Please use ‘invoke’ instead of (zero? (system* …)).

> +    "Leela-zero is a Go engine with no human-provided knowledge, modeled
> +after the AlphaGo Zero paper.  The current best network weights file for
> +the engine can be downloaded from https://zero.sjeng.org/best-network.")
                                     ^~~~
Please use @url for URLs.

Does the game run at all if we don’t download additional network weight
data?

There’s a debate as to whether trained neural network parameters can be
considered “source”: <https://lwn.net/Articles/760142/>.

Thanks,
Ludo’.

[bug#36071] [PATCH] gnu: Add leela-zero.

[Guillaume LE VAILLANT]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

Hi,

Ludovic Courtès <ludo@gnu.org> a écrit :

> Does the game run at all if we don’t download additional network
> weight data?
> 
> There’s a debate as to whether trained neural network parameters can
> be considered “source”: <https://lwn.net/Articles/760142/>.

The game needs neural network weights to run. The user must download
a file with the weights (usually from
<https://zero.sjeng.org/best-network>, which is updated daily I think)
and put it in '$HOME/.local/share/leela-zero/'.

Do you think some weigths (e.g. taken from
<https://leela.online-go.com/networks>, for reproducibility) should be
embedded in the package?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-leela-zero.patch --]
[-- Type: text/x-patch, Size: 6466 bytes --]

From a3578a986661f6ad3fd08e8e4eb2d8f7cc8e3351 Mon Sep 17 00:00:00 2001
From: Guillaume LE VAILLANT <glv@posteo.net>
Date: Wed, 5 Jun 2019 18:11:20 +0200
Subject: [PATCH] gnu: Add leela-zero.

* gnu/packages/games.scm (leela-zero): New variable.
* gnu/packages/patches/leela-zero-gtest.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
---
 gnu/local.mk                                |  2 +
 gnu/packages/games.scm                      | 51 +++++++++++++++++++++
 gnu/packages/patches/leela-zero-gtest.patch | 40 ++++++++++++++++
 3 files changed, 93 insertions(+)
 create mode 100644 gnu/packages/patches/leela-zero-gtest.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 6878aef44a..5e2f958f26 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -20,6 +20,7 @@
 # Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 # Copyright © 2018 Stefan Stefanović <stefanx2ovic@gmail.com>
 # Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+# Copyright © 2019 Guillaume Le Vaillant <glv@posteo.net>
 #
 # This file is part of GNU Guix.
 #
@@ -968,6 +969,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/lcms-CVE-2018-16435.patch		\
   %D%/packages/patches/ldc-bootstrap-disable-tests.patch	\
   %D%/packages/patches/ldc-disable-phobos-tests.patch		\
+  %D%/packages/patches/leela-zero-gtest.patch			\
   %D%/packages/patches/liba52-enable-pic.patch			\
   %D%/packages/patches/liba52-link-with-libm.patch		\
   %D%/packages/patches/liba52-set-soname.patch			\
diff --git a/gnu/packages/games.scm b/gnu/packages/games.scm
index 4fdc9b01e6..2b3552eba8 100644
--- a/gnu/packages/games.scm
+++ b/gnu/packages/games.scm
@@ -40,6 +40,7 @@
 ;;; Copyright © 2019 Oleg Pykhalov <go.wigust@gmail.com>
 ;;; Copyright © 2019 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2019 Julien Lepiller <julien@lepiller.eu>
+;;; Copyright © 2019 Guillaume Le Vaillant <glv@posteo.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -128,6 +129,7 @@
   #:use-module (gnu packages netpbm)
   #:use-module (gnu packages networking)
   #:use-module (gnu packages ocaml)
+  #:use-module (gnu packages opencl)
   #:use-module (gnu packages pcre)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages perl-check)
@@ -7336,3 +7338,52 @@ Unfortunately, Hacker is not aware of Drascula's real ambitions: DOMINATING
 the World and demonstrating that he is even more evil than his brother Vlad.")
     ;; Drascula uses a BSD-like license.
     (license (license:non-copyleft "file:///readme.txt"))))
+
+(define-public leela-zero
+  (package
+   (name "leela-zero")
+   (version "0.17")
+   (source
+    (origin
+     (method git-fetch)
+     (uri (git-reference
+           (url "https://github.com/leela-zero/leela-zero.git")
+           (commit (string-append "v" version))))
+     (file-name (git-file-name name version))
+     (sha256
+      (base32
+       "17px5iny8mql5c01bymcli7zfssswkzvb2i8gnsmjcck6i2n8srl"))
+     (patches (search-patches "leela-zero-gtest.patch"))))
+   (build-system cmake-build-system)
+   (native-inputs
+    `(("googletest" ,googletest)))
+   (inputs
+    `(("boost" ,boost)
+      ("ocl-icd" ,ocl-icd)
+      ("openblas" ,openblas)
+      ("opencl-headers" ,opencl-headers)
+      ("qtbase" ,qtbase)
+      ("zlib" ,zlib)))
+   (arguments
+    '(#:configure-flags '("-DUSE_BLAS=YES")
+      #:phases (modify-phases %standard-phases
+                 (add-before 'configure 'fix-tests
+                   (lambda* (#:key outputs #:allow-other-keys)
+                     (let ((home (getcwd)))
+                       (setenv "HOME" home)
+                       (substitute* "src/tests/gtests.cpp"
+                         (("\\.\\./src/tests/0k\\.txt")
+                          (string-append home "/src/tests/0k.txt"))
+                         (("cfg_gtp_mode = true;")
+                          "cfg_gtp_mode = true; cfg_cpu_only = true;")))
+                     #t))
+                 (replace 'check
+                   (lambda _
+                     (invoke "./tests"))))))
+   (home-page "https://github.com/leela-zero/leela-zero")
+   (synopsis "Program playing the game of Go")
+   (description
+    "Leela-zero is a Go engine with no human-provided knowledge, modeled after
+the AlphaGo Zero paper.  The current best network weights file for the engine
+can be downloaded from @url{https://zero.sjeng.org/best-network}.")
+   (license license:gpl3+)))
diff --git a/gnu/packages/patches/leela-zero-gtest.patch b/gnu/packages/patches/leela-zero-gtest.patch
new file mode 100644
index 0000000000..7291b4ab72
--- /dev/null
+++ b/gnu/packages/patches/leela-zero-gtest.patch
@@ -0,0 +1,40 @@
+Use shared gtest instead of submodule
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 617e4b2..2857875 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -18,7 +18,6 @@ set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
+ include(GNUInstallDirs)
+ 
+ project(leelaz)
+-add_subdirectory(gtest EXCLUDE_FROM_ALL) # We don't want to install gtest, exclude it from `all`
+ 
+ # Required Packages
+ set(Boost_MIN_VERSION "1.58.0")
+@@ -27,6 +26,7 @@ find_package(Boost 1.58.0 REQUIRED program_options filesystem)
+ find_package(Threads REQUIRED)
+ find_package(ZLIB REQUIRED)
+ find_package(OpenCL REQUIRED)
++find_package(GTest REQUIRED)
+ # We need OpenBLAS for now, because we make some specific
+ # calls. Ideally we'd use OpenBLAS is possible and fall back to
+ # not doing those calls if it's not present.
+@@ -93,6 +93,7 @@ include_directories(${IncludePath})
+ include_directories(${Boost_INCLUDE_DIRS})
+ include_directories(${OpenCL_INCLUDE_DIRS})
+ include_directories(${ZLIB_INCLUDE_DIRS})
++include_directories(${GTEST_INCLUDE_DIRS})
+ 
+ if((UNIX AND NOT APPLE) OR WIN32)
+     include_directories(${BLAS_INCLUDE_DIRS})
+@@ -141,7 +142,8 @@ target_link_libraries(tests ${Boost_LIBRARIES})
+ target_link_libraries(tests ${BLAS_LIBRARIES})
+ target_link_libraries(tests ${OpenCL_LIBRARIES})
+ target_link_libraries(tests ${ZLIB_LIBRARIES})
+-target_link_libraries(tests gtest_main ${CMAKE_THREAD_LIBS_INIT})
++target_link_libraries(tests ${GTEST_BOTH_LIBRARIES})
++target_link_libraries(tests ${CMAKE_THREAD_LIBS_INIT})
+ 
+ include(GetGitRevisionDescription)
+ git_describe(VERSION --tags)
-- 
2.21.0

[bug#36071] [PATCH] gnu: Add leela-zero.

[Ludovic Courtès]

Hi Guillaume,

Guillaume LE VAILLANT <glv@posteo.net> skribis:

> Ludovic Courtès <ludo@gnu.org> a écrit :
>
>> Does the game run at all if we don’t download additional network
>> weight data?
>> 
>> There’s a debate as to whether trained neural network parameters can
>> be considered “source”: <https://lwn.net/Articles/760142/>.
>
> The game needs neural network weights to run. The user must download
> a file with the weights (usually from
> <https://zero.sjeng.org/best-network>, which is updated daily I think)
> and put it in '$HOME/.local/share/leela-zero/'.
>
> Do you think some weigths (e.g. taken from
> <https://leela.online-go.com/networks>, for reproducibility) should be
> embedded in the package?

My point is about whether these trained neural network data are
something that we could distribute per the FSDG.

The LWN article above mentions ‘leela-zero’.  In Debian (which has a
different policy), ‘leela-zero’ has apparently been accepted, but
without the data, and with a link to said data (as in your original
patch):

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903634

Perhaps we could do the same, but I’d like to hear what others think.

Thank you,
Ludo’.

[bug#36071] [PATCH] gnu: Add leela-zero.

[zimoun]

Hi,

Thank you for the package.

On Wed, 5 Jun 2019 at 23:03, Ludovic Courtès <ludo@gnu.org> wrote:

> > Ludovic Courtès <ludo@gnu.org> a écrit :
> >
> >> Does the game run at all if we don’t download additional network
> >> weight data?
> >>
> >> There’s a debate as to whether trained neural network parameters can
> >> be considered “source”: <https://lwn.net/Articles/760142/>.
> >

[...]

> My point is about whether these trained neural network data are
> something that we could distribute per the FSDG.

[...]

>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903634

In addition, the thread on debian-devel exposes arguments (summarized by LWN).
See there:
https://lists.debian.org/debian-devel/2018/07/msg00153.html


> Perhaps we could do the same, but I’d like to hear what others think.

I need myself to digest the issue.
Hum? FSF has never risen the issue---or I am not aware of.

Aside the practical discussion about this package---thank you for the
work!---, there is a real concern about Reproducibility and Freedom
with the weights of statistical models. However, this is perhaps not
so far from free client / proprietary database situation.

Well, it should be better to discuss this concern on
guix-devel---risking bikeshedding. :-)
At least to collect contradictory arguments.


All the best,
simon

[bug#36071] [PATCH] gnu: Add leela-zero.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> Ludovic Courtès <ludo@gnu.org> a écrit :
>>
>>> Does the game run at all if we don’t download additional network
>>> weight data?
>>>
>>> There’s a debate as to whether trained neural network parameters can
>>> be considered “source”: <https://lwn.net/Articles/760142/>.
>>
>> The game needs neural network weights to run. The user must download
>> a file with the weights (usually from
>> <https://zero.sjeng.org/best-network>, which is updated daily I think)
>> and put it in '$HOME/.local/share/leela-zero/'.
>>
>> Do you think some weigths (e.g. taken from
>> <https://leela.online-go.com/networks>, for reproducibility) should be
>> embedded in the package?
>
> My point is about whether these trained neural network data are
> something that we could distribute per the FSDG.
>
> The LWN article above mentions ‘leela-zero’.  In Debian (which has a
> different policy), ‘leela-zero’ has apparently been accepted, but
> without the data, and with a link to said data (as in your original
> patch):
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903634
>
> Perhaps we could do the same, but I’d like to hear what others think.

We need to ensure that the software necessary to train the networks is
included.  Is this the case?

Distributing the training data is not an option, as I see it, because
the files are very large.

I don’t know about the networks that are offered for download at
https://leela.online-go.com/networks/.   These are files of very
different sizes, and it is neither clear what license applies nor how
they were arrived at.  They are zipped up magic numbers.

Arguably a similar statement could be made about impulse response files
for audio filters (such as convolution reverbs).  While we offer
convolution filters (such as the “ir” package), we don’t include data
files.  Had I not made the link to network weights for ML models I would
have considered impulse response files no different from e.g. game
assets.  Now I’m not so sure any more.

I don’t really know what it really means for data to be
“non-functional”.

Back to this patch: I think it’s fine to accept it as long as the
software necessary for training is included.

--
Ricardo

[bug#36071] [PATCH] gnu: Add leela-zero.

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

Ricardo Wurmus wrote:
> Back to this patch: I think it’s fine to accept it as long as 
> the
> software necessary for training is included.

I can't assess its accuracy, but this comment[0] from the linked 
LWN article looks relevant.  Particularly:

--8<---------------cut here---------------start------------->8---
So this is basically a distributed computing client as well as a 
Go
engine that runs with the results of that distributed computing 
effort.
--8<---------------cut here---------------end--------------->8---

If that's true, there is no separate ‘training software’ to worry 
about.

The whole link is worth a click since there seems to be a ‘server 
component’ involved as well.

Kind regards,

T G-R

[0]: https://lwn.net/Articles/760483/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#36071: [PATCH] gnu: Add leela-zero.

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Back to this patch: I think it’s fine to accept it as long as the
> software necessary for training is included.

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> I can't assess its accuracy, but this comment[0] from the linked 
> LWN article looks relevant.  Particularly:
>
> So this is basically a distributed computing client as well as a 
> Go
> engine that runs with the results of that distributed computing 
> effort.
>
> If that's true, there is no separate ‘training software’ to worry 
> about.
>
> The whole link is worth a click since there seems to be a ‘server 
> component’ involved as well.

Based on the info we have so far, including the fact that Debian
includes it (see <https://packages.debian.org/leela-zero>), I went ahead
and applied the patch.

We can obviously revisit this issue later should someone have concerns
and/or additional info.

Thanks everyone!

Ludo’.