[bug#38208] [PATCH] gnu: Add python-3.8.

[bug#38208] [PATCH] gnu: Add python-3.8.

[Tanguy Le Carrour]

* gnu/packages/python.scm (python-3.8): New public variable.
[source]: Add patches to skip four tests.
* gnu/packages/patches/python-3.8-search-paths.patch: New file.
* gnu/packages/patches/python-3.8-fix-tests.patch: New file.
---
 .../patches/python-3.8-fix-tests.patch        | 42 +++++++++++++++++++
 .../patches/python-3.8-search-paths.patch     | 17 ++++++++
 gnu/packages/python.scm                       | 25 +++++++++++
 3 files changed, 84 insertions(+)
 create mode 100644 gnu/packages/patches/python-3.8-fix-tests.patch
 create mode 100644 gnu/packages/patches/python-3.8-search-paths.patch

diff --git a/gnu/packages/patches/python-3.8-fix-tests.patch b/gnu/packages/patches/python-3.8-fix-tests.patch
new file mode 100644
index 0000000000..bea311992a
--- /dev/null
+++ b/gnu/packages/patches/python-3.8-fix-tests.patch
@@ -0,0 +1,42 @@
+diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
+index 1474624..887f8ee 100644
+--- a/Lib/test/_test_multiprocessing.py
++++ b/Lib/test/_test_multiprocessing.py
+@@ -3801,6 +3801,7 @@ class _TestSharedMemory(BaseTestCase):
+         sms.close()
+
+     @unittest.skipIf(os.name != "posix", "not feasible in non-posix platforms")
++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
+     def test_shared_memory_SharedMemoryServer_ignores_sigint(self):
+         # bpo-36368: protect SharedMemoryManager server process from
+         # KeyboardInterrupt signals.
+diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py
+index d41e94b..a1c15e7 100644
+--- a/Lib/test/test_signal.py
++++ b/Lib/test/test_signal.py
+@@ -78,6 +78,7 @@ class PosixTests(unittest.TestCase):
+         self.assertLess(len(s), signal.NSIG)
+
+     @unittest.skipUnless(sys.executable, "sys.executable required.")
++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
+     def test_keyboard_interrupt_exit_code(self):
+         """KeyboardInterrupt triggers exit via SIGINT."""
+         process = subprocess.run(
+@@ -128,6 +129,7 @@ class WindowsSignalTests(unittest.TestCase):
+             signal.signal(7, handler)
+
+     @unittest.skipUnless(sys.executable, "sys.executable required.")
++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
+     def test_keyboard_interrupt_exit_code(self):
+         """KeyboardInterrupt triggers an exit using STATUS_CONTROL_C_EXIT."""
+         # We don't test via os.kill(os.getpid(), signal.CTRL_C_EVENT) here
+@@ -1245,6 +1247,7 @@ class StressTest(unittest.TestCase):
+
+ class RaiseSignalTest(unittest.TestCase):
+
++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
+     def test_sigint(self):
+         with self.assertRaises(KeyboardInterrupt):
+             signal.raise_signal(signal.SIGINT)
+--
+2.23.0
diff --git a/gnu/packages/patches/python-3.8-search-paths.patch b/gnu/packages/patches/python-3.8-search-paths.patch
new file mode 100644
index 0000000000..22e102ccbe
--- /dev/null
+++ b/gnu/packages/patches/python-3.8-search-paths.patch
@@ -0,0 +1,17 @@
+diff --git a/setup.py b/setup.py
+index 20d7f35..5751083 100644
+--- a/setup.py
++++ b/setup.py
+@@ -676,8 +676,8 @@ class PyBuildExt(build_ext):
+         # if a file is found in one of those directories, it can
+         # be assumed that no additional -I,-L directives are needed.
+         if not CROSS_COMPILING:
+-            self.lib_dirs = self.compiler.library_dirs + system_lib_dirs
+-            self.inc_dirs = self.compiler.include_dirs + system_include_dirs
++            self.lib_dirs = os.getenv('LIBRARY_PATH', '').split(os.pathsep)
++            self.inc_dirs = os.getenv('CPATH', '').split(os.pathsep)
+         else:
+             # Add the sysroot paths. 'sysroot' is a compiler option used to
+             # set the logical path of the standard system headers and
+--
+2.23.0
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 644c9d7666..8a2d3e9721 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -57,6 +57,7 @@
 ;;; Copyright © 2018, 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2018 Luther Thompson <lutheroto@gmail.com>
 ;;; Copyright © 2018 Vagrant Cascadian <vagrant@debian.org>
+;;; Copyright © 2019 Tanguy Le Carrour <tanguy@bioneland.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -408,6 +409,30 @@ data types.")
                                         (version-major+minor version)
                                         "/site-packages"))))))))
 
+(define-public python-3.8
+  (package
+    (inherit python-3.7)
+    (version "3.8.0")
+    (source
+     (origin
+       (inherit (package-source python-3.7))
+       (uri (string-append "https://www.python.org/ftp/python/"
+                           version "/Python-" version ".tar.xz"))
+       (sha256 (base32 "110d0did9rxn7rg85kf2fwli5hqq44xv2d8bi7d92m7v2d728mmk"))
+       (patches (search-patches
+                 "python-3.8-search-paths.patch"
+                 "python-3-fix-tests.patch"
+                 "python-3.8-fix-tests.patch"
+                 "python-3-deterministic-build-info.patch"))
+       (snippet
+        '(begin
+           ;; Delete the bundled copy of libexpat.
+           (delete-file-recursively "Modules/expat")
+           (substitute* "Modules/Setup"
+             ;; Link Expat instead of embedding the bundled one.
+             (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+           #t))))))
+
 ;; Current 3.x version.
 (define-public python-3 python-3.7)
 
-- 
2.24.0

[bug#38208] [PATCH] gnu: Add python-3.8.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2415 bytes --]

Hello Tanguy,

Tanguy Le Carrour <tanguy@bioneland.org> writes:

> * gnu/packages/python.scm (python-3.8): New public variable.
> [source]: Add patches to skip four tests.
> * gnu/packages/patches/python-3.8-search-paths.patch: New file.
> * gnu/packages/patches/python-3.8-fix-tests.patch: New file.

Thank you for this!

> ++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
> +     def test_keyboard_interrupt_exit_code(self):
> +         """KeyboardInterrupt triggers exit via SIGINT."""
> +         process = subprocess.run(

I'm fairly confident that these failures are because stdin is not a
TTY.  Can you try changing to this?

@unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")

It would be good to submit a patch along these lines upstream if that
works.

> +(define-public python-3.8
> +  (package
> +    (inherit python-3.7)
> +    (version "3.8.0")
> +    (source
> +     (origin
> +       (inherit (package-source python-3.7))
> +       (uri (string-append "https://www.python.org/ftp/python/"
> +                           version "/Python-" version ".tar.xz"))
> +       (sha256 (base32 "110d0did9rxn7rg85kf2fwli5hqq44xv2d8bi7d92m7v2d728mmk"))
> +       (patches (search-patches
> +                 "python-3.8-search-paths.patch"
> +                 "python-3-fix-tests.patch"
> +                 "python-3.8-fix-tests.patch"
> +                 "python-3-deterministic-build-info.patch"))
> +       (snippet
> +        '(begin
> +           ;; Delete the bundled copy of libexpat.
> +           (delete-file-recursively "Modules/expat")
> +           (substitute* "Modules/Setup"
> +             ;; Link Expat instead of embedding the bundled one.
> +             (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
> +           #t))))))

One final question...  Do you know how stable the Python ABI is?  Are
packages compiled with 3.7 guaranteed to work with 3.8?

I wonder if we should 1) rename it to 'python-next', and 2) override
'native-search-paths' so that it searches 'lib/python3.8/site-packages'.

My concern is that e.g. 'guix environment --ad-hoc python python-numpy'
might stop working unless we rename it, and/or adjust search paths.

If Python 3.8 is fully backwards compatible, we probably don't have to
do anything, otherwise we should adjust accordingly.

Thoughts?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#38208] [PATCH] gnu: Add python-3.8.

[Tanguy Le Carrour]

Hi Marius,


Le 11/15, Marius Bakke a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > ++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
> > +     def test_keyboard_interrupt_exit_code(self):
> > +         """KeyboardInterrupt triggers exit via SIGINT."""
> > +         process = subprocess.run(
> 
> I'm fairly confident that these failures are because stdin is not a
> TTY.  Can you try changing to this?
> 
> @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
> 
> It would be good to submit a patch along these lines upstream if that
> works.

Thanks for the snippet! It works!

I've been trying for 2 days to register on https://bugs.python.org to submit
this patch, but failed! :-(
I'll keep on trying with different email addresses…

Does this mean that we have to wait for upstream to integrate it, or can
we use the patched wersion for Guix?
Anyway, I'll update this patch according to your suggestion and push it.


> > +(define-public python-3.8
> > +  (package
> > +    (inherit python-3.7)
> > +    (version "3.8.0")
> > +    (source
> > +     (origin
> > +       (inherit (package-source python-3.7))
> > +       (uri (string-append "https://www.python.org/ftp/python/"
> > +                           version "/Python-" version ".tar.xz"))
> > +       (sha256 (base32 "110d0did9rxn7rg85kf2fwli5hqq44xv2d8bi7d92m7v2d728mmk"))
> > +       (patches (search-patches
> > +                 "python-3.8-search-paths.patch"
> > +                 "python-3-fix-tests.patch"
> > +                 "python-3.8-fix-tests.patch"
> > +                 "python-3-deterministic-build-info.patch"))
> > +       (snippet
> > +        '(begin
> > +           ;; Delete the bundled copy of libexpat.
> > +           (delete-file-recursively "Modules/expat")
> > +           (substitute* "Modules/Setup"
> > +             ;; Link Expat instead of embedding the bundled one.
> > +             (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
> > +           #t))))))
> 
> One final question...  Do you know how stable the Python ABI is?  Are
> packages compiled with 3.7 guaranteed to work with 3.8?
> 
> I wonder if we should 1) rename it to 'python-next', and 2) override
> 'native-search-paths' so that it searches 'lib/python3.8/site-packages'.
> 
> My concern is that e.g. 'guix environment --ad-hoc python python-numpy'
> might stop working unless we rename it, and/or adjust search paths.
> 
> If Python 3.8 is fully backwards compatible, we probably don't have to
> do anything, otherwise we should adjust accordingly.

I don't know anything about ABI, but what I read here:
https://www.python.org/dev/peps/pep-0384/
So I guess it's been taken into consideration!

I might have some time later this week to test using Python 3.8 as
default Python. I'll let you know!

Regards,

-- 
Tanguy

[bug#38208] [PATCH v2] gnu: Add python-3.8.

[Tanguy Le Carrour]

* gnu/packages/python.scm (python-3.8): New public variable.
[source]: Add patches to skip four tests.
* gnu/packages/patches/python-3.8-search-paths.patch: New file.
* gnu/packages/patches/python-3.8-fix-tests.patch: New file.
---
 .../patches/python-3.8-fix-tests.patch        | 42 +++++++++++++++++++
 .../patches/python-3.8-search-paths.patch     | 17 ++++++++
 gnu/packages/python.scm                       | 25 +++++++++++
 3 files changed, 84 insertions(+)
 create mode 100644 gnu/packages/patches/python-3.8-fix-tests.patch
 create mode 100644 gnu/packages/patches/python-3.8-search-paths.patch

diff --git a/gnu/packages/patches/python-3.8-fix-tests.patch b/gnu/packages/patches/python-3.8-fix-tests.patch
new file mode 100644
index 0000000000..45af383631
--- /dev/null
+++ b/gnu/packages/patches/python-3.8-fix-tests.patch
@@ -0,0 +1,42 @@
+diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
+index 1474624..887f8ee 100644
+--- a/Lib/test/_test_multiprocessing.py
++++ b/Lib/test/_test_multiprocessing.py
+@@ -3801,6 +3801,7 @@ class _TestSharedMemory(BaseTestCase):
+         sms.close()
+
+     @unittest.skipIf(os.name != "posix", "not feasible in non-posix platforms")
++    @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
+     def test_shared_memory_SharedMemoryServer_ignores_sigint(self):
+         # bpo-36368: protect SharedMemoryManager server process from
+         # KeyboardInterrupt signals.
+diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py
+index d41e94b..a1c15e7 100644
+--- a/Lib/test/test_signal.py
++++ b/Lib/test/test_signal.py
+@@ -78,6 +78,7 @@ class PosixTests(unittest.TestCase):
+         self.assertLess(len(s), signal.NSIG)
+
+     @unittest.skipUnless(sys.executable, "sys.executable required.")
++    @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
+     def test_keyboard_interrupt_exit_code(self):
+         """KeyboardInterrupt triggers exit via SIGINT."""
+         process = subprocess.run(
+@@ -128,6 +129,7 @@ class WindowsSignalTests(unittest.TestCase):
+             signal.signal(7, handler)
+
+     @unittest.skipUnless(sys.executable, "sys.executable required.")
++    @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
+     def test_keyboard_interrupt_exit_code(self):
+         """KeyboardInterrupt triggers an exit using STATUS_CONTROL_C_EXIT."""
+         # We don't test via os.kill(os.getpid(), signal.CTRL_C_EVENT) here
+@@ -1245,6 +1247,7 @@ class StressTest(unittest.TestCase):
+
+ class RaiseSignalTest(unittest.TestCase):
+
++    @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
+     def test_sigint(self):
+         with self.assertRaises(KeyboardInterrupt):
+             signal.raise_signal(signal.SIGINT)
+--
+2.23.0
diff --git a/gnu/packages/patches/python-3.8-search-paths.patch b/gnu/packages/patches/python-3.8-search-paths.patch
new file mode 100644
index 0000000000..22e102ccbe
--- /dev/null
+++ b/gnu/packages/patches/python-3.8-search-paths.patch
@@ -0,0 +1,17 @@
+diff --git a/setup.py b/setup.py
+index 20d7f35..5751083 100644
+--- a/setup.py
++++ b/setup.py
+@@ -676,8 +676,8 @@ class PyBuildExt(build_ext):
+         # if a file is found in one of those directories, it can
+         # be assumed that no additional -I,-L directives are needed.
+         if not CROSS_COMPILING:
+-            self.lib_dirs = self.compiler.library_dirs + system_lib_dirs
+-            self.inc_dirs = self.compiler.include_dirs + system_include_dirs
++            self.lib_dirs = os.getenv('LIBRARY_PATH', '').split(os.pathsep)
++            self.inc_dirs = os.getenv('CPATH', '').split(os.pathsep)
+         else:
+             # Add the sysroot paths. 'sysroot' is a compiler option used to
+             # set the logical path of the standard system headers and
+--
+2.23.0
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 644c9d7666..8a2d3e9721 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -57,6 +57,7 @@
 ;;; Copyright © 2018, 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2018 Luther Thompson <lutheroto@gmail.com>
 ;;; Copyright © 2018 Vagrant Cascadian <vagrant@debian.org>
+;;; Copyright © 2019 Tanguy Le Carrour <tanguy@bioneland.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -408,6 +409,30 @@ data types.")
                                         (version-major+minor version)
                                         "/site-packages"))))))))
 
+(define-public python-3.8
+  (package
+    (inherit python-3.7)
+    (version "3.8.0")
+    (source
+     (origin
+       (inherit (package-source python-3.7))
+       (uri (string-append "https://www.python.org/ftp/python/"
+                           version "/Python-" version ".tar.xz"))
+       (sha256 (base32 "110d0did9rxn7rg85kf2fwli5hqq44xv2d8bi7d92m7v2d728mmk"))
+       (patches (search-patches
+                 "python-3.8-search-paths.patch"
+                 "python-3-fix-tests.patch"
+                 "python-3.8-fix-tests.patch"
+                 "python-3-deterministic-build-info.patch"))
+       (snippet
+        '(begin
+           ;; Delete the bundled copy of libexpat.
+           (delete-file-recursively "Modules/expat")
+           (substitute* "Modules/Setup"
+             ;; Link Expat instead of embedding the bundled one.
+             (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+           #t))))))
+
 ;; Current 3.x version.
 (define-public python-3 python-3.7)
 
-- 
2.24.0

[bug#38208] Acknowledgement ([PATCH] gnu: Add python-3.8.)

[Tanguy Le Carrour]

Patch submitted upstream: https://bugs.python.org/issue38845

[bug#38208] [PATCH] gnu: Add python-3.8.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2915 bytes --]

Tanguy Le Carrour <tanguy@bioneland.org> writes:

> Hi Marius,
>
>
> Le 11/15, Marius Bakke a écrit :
>> Tanguy Le Carrour <tanguy@bioneland.org> writes:
>> > ++    @unittest.skipIf(True, "This fails for unknown reasons on Guix")
>> > +     def test_keyboard_interrupt_exit_code(self):
>> > +         """KeyboardInterrupt triggers exit via SIGINT."""
>> > +         process = subprocess.run(
>> 
>> I'm fairly confident that these failures are because stdin is not a
>> TTY.  Can you try changing to this?
>> 
>> @unittest.skipUnless(sys.stdin.isatty(), "KeyboardInterrupts require a TTY device")
>> 
>> It would be good to submit a patch along these lines upstream if that
>> works.
>
> Thanks for the snippet! It works!
>
> I've been trying for 2 days to register on https://bugs.python.org to submit
> this patch, but failed! :-(
> I'll keep on trying with different email addresses…

Excellent, thank you very much for taking care of this.  There are five
other tests skipped for the same reason in 'python-3-fix-tests.patch',
can you include those in the upstream pull request?

> Does this mean that we have to wait for upstream to integrate it, or can
> we use the patched wersion for Guix?

I think the patch looks OK, but looking forward to hear upstreams take
on it.  In any case, it should not be a blocker.

> Anyway, I'll update this patch according to your suggestion and push it.

Push it to this bug report, presumably?  :-)

>> One final question...  Do you know how stable the Python ABI is?  Are
>> packages compiled with 3.7 guaranteed to work with 3.8?
>> 
>> I wonder if we should 1) rename it to 'python-next', and 2) override
>> 'native-search-paths' so that it searches 'lib/python3.8/site-packages'.
>> 
>> My concern is that e.g. 'guix environment --ad-hoc python python-numpy'
>> might stop working unless we rename it, and/or adjust search paths.
>> 
>> If Python 3.8 is fully backwards compatible, we probably don't have to
>> do anything, otherwise we should adjust accordingly.
>
> I don't know anything about ABI, but what I read here:
> https://www.python.org/dev/peps/pep-0384/
> So I guess it's been taken into consideration!

Cool, thanks for the link.  So it is safe to swap out Python 3.7 with
3.8 "in-place".  However I think we should still call it differently,
because the new Python won't necessarily be compatible with all the
manifests/environments out there that reference Python by name.

So if you send a patch that adds (name "python-next") to the inherited
Python package, I think we are all set.

Thank you!

> I might have some time later this week to test using Python 3.8 as
> default Python. I'll let you know!

Great, good luck!  I suspect you'll have to update many of the 'core'
Python packages along the way; feel free to drop by #guix on Freenode if
you get stuck anywhere.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#38208: [PATCH v2] gnu: Add python-3.8.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 134 bytes --]

Hi Tanguy,

Whoops, did not notice this patch in my previous message!

I've applied it with the name change as suggested.

Thank you!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]