[bug#33759] [PATCH] gnu: linkchecker: Fix build.

[bug#33759] [PATCH] gnu: linkchecker: Fix build.

[Christopher Baines]

For some reason, the tests fail due to the lack of pytest. But when that's
added as an input, then some tests fail in other ways.

* gnu/packages/web.scm (linkchecker)[native-inputs]: Add python2-pytest.
[arguments]: Set #:tests? to #f.
---
 gnu/packages/web.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 84bd795a8..607f8efe6 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -5814,8 +5814,13 @@ Instagram and YouTube.")
      `(("python2-dnspython" ,python2-dnspython)
        ("python2-pyxdg" ,python2-pyxdg)
        ("python2-requests" ,python2-requests)))
+    (native-inputs
+     `(("python2-pytest" ,python2-pytest)))
     (arguments
-     `(#:python ,python-2))
+     `(#:python ,python-2
+       ;; TODO: Tests currently fail. Looks to be a mixture of trying to use
+       ;; /homeless-shelter and maybe the network.
+       #:tests? #f))
     (home-page "https://linkcheck.github.io/linkchecker")
     (synopsis "Check websites for broken links")
     (description "LinkChecker is a website validator.  It checks for broken
-- 
2.20.0

[bug#33759] [PATCH] gnu: linkchecker: Fix build.

[Ludovic Courtès]

Hello Chris!

Christopher Baines <mail@cbaines.net> skribis:

> For some reason, the tests fail due to the lack of pytest. But when that's
> added as an input, then some tests fail in other ways.
>
> * gnu/packages/web.scm (linkchecker)[native-inputs]: Add python2-pytest.
> [arguments]: Set #:tests? to #f.

[...]

> +     `(#:python ,python-2
> +       ;; TODO: Tests currently fail. Looks to be a mixture of trying to use
> +       ;; /homeless-shelter and maybe the network.
> +       #:tests? #f))

Did you try adding a phase that does (setenv "HOME" (getcwd))?

IMO it would be best to get some confidence that the test failures are
benign, possibly working around them with tricks as above.

Thanks!

Ludo’.

[bug#33759] [PATCH] gnu: linkchecker: Fix build.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello Chris!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> For some reason, the tests fail due to the lack of pytest. But when that's
>> added as an input, then some tests fail in other ways.
>>
>> * gnu/packages/web.scm (linkchecker)[native-inputs]: Add python2-pytest.
>> [arguments]: Set #:tests? to #f.
>
> [...]
>
>> +     `(#:python ,python-2
>> +       ;; TODO: Tests currently fail. Looks to be a mixture of trying to use
>> +       ;; /homeless-shelter and maybe the network.
>> +       #:tests? #f))
>
> Did you try adding a phase that does (setenv "HOME" (getcwd))?
>
> IMO it would be best to get some confidence that the test failures are
> benign, possibly working around them with tricks as above.

I've finally made some time to look at thsi more. The HOME issue seems
to not exist, and I think I've got the testsuite working now.

I noticed that some tests are skipped because they depend on the
network, and this seems helpful, but I think it's just been missed for
some tests. I've pushed a patch upstream to mark some more tests as
needing the network [1].

1: https://github.com/linkchecker/linkchecker/pull/202

I'll try to wrangle that patch in to Guix, and neaten up the other
changes, and send some updated patches soon.

Thanks for taking a look,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

[bug#33759] [PATCH v2 1/5] gnu: linkchecker: Fix build.

[Christopher Baines]

For some reason, the tests fail due to the lack of pytest. But when that's
added as an input, then some tests fail in other ways.

* gnu/packages/web.scm (linkchecker)[native-inputs]: Add python2-pytest.
[arguments]: Set #:tests? to #f.
---
 gnu/packages/web.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 84bd795a85..607f8efe69 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -5814,8 +5814,13 @@ Instagram and YouTube.")
      `(("python2-dnspython" ,python2-dnspython)
        ("python2-pyxdg" ,python2-pyxdg)
        ("python2-requests" ,python2-requests)))
+    (native-inputs
+     `(("python2-pytest" ,python2-pytest)))
     (arguments
-     `(#:python ,python-2))
+     `(#:python ,python-2
+       ;; TODO: Tests currently fail. Looks to be a mixture of trying to use
+       ;; /homeless-shelter and maybe the network.
+       #:tests? #f))
     (home-page "https://linkcheck.github.io/linkchecker")
     (synopsis "Check websites for broken links")
     (description "LinkChecker is a website validator.  It checks for broken
-- 
2.20.1

[bug#33759] [PATCH v2 2/5] gnu: Add python-miniboa.

[Christopher Baines]

python2-miniboa is required for the linkchecker testsuite.

* gnu/packages/python.scm (python-miniboa, python2-miniboa): New variable.
---
 gnu/packages/python.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 00d60333a4..852beb73da 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -1755,6 +1755,28 @@ matching them against a list of media-ranges.")
 (define-public python2-mimeparse
   (package-with-python2 python-mimeparse))
 
+(define-public python-miniboa
+  (package
+    (name "python-miniboa")
+    (version "1.0.7")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "miniboa" version))
+       (sha256
+        (base32
+         "09jh3pn4rh7kh7jayanf8jzy6gp03791b5a193w6148cf3i6k9m3"))))
+    (build-system python-build-system)
+    (home-page "https://github.com/shmup/miniboa")
+    (synopsis "Simple, single-threaded Telnet server")
+    (description
+     "Miniboa is a simple, asynchronous, single-threaded, poll-based Telnet
+server.")
+    (license license:asl2.0)))
+
+(define-public python2-miniboa
+  (package-with-python2 python-miniboa))
+
 (define-public python-pafy
   (package
     (name "python-pafy")
-- 
2.20.1

[bug#33759] [PATCH v2 3/5] gnu: python-2.7: Include /lib/python.../test/support.

[Christopher Baines]

Don't remove the support directory from the /lib/python.../test/ directory, as
this is used from the test_support module.

  /gnu/store/...-python2-2.7.15/lib/python2.7/test/test_support.py:2: in <module>
      import test.support
  E   ImportError: No module named support

* gnu/packages/python.scm (python-2.7)[arguments]: Change the 'remove-tests
phase to not remove the support directory.
---
 gnu/packages/python.scm | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 852beb73da..550643adb8 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -260,12 +260,9 @@
                                  (scandir testdir
                                           (match-lambda
                                             ((or "." "..") #f)
+                                            ("support" #f)
                                             (file
                                              (not
-                                              ;; FIXME: Add the 'support' directory
-                                              ;; in the next rebuild cycle, since it
-                                              ;; moved in 2.7.14.  See also
-                                              ;; python2-futures below.
                                               (string-prefix? "test_support."
                                                               file))))))
                        (call-with-output-file "__init__.py" (const #t))
-- 
2.20.1

[bug#33759] [PATCH v2 4/5] gnu: python-futures: Enable tests.

[Christopher Baines]

Re-enable the tests, as the python-2 package now has the support
directory. Also, switch to running the tests via the test_futures.py script to
avoid errors like:

  AttributeError: 'AsCompletedTests' object has no attribute 'executor'

* gnu/packages/python.scm (python2-futures)[arguments]: Run the
test_futures.py script to run the tests.
---
 gnu/packages/python.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 550643adb8..54bc1d4d02 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -7918,11 +7918,14 @@ otherwise matches 3.2’s API.")
          (base32
           "0rdjmmsab550kxsssdq49jcniz77zlkpw4pvi9hvib3lsskjmh4y"))))
     (build-system python-build-system)
-    (arguments `(#:python ,python-2
-                 ;; FIXME: Python 2.7.14 moved the test.support library,
-                 ;; but our package has not yet been adjusted.  Enable
-                 ;; tests when the python2 package has been fixed.
-                 #:tests? #f))
+    (arguments
+     `(#:python ,python-2
+       #:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda _
+             (invoke "python" "test_futures.py")
+             #t)))))
     (home-page "https://github.com/agronholm/pythonfutures")
     (synopsis
      "Backport of the concurrent.futures package from Python 3.2")
-- 
2.20.1

[bug#33759] [PATCH v2 5/5] gnu: linkchecker: Fix the package building.

[Christopher Baines]

* gnu/packages/python.scm (linkchecker)[source]: Add patch.
[native-inputs]: Add more inputs required for the tests.
[arguments]: Enable the tests, move the 'check phase after 'install. Test
using the installed software, and use py.test.
---
 gnu/local.mk                                  |   1 +
 ...-more-tests-that-require-the-network.patch | 182 ++++++++++++++++++
 gnu/packages/web.scm                          |  30 ++-
 3 files changed, 209 insertions(+), 4 deletions(-)
 create mode 100644 gnu/packages/patches/linkchecker-mark-more-tests-that-require-the-network.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 0bb0203354..e0a5fa60ae 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -935,6 +935,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/lierolibre-newer-libconfig.patch		\
   %D%/packages/patches/lierolibre-remove-arch-warning.patch	\
   %D%/packages/patches/lierolibre-try-building-other-arch.patch	\
+  %D%/packages/patches/linkchecker-mark-more-tests-that-require-the-network.patch.patch	\
   %D%/packages/patches/linux-pam-no-setfsuid.patch		\
   %D%/packages/patches/lirc-localstatedir.patch			\
   %D%/packages/patches/lirc-reproducible-build.patch		\
diff --git a/gnu/packages/patches/linkchecker-mark-more-tests-that-require-the-network.patch b/gnu/packages/patches/linkchecker-mark-more-tests-that-require-the-network.patch
new file mode 100644
index 0000000000..f3e488cec2
--- /dev/null
+++ b/gnu/packages/patches/linkchecker-mark-more-tests-that-require-the-network.patch
@@ -0,0 +1,182 @@
+From f24c88a0732024028fffe0372039a847e91722ea Mon Sep 17 00:00:00 2001
+From: Christopher Baines <mail@cbaines.net>
+Date: Tue, 1 Jan 2019 22:36:29 +0000
+Subject: [PATCH] Mark more tests that require the network
+
+I believe all these tests require the network, at least they seem to
+fail if it's I run them without connecting my computer to the web.
+
+I'm looking at this as part of packaging linkchecker for GNU Guix,
+where the package is build and the tests are run in a isolated
+environment, intentionally without network access, to avoid issues
+with non-reproducible package builds.
+---
+ tests/checker/test_http.py          | 2 ++
+ tests/checker/test_http_misc.py     | 2 ++
+ tests/checker/test_http_redirect.py | 2 ++
+ tests/checker/test_httpbin.py       | 5 +++++
+ tests/checker/test_misc.py          | 4 ++++
+ tests/checker/test_whitespace.py    | 3 +++
+ 6 files changed, 18 insertions(+)
+
+diff --git a/tests/checker/test_http.py b/tests/checker/test_http.py
+index e4c1e097..8a8af567 100644
+--- a/tests/checker/test_http.py
++++ b/tests/checker/test_http.py
+@@ -20,6 +20,7 @@
+ 
+ import pytest
+ 
++from tests import need_network
+ from .httpserver import HttpServerTest, CookieRedirectHttpRequestHandler
+ 
+ class TestHttp (HttpServerTest):
+@@ -29,6 +30,7 @@ def __init__(self, methodName='runTest'):
+         super(TestHttp, self).__init__(methodName=methodName)
+         self.handler = CookieRedirectHttpRequestHandler
+ 
++    @need_network
+     def test_html (self):
+         confargs = dict(recursionlevel=1)
+         self.file_test("http.html", confargs=confargs)
+diff --git a/tests/checker/test_http_misc.py b/tests/checker/test_http_misc.py
+index 9922d85f..c6b6afdb 100644
+--- a/tests/checker/test_http_misc.py
++++ b/tests/checker/test_http_misc.py
+@@ -20,11 +20,13 @@
+ import os
+ import sys
+ from .httpserver import HttpServerTest
++from tests import need_network
+ from linkcheck.network import iputil
+ 
+ class TestHttpMisc (HttpServerTest):
+     """Test http:// misc link checking."""
+ 
++    @need_network
+     def test_html (self):
+         self.swf_test()
+         self.obfuscate_test()
+diff --git a/tests/checker/test_http_redirect.py b/tests/checker/test_http_redirect.py
+index f212d98e..2253a70d 100644
+--- a/tests/checker/test_http_redirect.py
++++ b/tests/checker/test_http_redirect.py
+@@ -17,6 +17,7 @@
+ """
+ Test http checking.
+ """
++from tests import need_network
+ from .httpserver import HttpServerTest, CookieRedirectHttpRequestHandler
+ 
+ class TestHttpRedirect (HttpServerTest):
+@@ -26,6 +27,7 @@ def __init__(self, methodName='runTest'):
+         super(TestHttpRedirect, self).__init__(methodName=methodName)
+         self.handler = CookieRedirectHttpRequestHandler
+ 
++    @need_network
+     def test_redirect (self):
+         self.redirect1()
+         self.redirect2()
+diff --git a/tests/checker/test_httpbin.py b/tests/checker/test_httpbin.py
+index 0319c2f6..4c8fa846 100644
+--- a/tests/checker/test_httpbin.py
++++ b/tests/checker/test_httpbin.py
+@@ -18,6 +18,7 @@
+ Test http stuff with httpbin.org.
+ """
+ import re
++from tests import need_network
+ from . import LinkCheckTest
+ 
+ 
+@@ -30,6 +31,7 @@ def get_httpbin_url(path):
+ class TestHttpbin(LinkCheckTest):
+     """Test http:// link redirection checking."""
+ 
++    @need_network
+     def test_http_link(self):
+         linkurl = u"http://www.example.com"
+         nlinkurl = self.norm(linkurl)
+@@ -48,6 +50,7 @@ def test_http_link(self):
+         ]
+         self.direct(url, resultlines, recursionlevel=1)
+ 
++    @need_network
+     def test_basic_auth(self):
+         user = u"testuser"
+         password = u"testpassword"
+@@ -67,6 +70,7 @@ def test_basic_auth(self):
+         ]
+         self.direct(url, resultlines, confargs=confargs)
+ 
++    @need_network
+     def test_http_refresh_header(self):
+         linkurl = u"http://www.example.com"
+         nlinkurl = self.norm(linkurl)
+@@ -85,6 +89,7 @@ def test_http_refresh_header(self):
+         ]
+         self.direct(url, resultlines, recursionlevel=1)
+ 
++    @need_network
+     def test_http_content_location_header(self):
+         linkurl = u"http://www.example.com"
+         nlinkurl = self.norm(linkurl)
+diff --git a/tests/checker/test_misc.py b/tests/checker/test_misc.py
+index 2e4cfd07..f9591f9d 100644
+--- a/tests/checker/test_misc.py
++++ b/tests/checker/test_misc.py
+@@ -17,6 +17,7 @@
+ """
+ Test miscellaneous html tag parsing and URL types
+ """
++from tests import need_network
+ from . import LinkCheckTest
+ 
+ 
+@@ -25,15 +26,18 @@ class TestMisc (LinkCheckTest):
+     Test misc link types.
+     """
+ 
++    @need_network
+     def test_misc (self):
+         self.file_test("misc.html")
+ 
+     def test_html5 (self):
+         self.file_test("html5.html")
+ 
++    @need_network
+     def test_archive (self):
+         self.file_test("archive.html")
+ 
++    @need_network
+     def test_itms_services(self):
+         url = u"itms-services:?action=download-manifest&url=http://www.example.com/"
+         resultlines = [
+diff --git a/tests/checker/test_whitespace.py b/tests/checker/test_whitespace.py
+index 609c108a..fc2727d6 100644
+--- a/tests/checker/test_whitespace.py
++++ b/tests/checker/test_whitespace.py
+@@ -17,6 +17,7 @@
+ """
+ Test whitespace handling.
+ """
++from tests import need_network
+ from . import LinkCheckTest
+ 
+ 
+@@ -25,6 +26,7 @@ class TestWhitespace (LinkCheckTest):
+     Test whitespace in URLs.
+     """
+ 
++    @need_network
+     def test_leading_whitespace (self):
+         # Leading whitespace
+         url = u" http://www.example.org/"
+@@ -50,6 +52,7 @@ def test_leading_whitespace (self):
+         ]
+         self.direct(url, resultlines)
+ 
++    @need_network
+     def test_trailing_whitespace (self):
+         # Trailing whitespace
+         url = u"http://www.example.org/ "
diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 607f8efe69..65901a5aeb 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -5805,6 +5805,9 @@ Instagram and YouTube.")
        (uri (git-reference
              (url "https://github.com/linkchecker/linkchecker")
              (commit (string-append "v" version))))
+       (patches
+        (search-patches
+         "linkchecker-mark-more-tests-that-require-the-network.patch"))
        (file-name (git-file-name name version))
        (sha256
         (base32
@@ -5815,12 +5818,31 @@ Instagram and YouTube.")
        ("python2-pyxdg" ,python2-pyxdg)
        ("python2-requests" ,python2-requests)))
     (native-inputs
-     `(("python2-pytest" ,python2-pytest)))
+     `(("gettext" ,gettext-minimal)
+       ("python2-pytest" ,python2-pytest)
+       ("python2-miniboa" ,python2-miniboa)
+       ("python2-parameterized" ,python2-parameterized)))
     (arguments
      `(#:python ,python-2
-       ;; TODO: Tests currently fail. Looks to be a mixture of trying to use
-       ;; /homeless-shelter and maybe the network.
-       #:tests? #f))
+       #:phases
+       (modify-phases %standard-phases
+         ;; Move the 'check phase to after 'install, so that the installed
+         ;; library can be used
+         (delete 'check)
+         (add-after 'install 'check
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((out (assoc-ref outputs "out")))
+               ;; Set PYTHONPATH so that the installed linkchecker is used
+               (setenv "PYTHONPATH"
+                       (string-append out "/lib/python2.7/site-packages"
+                                      ":"
+                                      (getenv "PYTHONPATH")))
+               ;; Remove this directory to avoid it being used when running
+               ;; the tests
+               (delete-file-recursively "linkcheck")
+
+               (invoke "py.test" "tests"))
+             #t)))))
     (home-page "https://linkcheck.github.io/linkchecker")
     (synopsis "Check websites for broken links")
     (description "LinkChecker is a website validator.  It checks for broken
-- 
2.20.1

[bug#33759] [PATCH] gnu: linkchecker: Fix build.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

I've pushed all these patches now in one form or another as I didn't
want to forget about them. The linkchecker package still isn't building
anywhere, but once the Python fix on core-updates reaches master, then
it should be fixed.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#33759: [PATCH] gnu: linkchecker: Fix build.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> I've pushed all these patches now in one form or another as I didn't
> want to forget about them. The linkchecker package still isn't building
> anywhere, but once the Python fix on core-updates reaches master, then
> it should be fixed.

Core updates has been merged! So linkchecker finally builds on master
again.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]