[bug#27468] [PATCH] gnu: Add linkchecker.

[bug#27468] [PATCH] gnu: Add linkchecker.

[Arun Isaac]

* gnu/packages/web.scm (linkchecker): New variable.
---
 gnu/packages/web.scm | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 3625a1034..81bee1ec9 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -4674,3 +4674,43 @@ message stream (in a web server that is per connection).")
     (description "@code{httpretty} is a helper for faking web requests,
 inspired by Ruby's @code{fakeweb}.")
     (license l:expat)))
+
+(define-public linkchecker
+  (package
+    (name "linkchecker")
+    (version "9.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "LinkChecker" version))
+       (sha256
+        (base32
+         "0v8pavf0bx33xnz1kwflv0r7lxxwj7vg3syxhy2wzza0wh6sc2pf"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python2-dnspython" ,python2-dnspython)
+       ("python2-requests" ,python2-requests)))
+    (arguments
+     `(#:python ,python-2
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove faulty python-requests version check. This has been fixed
+         ;; upstream, and can be removed in version 9.4.
+         (add-after 'unpack 'remove-python-requests-version
+           (lambda _
+             (substitute* "linkcheck/__init__.py"
+               (("requests.__version__ <= '2.2.0'") "False")))))))
+    (home-page "https://linkcheck.github.io/linkchecker/")
+    (synopsis "Check websites for broken links")
+    (description "LinkChecker is a website validator.  It checks for broken
+links in websites.  It is recursive and multithreaded providing output in
+colored or normal text, HTML, SQL, CSV, XML or as a sitemap graph.  It
+supports checking HTTP/1.1, HTTPS, FTP, mailto, news, nntp, telnet and local
+file links.")
+    (license (list l:gpl2+
+                   l:bsd-2 ; linkcheck/better_exchook2.py
+                   l:bsd-3 ; linkcheck/colorama.py
+                   l:psfl  ; linkcheck/gzip2.py
+                   l:expat ; linkcheck/mem.py
+                   l:isc   ; third_party/dnspython
+                   l:asl2.0)))) ; third_party/miniboa
-- 
2.12.2

[bug#27468] Acknowledgement ([PATCH] gnu: Add linkchecker.)

[Arun Isaac]

This package has bundled dependencies -- dnspython and miniboa -- under
"third_party/". They have been modified as described in
third_party/dnspython/changelog.txt and
third_party/miniboa-r42/CHANGES.txt respectively. Should I unbundle
these dependencies? If so, how do I handle the fact that the bundled
dependencies have been locally modified?

[bug#27468] Acknowledgement ([PATCH] gnu: Add linkchecker.)

[Arun Isaac]

This package seems to have fallen through the cracks. I just sent a new
patch made against the latest master.

> This package has bundled dependencies -- dnspython and miniboa -- under
> "third_party/". They have been modified as described in
> third_party/dnspython/changelog.txt and
> third_party/miniboa-r42/CHANGES.txt respectively. Should I unbundle
> these dependencies? If so, how do I handle the fact that the bundled
> dependencies have been locally modified?

I have raised the issue of unbundling with upstream. But, there has been
no response after 10 days.

https://github.com/wummel/linkchecker/issues/729

I recommend we don't unbundle anything on our own, and let upstream do
their homework. WDYT?

[bug#27468] Acknowledgement ([PATCH] gnu: Add linkchecker.)

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Arun Isaac <arunisaac@systemreboot.net> writes:

> This package seems to have fallen through the cracks. I just sent a new
> patch made against the latest master.
>
>> This package has bundled dependencies -- dnspython and miniboa -- under
>> "third_party/". They have been modified as described in
>> third_party/dnspython/changelog.txt and
>> third_party/miniboa-r42/CHANGES.txt respectively. Should I unbundle
>> these dependencies? If so, how do I handle the fact that the bundled
>> dependencies have been locally modified?
>
> I have raised the issue of unbundling with upstream. But, there has been
> no response after 10 days.
>
> https://github.com/wummel/linkchecker/issues/729
>
> I recommend we don't unbundle anything on our own, and let upstream do
> their homework. WDYT?

Sounds reasonable.  Can you add this URL in a comment somewhere?  LGTM
apart from the cosmetic issues raised in the other message.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#27468] [PATCH] gnu: Add linkchecker.

[Arun Isaac]

* gnu/packages/web.scm (linkchecker): New variable.
---
 gnu/packages/web.scm | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 652eb6792..8116b2fd5 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -4979,3 +4979,43 @@ websites lacking feeds.  Supported websites include Facebook, Twitter,
 Instagram and YouTube.")
     (license (list l:public-domain
                    l:expat)))) ;; vendor/simplehtmldom/simple_html_dom.php
+
+(define-public linkchecker
+  (package
+    (name "linkchecker")
+    (version "9.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "LinkChecker" version))
+       (sha256
+        (base32
+         "0v8pavf0bx33xnz1kwflv0r7lxxwj7vg3syxhy2wzza0wh6sc2pf"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python2-dnspython" ,python2-dnspython)
+       ("python2-requests" ,python2-requests)))
+    (arguments
+     `(#:python ,python-2
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove faulty python-requests version check. This has been fixed
+         ;; upstream, and can be removed in version 9.4.
+         (add-after 'unpack 'remove-python-requests-version
+           (lambda _
+             (substitute* "linkcheck/__init__.py"
+               (("requests.__version__ <= '2.2.0'") "False")))))))
+    (home-page "https://linkcheck.github.io/linkchecker/")
+    (synopsis "Check websites for broken links")
+    (description "LinkChecker is a website validator.  It checks for broken
+links in websites.  It is recursive and multithreaded providing output in
+colored or normal text, HTML, SQL, CSV, XML or as a sitemap graph.  It
+supports checking HTTP/1.1, HTTPS, FTP, mailto, news, nntp, telnet and local
+file links.")
+    (license (list l:gpl2+
+                   l:bsd-2 ; linkcheck/better_exchook2.py
+                   l:bsd-3 ; linkcheck/colorama.py
+                   l:psfl  ; linkcheck/gzip2.py
+                   l:expat ; linkcheck/mem.py
+                   l:isc   ; third_party/dnspython
+                   l:asl2.0)))) ; third_party/miniboa
-- 
2.13.4

[bug#27468] [PATCH] gnu: Add linkchecker.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

Arun Isaac <arunisaac@systemreboot.net> writes:

> * gnu/packages/web.scm (linkchecker): New variable.

[...]

> +       (modify-phases %standard-phases
> +         ;; Remove faulty python-requests version check. This has been fixed
> +         ;; upstream, and can be removed in version 9.4.
> +         (add-after 'unpack 'remove-python-requests-version
> +           (lambda _
> +             (substitute* "linkcheck/__init__.py"
> +               (("requests.__version__ <= '2.2.0'") "False")))))))

Please end this phase on a #t since (substitute* ...) has an unspecified
return value.

> +    (home-page "https://linkcheck.github.io/linkchecker/")
> +    (synopsis "Check websites for broken links")
> +    (description "LinkChecker is a website validator.  It checks for broken
> +links in websites.  It is recursive and multithreaded providing output in
> +colored or normal text, HTML, SQL, CSV, XML or as a sitemap graph.  It
> +supports checking HTTP/1.1, HTTPS, FTP, mailto, news, nntp, telnet and local
> +file links.")
> +    (license (list l:gpl2+
> +                   l:bsd-2 ; linkcheck/better_exchook2.py
> +                   l:bsd-3 ; linkcheck/colorama.py
> +                   l:psfl  ; linkcheck/gzip2.py
> +                   l:expat ; linkcheck/mem.py
> +                   l:isc   ; third_party/dnspython
> +                   l:asl2.0)))) ; third_party/miniboa

Is it possible to remove these third party libraries and use packaged
copies instead?  At least 'dnspython' is already in Guix.

Also, since GPL2+ is stricter than the other licenses, the program as a
whole is effectively GPL2+, so we don't really need to mention those
one-off source files in the license field (but comments are great).

Not sure about the 3rd party bundles though, since they probably don't
depend on anything from the GPL2+ distribution.  If they are installed,
I guess we should mention them with a FIXME.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#27468] [PATCH] gnu: Add linkchecker.

[Arun Isaac]

[-- Attachment #1: Type: text/plain, Size: 2111 bytes --]


Marius Bakke writes:

>> +       (modify-phases %standard-phases
>> +         ;; Remove faulty python-requests version check. This has been fixed
>> +         ;; upstream, and can be removed in version 9.4.
>> +         (add-after 'unpack 'remove-python-requests-version
>> +           (lambda _
>> +             (substitute* "linkcheck/__init__.py"
>> +               (("requests.__version__ <= '2.2.0'") "False")))))))
>
> Please end this phase on a #t since (substitute* ...) has an unspecified
> return value.

Done.

>> +    (license (list l:gpl2+
>> +                   l:bsd-2 ; linkcheck/better_exchook2.py
>> +                   l:bsd-3 ; linkcheck/colorama.py
>> +                   l:psfl  ; linkcheck/gzip2.py
>> +                   l:expat ; linkcheck/mem.py
>> +                   l:isc   ; third_party/dnspython
>> +                   l:asl2.0)))) ; third_party/miniboa
>
> Is it possible to remove these third party libraries and use packaged
> copies instead?  At least 'dnspython' is already in Guix.

linkchecker is using a modified version of dnspython and miniboa. So, I
don't think it will be easy. For a list of modifications, look at
https://github.com/wummel/linkchecker/blob/master/third_party/dnspython/changelog.txt
and https://github.com/wummel/linkchecker/blob/master/third_party/miniboa-r42/CHANGES.txt

> Also, since GPL2+ is stricter than the other licenses, the program as a
> whole is effectively GPL2+, so we don't really need to mention those
> one-off source files in the license field (but comments are great).

But, we generally don't hide this information from the user, right? Many
Guix packages have licenses of even single files mentioned in the
`license' field. Do you still think I should remove them?

> Not sure about the 3rd party bundles though, since they probably don't
> depend on anything from the GPL2+ distribution.  If they are installed,
> I guess we should mention them with a FIXME.

I've added a FIXME.

I'm attaching a new patch with the discussed changes. Do let me know if
everything looks good.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-linkchecker.patch --]
[-- Type: text/x-patch, Size: 2605 bytes --]

From 63f4192ff8ed7341767797f4ca67ab87773e2338 Mon Sep 17 00:00:00 2001
From: Arun Isaac <arunisaac@systemreboot.net>
Date: Sat, 24 Jun 2017 05:37:45 +0530
Subject: [PATCH] gnu: Add linkchecker.

* gnu/packages/web.scm (linkchecker): New variable.
---
 gnu/packages/web.scm | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 652eb6792..68a9c2302 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -4979,3 +4979,47 @@ websites lacking feeds.  Supported websites include Facebook, Twitter,
 Instagram and YouTube.")
     (license (list l:public-domain
                    l:expat)))) ;; vendor/simplehtmldom/simple_html_dom.php
+
+(define-public linkchecker
+  (package
+    (name "linkchecker")
+    (version "9.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "LinkChecker" version))
+       (sha256
+        (base32
+         "0v8pavf0bx33xnz1kwflv0r7lxxwj7vg3syxhy2wzza0wh6sc2pf"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python2-dnspython" ,python2-dnspython)
+       ("python2-requests" ,python2-requests)))
+    (arguments
+     `(#:python ,python-2
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove faulty python-requests version check. This has been fixed
+         ;; upstream, and can be removed in version 9.4.
+         (add-after 'unpack 'remove-python-requests-version
+           (lambda _
+             (substitute* "linkcheck/__init__.py"
+               (("requests.__version__ <= '2.2.0'") "False"))
+             #t)))))
+    (home-page "https://linkcheck.github.io/linkchecker")
+    (synopsis "Check websites for broken links")
+    (description "LinkChecker is a website validator.  It checks for broken
+links in websites.  It is recursive and multithreaded providing output in
+colored or normal text, HTML, SQL, CSV, XML or as a sitemap graph.  It
+supports checking HTTP/1.1, HTTPS, FTP, mailto, news, nntp, telnet and local
+file links.")
+    (license (list l:gpl2+
+                   l:bsd-2 ; linkcheck/better_exchook2.py
+                   l:bsd-3 ; linkcheck/colorama.py
+                   l:psfl  ; linkcheck/gzip2.py
+                   l:expat ; linkcheck/mem.py
+                   ;; FIXME: Unbundle dnspython and miniboa
+                   ;; This issue has been raised upstream
+                   ;; https://github.com/wummel/linkchecker/issues/729
+                   l:isc   ; third_party/dnspython
+                   l:asl2.0)))) ; third_party/miniboa
-- 
2.13.4

[bug#27468] [PATCH] gnu: Add linkchecker.

[Arun Isaac]

[-- Attachment #1: Type: text/plain, Size: 145 bytes --]


Actually, if dnspython is currently bundled in linkchecker, there is no
need to have it in `inputs'. Attached is a patch with this change.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-linkchecker.patch --]
[-- Type: text/x-patch, Size: 2555 bytes --]

From bc092d955c8939bcf965cd3a92810e423c516fc5 Mon Sep 17 00:00:00 2001
From: Arun Isaac <arunisaac@systemreboot.net>
Date: Sat, 24 Jun 2017 05:37:45 +0530
Subject: [PATCH] gnu: Add linkchecker.

* gnu/packages/web.scm (linkchecker): New variable.
---
 gnu/packages/web.scm | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/gnu/packages/web.scm b/gnu/packages/web.scm
index 652eb6792..7f4a514db 100644
--- a/gnu/packages/web.scm
+++ b/gnu/packages/web.scm
@@ -4979,3 +4979,46 @@ websites lacking feeds.  Supported websites include Facebook, Twitter,
 Instagram and YouTube.")
     (license (list l:public-domain
                    l:expat)))) ;; vendor/simplehtmldom/simple_html_dom.php
+
+(define-public linkchecker
+  (package
+    (name "linkchecker")
+    (version "9.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "LinkChecker" version))
+       (sha256
+        (base32
+         "0v8pavf0bx33xnz1kwflv0r7lxxwj7vg3syxhy2wzza0wh6sc2pf"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python2-requests" ,python2-requests)))
+    (arguments
+     `(#:python ,python-2
+       #:phases
+       (modify-phases %standard-phases
+         ;; Remove faulty python-requests version check. This has been fixed
+         ;; upstream, and can be removed in version 9.4.
+         (add-after 'unpack 'remove-python-requests-version
+           (lambda _
+             (substitute* "linkcheck/__init__.py"
+               (("requests.__version__ <= '2.2.0'") "False"))
+             #t)))))
+    (home-page "https://linkcheck.github.io/linkchecker")
+    (synopsis "Check websites for broken links")
+    (description "LinkChecker is a website validator.  It checks for broken
+links in websites.  It is recursive and multithreaded providing output in
+colored or normal text, HTML, SQL, CSV, XML or as a sitemap graph.  It
+supports checking HTTP/1.1, HTTPS, FTP, mailto, news, nntp, telnet and local
+file links.")
+    (license (list l:gpl2+
+                   l:bsd-2 ; linkcheck/better_exchook2.py
+                   l:bsd-3 ; linkcheck/colorama.py
+                   l:psfl  ; linkcheck/gzip2.py
+                   l:expat ; linkcheck/mem.py
+                   ;; FIXME: Unbundle dnspython and miniboa
+                   ;; This issue has been raised upstream
+                   ;; https://github.com/wummel/linkchecker/issues/729
+                   l:isc   ; third_party/dnspython
+                   l:asl2.0)))) ; third_party/miniboa
-- 
2.13.4

[bug#27468] [PATCH] gnu: Add linkchecker.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2127 bytes --]

Arun Isaac <arunisaac@systemreboot.net> writes:

>>> +    (license (list l:gpl2+
>>> +                   l:bsd-2 ; linkcheck/better_exchook2.py
>>> +                   l:bsd-3 ; linkcheck/colorama.py
>>> +                   l:psfl  ; linkcheck/gzip2.py
>>> +                   l:expat ; linkcheck/mem.py
>>> +                   l:isc   ; third_party/dnspython
>>> +                   l:asl2.0)))) ; third_party/miniboa
>>
>> Is it possible to remove these third party libraries and use packaged
>> copies instead?  At least 'dnspython' is already in Guix.
>
> linkchecker is using a modified version of dnspython and miniboa. So, I
> don't think it will be easy. For a list of modifications, look at
> https://github.com/wummel/linkchecker/blob/master/third_party/dnspython/changelog.txt
> and https://github.com/wummel/linkchecker/blob/master/third_party/miniboa-r42/CHANGES.txt

OK, thanks for checking.

>> Also, since GPL2+ is stricter than the other licenses, the program as a
>> whole is effectively GPL2+, so we don't really need to mention those
>> one-off source files in the license field (but comments are great).
>
> But, we generally don't hide this information from the user, right? Many
> Guix packages have licenses of even single files mentioned in the
> `license' field. Do you still think I should remove them?

I don't know :-) Licensing is a tricky topic.  The GPL is rather
explicit in that it covers the "program as a whole".  If we treat
individual parts differently, it may be misleading to the end user.

OTOH, the GPL permits using those source files individually under their
own terms.  Perhaps the most practical solution would be to add a
"supplementary licenses" interface...

I am not a lawyer, so please use your own judgement.  No strong opinion
either way.  :-)

>> Not sure about the 3rd party bundles though, since they probably don't
>> depend on anything from the GPL2+ distribution.  If they are installed,
>> I guess we should mention them with a FIXME.
>
> I've added a FIXME.
>
> I'm attaching a new patch with the discussed changes. Do let me know if
> everything looks good.

LGTM!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#27468] [PATCH] gnu: Add linkchecker.

[Arun Isaac]

>>> Also, since GPL2+ is stricter than the other licenses, the program as a
>>> whole is effectively GPL2+, so we don't really need to mention those
>>> one-off source files in the license field (but comments are great).
>>
>> But, we generally don't hide this information from the user, right? Many
>> Guix packages have licenses of even single files mentioned in the
>> `license' field. Do you still think I should remove them?
>
> I don't know :-) Licensing is a tricky topic.  The GPL is rather
> explicit in that it covers the "program as a whole".  If we treat
> individual parts differently, it may be misleading to the end user.
>
> OTOH, the GPL permits using those source files individually under their
> own terms.  Perhaps the most practical solution would be to add a
> "supplementary licenses" interface...
>
> I am not a lawyer, so please use your own judgement.  No strong opinion
> either way.  :-)

I'm confused too. So, I have escalated this question to
guix-devel. Let's see what others have to say.

bug#27468: [PATCH] gnu: Add linkchecker.

[Arun Isaac]

Pushed! Thanks for the review!