[bug#55044] [PATCH 0/8] Add python-scrapy.

[bug#55044] [PATCH 0/8] Add python-scrapy.

[Felix Gruber]

This patchset adds the Scrapy webscraping framework.

Felix Gruber (8):
  gnu: Add python-sybil.
  gnu: Add python-pydispatcher.
  gnu: Add python-queuelib.
  gnu: Add python-itemadapter.
  gnu: Add python-protego.
  gnu: Add python-parsel.
  gnu: Add python-itemloaders.
  gnu: Add python-scrapy.

 gnu/packages/python-check.scm |  30 +++++++++
 gnu/packages/python-web.scm   | 113 ++++++++++++++++++++++++++++++++++
 gnu/packages/python-xyz.scm   | 111 ++++++++++++++++++++++++++++++++-
 3 files changed, 253 insertions(+), 1 deletion(-)

-- 
2.30.2

[bug#55044] [PATCH 1/8] gnu: Add python-sybil.

[Felix Gruber]

* gnu/packages/python-check.scm (python-sybil): New variable.
---
 gnu/packages/python-check.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm
index 2d35eb720e..1576a38b09 100644
--- a/gnu/packages/python-check.scm
+++ b/gnu/packages/python-check.scm
@@ -14,6 +14,7 @@
 ;;; Copyright © 2021, 2022 Guillaume Le Vaillant <glv@posteo.net>
 ;;; Copyright © 2021 Bonface Munyoki Kilyungi <me@bonfacemunyoki.com>
 ;;; Copyright © 2022 Malte Frank Gerdes <malte.f.gerdes@gmail.com>
+;;; Copyright © 2022 Felix Gruber <felgru@posteo.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -2056,3 +2057,32 @@ eliminate flaky failures.")
 Python objects.  It tries to use the objects available in the standard
 @code{unittest} module.")
     (license license:expat)))
+
+(define-public python-sybil
+  (package
+    (name "python-sybil")
+    (version "3.0.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "sybil" version))
+        (sha256
+          (base32 "03ak1w93linfqx6c9lwgq5niyy3j9yblv4ip40hmlzmg0hidq0kg"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+        (modify-phases %standard-phases
+          (replace 'check
+            (lambda* (#:key tests? #:allow-other-keys)
+              (when tests?
+                (invoke "pytest")))))))
+    (native-inputs (list python-pytest python-pytest-cov))
+    (home-page "https://github.com/simplistix/sybil")
+    (synopsis
+      "Automated testing for the examples in your code and documentation.")
+    (description
+      "This library provides a way to check examples in your code and
+documentation by parsing them from their source and evaluating the
+parsed examples as part of your normal test run.  Integration is
+provided for the main Python test runners.")
+    (license license:expat)))
-- 
2.30.2

[bug#55044] [PATCH 2/8] gnu: Add python-pydispatcher.

[Felix Gruber]

* gnu/packages/python-xyz.scm (python-pydispatcher): New variable.
---
 gnu/packages/python-xyz.scm | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 9a5d311680..78310e373a 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -108,7 +108,7 @@
 ;;; Copyright © 2021 Simon Streit <simon@netpanic.org>
 ;;; Copyright © 2021 Daniel Meißner <daniel.meissner-i4k@ruhr-uni-bochum.de>
 ;;; Copyright © 2021 Pradana Aumars <paumars@courrier.dev>
-;;; Copyright © 2021 Felix Gruber <felgru@posteo.net>
+;;; Copyright © 2021, 2022 Felix Gruber <felgru@posteo.net>
 ;;; Copyright © 2021 Sébastien Lerique <sl@eauchat.org>
 ;;; Copyright © 2021 Raphaël Mélotte <raphael.melotte@mind.be>
 ;;; Copyright © 2021 ZmnSCPxj <ZmnSCPxj@protonmail.com>
@@ -29534,3 +29534,34 @@ match a given regular expression, or count possible matches efficiently. It
 uses the parsed regular expression, so you get a much more accurate result
 than trying to just split strings.")
     (license license:asl2.0)))
+
+(define-public python-pydispatcher
+  (package
+    (name "python-pydispatcher")
+    (version "2.0.5")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "PyDispatcher" version))
+        (sha256
+          (base32 "1bswbmhlbqdxlgbxlb6xrlm4k253sg8nvpl1whgsys8p3fg0cw2m"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+        (modify-phases %standard-phases
+          (replace 'check
+            (lambda* (#:key tests? #:allow-other-keys)
+              (when tests?
+                (invoke "pytest")))))))
+    (native-inputs (list python-pytest))
+    (home-page "http://pydispatcher.sourceforge.net")
+    (synopsis "Multi-producer-multi-consumer signal dispatching mechanism")
+    (description "PyDispatcher is an enhanced version of
+Patrick K. O’Brien’s original dispatcher.py module.  It provides the
+Python programmer with a robust mechanism for event routing within
+various application contexts.
+
+Included in the package are the robustapply and saferef modules, which
+provide the ability to selectively apply arguments to callable objects
+and to reference instance methods using weak-references.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 3/8] gnu: Add python-queuelib.

[Felix Gruber]

* gnu/packages/python-xyz.scm (python-queuelib): New variable.
---
 gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 78310e373a..6c1aee8695 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -29565,3 +29565,27 @@ Included in the package are the robustapply and saferef modules, which
 provide the ability to selectively apply arguments to callable objects
 and to reference instance methods using weak-references.")
     (license license:bsd-3)))
+
+(define-public python-queuelib
+  (package
+    (name "python-queuelib")
+    (version "1.6.2")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "queuelib" version))
+        (sha256
+          (base32 "1lpwq8wx3025i14y5h0hbald2ypbarf081pql6cqcak4y9kp482b"))))
+    (build-system python-build-system)
+    (native-inputs (list python-pytest))
+    (home-page "https://github.com/scrapy/queuelib")
+    (synopsis
+      "Collection of persistent (disk-based) and non-persistent (memory-based) queues")
+    (description "Queuelib is a Python library that implements object
+collections which are stored in memory or persisted to disk, provide a
+simple API, and run fast.
+
+Queuelib provides collections for queues (FIFO), stacks (LIFO), queues
+sorted by priority and queues that are emptied in a round-robin
+fashion.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 4/8] gnu: Add python-itemadapter.

[Felix Gruber]

* gnu/packages/python-xyz.scm (python-itemadapter): New variable.
---
 gnu/packages/python-xyz.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 6c1aee8695..a7f7cc91de 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -29589,3 +29589,33 @@ Queuelib provides collections for queues (FIFO), stacks (LIFO), queues
 sorted by priority and queues that are emptied in a round-robin
 fashion.")
     (license license:bsd-3)))
+
+(define-public python-itemadapter
+  (package
+    (name "python-itemadapter")
+    (version "0.5.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "itemadapter" version))
+        (sha256
+          (base32 "083wp3h2brh8x19jbdr8rz3biqwp3jlqd0rfzcyrjyhssffsgdh5"))))
+    (build-system python-build-system)
+    (home-page "https://github.com/scrapy/itemadapter")
+    (synopsis "Common interface for data container classes")
+    (description "The ItemAdapter class is a wrapper for data container
+objects, providing a common interface to handle objects of different
+types in an uniform manner, regardless of their underlying implementation.
+
+Currently supported types are:
+@itemize
+@item scrapy.item.Item
+@item dict
+@item dataclass-based classes
+@item attrs-based classes
+@item pydantic-based classes
+@end itemize
+
+Additionally, interaction with arbitrary types is supported by
+implementing a pre-defined interface.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 5/8] gnu: Add python-protego.

[Felix Gruber]

* gnu/packages/python-web.scm (python-protego): New variable.
---
 gnu/packages/python-web.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 7e333916b6..05104f8fe2 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -6466,3 +6466,30 @@ as Flask.")
      "This package provides a Python JSON-RPC 2.0 protocol and server powered
 by asyncio.")
     (license license:expat)))
+
+(define-public python-protego
+  (package
+    (name "python-protego")
+    (version "0.2.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "Protego" version))
+        (sha256
+          (base32 "1wigcjyhz8zbk562zhgfbkm733dcn65j1swzvki79dys0i1nsrnz"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+        (modify-phases %standard-phases
+          (replace 'check
+            (lambda* (#:key tests? #:allow-other-keys)
+              (when tests?
+                (invoke "pytest")))))))
+    (propagated-inputs (list python-six))
+    (native-inputs (list python-pytest))
+    (home-page "https://github.com/scrapy/protego")
+    (synopsis
+      "Pure-Python robots.txt parser with support for modern conventions")
+    (description
+      "Pure-Python robots.txt parser with support for modern conventions.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 6/8] gnu: Add python-parsel.

[Felix Gruber]

* gnu/packages/python-web.scm (python-parsel): New variable.
---
 gnu/packages/python-web.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 05104f8fe2..da3f9cf980 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -6493,3 +6493,29 @@ by asyncio.")
     (description
       "Pure-Python robots.txt parser with support for modern conventions.")
     (license license:bsd-3)))
+
+(define-public python-parsel
+  (package
+    (name "python-parsel")
+    (version "1.6.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "parsel" version))
+        (sha256
+          (base32 "0yawf9r3r863lwxj0n89i7h3n8xjbsl5b7n6xg76r68scl5yzvvh"))))
+    (build-system python-build-system)
+    (propagated-inputs
+      (list python-cssselect
+            python-lxml
+            python-six
+            python-w3lib))
+    (native-inputs
+      (list python-pytest python-pytest-runner))
+    (home-page "https://github.com/scrapy/parsel")
+    (synopsis
+      "Parsel is a library to extract data from HTML and XML using XPath and CSS selectors")
+    (description "Parsel is a library to extract and remove data from
+HTML and XML using XPath and CSS selectors, optionally combined with
+regular expressions.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 7/8] gnu: Add python-itemloaders.

[Felix Gruber]

* gnu/packages/python-xyz.scm (python-itemloaders): New variable.
---
 gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index a7f7cc91de..e6b4d51b2c 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -29619,3 +29619,27 @@ Currently supported types are:
 Additionally, interaction with arbitrary types is supported by
 implementing a pre-defined interface.")
     (license license:bsd-3)))
+
+(define-public python-itemloaders
+  (package
+    (name "python-itemloaders")
+    (version "1.0.4")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "itemloaders" version))
+        (sha256
+          (base32 "15hc78h90qhwass1bga1c3xar2dd6j8sxg61zg6jvh74lf6csxqj"))))
+    (build-system python-build-system)
+    (propagated-inputs
+      (list python-itemadapter python-jmespath python-parsel python-w3lib))
+    (home-page "https://github.com/scrapy/itemloaders")
+    (synopsis "Base library for scrapy's ItemLoader")
+    (description "Itemloaders is a library that helps you collect data
+from HTML and XML sources.  It comes in handy to extract data from web
+pages, as it supports data extraction using CSS and XPath Selectors.
+
+It’s specially useful when you need to standardize the data from many
+sources.  For example, it allows you to have all your casting and
+parsing rules in a single place.")
+    (license license:bsd-3)))
-- 
2.30.2

[bug#55044] [PATCH 8/8] gnu: Add python-scrapy.

[Felix Gruber]

* gnu/packages/python-web.scm (python-scrapy): New variable.
---
 gnu/packages/python-web.scm | 60 +++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index da3f9cf980..f4ff4f494c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -6519,3 +6519,63 @@ by asyncio.")
 HTML and XML using XPath and CSS selectors, optionally combined with
 regular expressions.")
     (license license:bsd-3)))
+
+(define-public python-scrapy
+  (package
+    (name "python-scrapy")
+    (version "2.6.1")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "Scrapy" version))
+        (sha256
+          (base32 "09rqalbwcz9ix8h0992mzjs50sssxsmmh8w9abkrqchgknjmbzan"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+        (modify-phases %standard-phases
+          (replace 'check
+            (lambda* (#:key tests? #:allow-other-keys)
+              (when tests?
+                (invoke "pytest"
+                        ;; requires network access
+                        "--ignore" "tests/test_command_check.py"
+                        "-k"
+                        (string-append
+                         ;; Failing for unknown reasons
+                         "not test_server_set_cookie_domain_suffix_public_private"
+                         " and not test_user_set_cookie_domain_suffix_public_private"
+                         " and not test_pformat")
+                        "tests")))))))
+    (propagated-inputs
+      (list python-botocore ; Optional: For S3FeedStorage class.
+            python-cryptography
+            python-cssselect
+            python-itemadapter
+            python-itemloaders
+            python-lxml
+            python-parsel
+            python-protego
+            python-pydispatcher
+            python-pyopenssl
+            python-queuelib
+            python-service-identity
+            python-setuptools
+            python-tldextract
+            python-twisted
+            python-w3lib
+            python-zope-interface))
+    (native-inputs
+      (list python-pytest
+            python-pyftpdlib
+            python-sybil
+            python-testfixtures
+            python-uvloop
+            ))
+    (home-page "https://scrapy.org")
+    (synopsis "A high-level Web Crawling and Web Scraping framework")
+    (description "Scrapy is a fast high-level web crawling and web
+scraping framework, used to crawl websites and extract structured data
+from their pages. It can be used for a wide range of purposes, from data
+mining to monitoring and automated testing.")
+    (license license:bsd-3)))
-- 
2.30.2

bug#55044: [PATCH 0/8] Add python-scrapy.

[Ludovic Courtès]

Hi,

Felix Gruber <felgru@posteo.net> skribis:

>   gnu: Add python-sybil.
>   gnu: Add python-pydispatcher.
>   gnu: Add python-queuelib.
>   gnu: Add python-itemadapter.
>   gnu: Add python-protego.
>   gnu: Add python-parsel.
>   gnu: Add python-itemloaders.
>   gnu: Add python-scrapy.

I’ve applied the whole series with a couple of tweaks to
synopses/descriptions, as reported by ‘guix lint’.

Please take a look at
<https://guix.gnu.org/manual/devel/en/html_node/Synopses-and-Descriptions.html>
if you haven’t already and maybe (maybe!) you’ll get shorter review
times as a result.  :-)

Thanks,
Ludo’.

[bug#55044] [PATCH 0/8] Add python-scrapy.

[Felix Gruber]

Hi Ludo‘,

On 5/2/22 15:14, Ludovic Courtès wrote:
> I’ve applied the whole series with a couple of tweaks to
> synopses/descriptions, as reported by ‘guix lint’.
> 
> Please take a look at
> <https://guix.gnu.org/manual/devel/en/html_node/Synopses-and-Descriptions.html>
> if you haven’t already and maybe (maybe!) you’ll get shorter review
> times as a result.  :-)

Thank you for the hint. I'll give it a read. (And I try to remember to 
run guix lint on my next patch sets.)

Kind regards,
Felix