[bug#71640] yt-dlp: Update to 2024-05-27

[bug#71640] yt-dlp: Update to 2024-05-27

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 168 bytes --]

This updates yt-dlp and performs other changes that are necessary for
the update.

All packages that depend directly on python-websockets still build with
this update.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#71640] [PATCH 1/6] gnu: yt-dlp: Don't inherit from youtube-dl.

[Leo Famulari]

yt-dlp has diverged from youtube-dl sufficiently that package inheritance
hinders package maintenance.

* gnu/packages/video.scm (yt-dlp): Don't inherit from YOUTUBE-DL.
[build-system, synopsis, license]: New fields.
[arguments]: Adjust accordingly.
[inputs]: Add FFMPEG.

Change-Id: I14c4cfb6a75ba0421c62eac778072ab3e76c72a1
---
 gnu/packages/video.scm | 95 ++++++++++++++++++++++--------------------
 1 file changed, 49 insertions(+), 46 deletions(-)

diff --git a/gnu/packages/video.scm b/gnu/packages/video.scm
index e7e61303a9..76e4a8a02f 100644
--- a/gnu/packages/video.scm
+++ b/gnu/packages/video.scm
@@ -3135,7 +3135,7 @@ (define-public youtube-dl
     (license license:public-domain)))
 
 (define-public yt-dlp
-  (package/inherit youtube-dl
+  (package
     (name "yt-dlp")
     (version "2023.10.13")
     (source
@@ -3147,51 +3147,52 @@ (define-public yt-dlp
        (file-name (git-file-name name version))
        (sha256
         (base32 "1cy8cpqwq6yfsbrnln3qqp9lsjckn20m6w7b890ha7jahyir5m1n"))))
+    (build-system python-build-system)
     (arguments
-     (substitute-keyword-arguments (package-arguments youtube-dl)
-       ((#:tests? _) (not (%current-target-system)))
-       ((#:phases phases)
-        #~(modify-phases #$phases
-            ;; See the comment for the corresponding phase in youtube-dl.
-            (replace 'default-to-the-ffmpeg-input
-              (lambda* (#:key inputs #:allow-other-keys)
-                (substitute* "yt_dlp/postprocessor/ffmpeg.py"
-                  (("location = self.get_param(.*)$")
-                   (string-append
-                     "location = '"
-                     (dirname (search-input-file inputs "bin/ffmpeg"))
-                     "'\n")))))
-            (replace 'build-generated-files
-              (lambda* (#:key inputs #:allow-other-keys)
-                (if (assoc-ref inputs "pandoc")
-                  (invoke "make"
-                          "PYTHON=python"
-                          "yt-dlp"
-                          "yt-dlp.1"
-                          "completions")
-                  (invoke "make"
-                          "PYTHON=python"
-                          "yt-dlp"
-                          "completions"))))
-            (replace 'fix-the-data-directories
-              (lambda* (#:key outputs #:allow-other-keys)
-                (let ((prefix (assoc-ref outputs "out")))
-                  (substitute* "setup.py"
-                    (("'etc/")
-                     (string-append "'" prefix "/etc/"))
-                    (("'share/")
-                     (string-append "'" prefix "/share/"))))))
-            (delete 'install-completion)
-            (replace 'check
-              (lambda* (#:key tests? #:allow-other-keys)
-                (when tests?
-                  (invoke "pytest" "-k" "not download"))))))))
-    (inputs (modify-inputs (package-inputs youtube-dl)
-              (append python-brotli
-                      python-certifi
-                      python-mutagen
-                      python-pycryptodomex
-                      python-websockets)))
+     `(#:tests? ,(not (%current-target-system))
+       #:phases
+       (modify-phases %standard-phases
+         ;; See <https://issues.guix.gnu.org/43418#5>.
+         ;; ffmpeg is big but required to request free formats from, e.g.,
+         ;; YouTube so pull it in unconditionally.  Continue respecting the
+         ;; --ffmpeg-location argument.
+         (add-after 'unpack 'default-to-the-ffmpeg-input
+           (lambda* (#:key inputs #:allow-other-keys)
+             (substitute* "yt_dlp/postprocessor/ffmpeg.py"
+               (("location = self.get_param(.*)$")
+                (string-append
+                  "location = '"
+                  (dirname (search-input-file inputs "bin/ffmpeg"))
+                  "'\n")))))
+         (add-before 'build 'build-generated-files
+           (lambda* (#:key inputs #:allow-other-keys)
+             (if (assoc-ref inputs "pandoc")
+               (invoke "make"
+                       "PYTHON=python"
+                       "yt-dlp"
+                       "yt-dlp.1"
+                       "completions")
+               (invoke "make"
+                       "PYTHON=python"
+                       "yt-dlp"
+                       "completions"))))
+         (add-before 'install 'fix-the-data-directories
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((prefix (assoc-ref outputs "out")))
+               (substitute* "setup.py"
+                 (("'etc/")
+                  (string-append "'" prefix "/etc/"))
+                 (("'share/")
+                  (string-append "'" prefix "/share/"))))))
+         (replace 'check
+           (lambda* (#:key tests? #:allow-other-keys)
+             (when tests?
+               (invoke "pytest" "-k" "not download")))))))
+    (inputs (list ffmpeg python-brotli
+                  python-certifi
+                  python-mutagen
+                  python-pycryptodomex
+                  python-websockets))
     (native-inputs
      (append
        ;; To generate the manpage.
@@ -3199,13 +3200,15 @@ (define-public yt-dlp
          (list pandoc)
          '())
        (list python-pytest zip)))
+    (synopsis "Download videos from YouTube.com and other sites")
     (description
      "yt-dlp is a small command-line program to download videos from
 YouTube.com and many more sites.  It is a fork of youtube-dl with a
 focus on adding new features while keeping up-to-date with the
 original project.")
     (properties '((release-monitoring-url . "https://pypi.org/project/yt-dlp/")))
-    (home-page "https://github.com/yt-dlp/yt-dlp")))
+    (home-page "https://github.com/yt-dlp/yt-dlp")
+    (license license:public-domain)))
 
 (define-public you-get
   (package

base-commit: b993f4735d41e690dbafb8ee2e17fce996a8cf20
-- 
2.41.0

[bug#71640] [PATCH 2/6] gnu: Add python-requests 2.31.0.

[Leo Famulari]

This is the minimum required version in recent versions of yt-dlp.

* gnu/packages/python-web.scm (python-requests-next): New variable.

Change-Id: I825cb928297cddc2fbbca1ce5acdbf6cff276bbc
---
 gnu/packages/python-web.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 1363e6ccdf..2686cb283f 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -3603,6 +3603,32 @@ (define-public python-requests
 than Python’s urllib2 library.")
     (license license:asl2.0)))
 
+(define-public python-requests-next
+  (package
+    (name "python-requests")
+    (version "2.31.0")
+    (source (origin
+             (method url-fetch)
+             (uri (pypi-uri "requests" version))
+             (sha256
+              (base32
+               "1qfidaynsrci4wymrw3srz8v1zy7xxpcna8sxpm91mwqixsmlb4l"))))
+    (build-system python-build-system)
+    (propagated-inputs
+     (list python-certifi
+           python-charset-normalizer
+           python-idna
+           python-urllib3))
+    (arguments
+     ;; FIXME: Some tests require network access.
+     '(#:tests? #f))
+    (home-page "http://python-requests.org/")
+    (synopsis "Python HTTP library")
+    (description
+     "Requests is a Python HTTP client library.  It aims to be easier to use
+than Python’s urllib2 library.")
+    (license license:asl2.0)))
+
 (define-public python-requests-kerberos
   (package
     (name "python-requests-kerberos")
-- 
2.41.0

[bug#71640] [PATCH 3/6] gnu: Add python-urllib3 1.26.17.

[Leo Famulari]

This is the minimum required version in recent versions of yt-dlp.

* gnu/packages/python-web.scm (python-urllib3-next): New variable.

Change-Id: Ib57bbe45fd1c9c9fa3872a00f161979762feacf6
---
 gnu/packages/python-web.scm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 2686cb283f..98aa46dafc 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -4024,6 +4024,41 @@ (define-public python-urllib3
 supports url redirection and retries, and also gzip and deflate decoding.")
     (license license:expat)))
 
+(define-public python-urllib3-next
+  (package
+    (name "python-urllib3")
+    (version "1.26.17")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (pypi-uri "urllib3" version))
+        (sha256
+         (base32
+          "08fzhaf77kbjj5abpl9xag6fpfxkdp1k5s7sqd3ayacdq91a5mi4"))))
+    (build-system python-build-system)
+    (arguments `(#:tests? #f))
+    (propagated-inputs
+     (append
+       ;; These 5 inputs are used to build urrlib3[secure]
+       (list python-certifi)
+       (if (member (%current-system)
+                   (package-transitive-supported-systems python-cryptography))
+         (list python-cryptography)
+         '())
+       (list python-idna)
+       (if (member (%current-system)
+                   (package-transitive-supported-systems python-pyopenssl))
+         (list python-pyopenssl)
+         '())
+       (list python-pysocks)))
+    (home-page "https://urllib3.readthedocs.io/")
+    (synopsis "HTTP library with thread-safe connection pooling")
+    (description
+     "Urllib3 supports features left out of urllib and urllib2 libraries.  It
+can reuse the same socket connection for multiple requests, it can POST files,
+supports url redirection and retries, and also gzip and deflate decoding.")
+    (license license:expat)))
+
 (define-public awscli
   (package
     ;; Note: updating awscli typically requires updating botocore as well.
-- 
2.41.0

[bug#71640] [PATCH 4/6] gnu: python-pytest-sanic: Further relax the websockets version requirements.

[Leo Famulari]

* gnu/packages/python-check.scm (python-pytest-sanic)[arguments]: Accept
PYTHON-WEBSOCKETS 12.0 in the custom 'relax-requirements' phase.

Change-Id: I279d7f98ff2f1fe3be46ffd4f06bbc3005c453b5
---
 gnu/packages/python-check.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm
index c5607df1ec..a895c7bb2c 100644
--- a/gnu/packages/python-check.scm
+++ b/gnu/packages/python-check.scm
@@ -2256,7 +2256,7 @@ (define-public python-pytest-sanic
            (lambda _
              (substitute* "setup.py"
                (("websockets.*<11.0")
-                "websockets<12.0")))))))
+                "websockets<13.0")))))))
     (propagated-inputs
      (list python-httpx python-async-generator python-pytest
            python-websockets))
-- 
2.41.0

[bug#71640] [PATCH 5/6] gnu: python-websockets: Update to 12.0

[Leo Famulari]

* gnu/packages/python-web.scm (python-websockets): Update to 12.0.

Change-Id: I6217093bbdfb81b35eb7c8d59ada3f612b7d9511
---
 gnu/packages/python-web.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 98aa46dafc..eef63351ef 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -6754,7 +6754,7 @@ (define-public python-warcio
 (define-public python-websockets
   (package
     (name "python-websockets")
-    (version "11.0.3")
+    (version "12.0")
     (source
      (origin
        (method git-fetch)
@@ -6764,7 +6764,7 @@ (define-public python-websockets
        (file-name (git-file-name name version))
        (sha256
         (base32
-         "1hn1qzpk1fvhi5j5nz4xlvzwkj9y16c9gryrb4n4dza84qi1pna5"))))
+         "1a587a1knjsy9zmgab9v2yncx0803pg2hfcvf7kz6vs8ixaggqmh"))))
     (build-system python-build-system)
     (arguments
      (list #:phases
-- 
2.41.0

[bug#71640] yt-dlp: Update to 2024-05-27

[Maxim Cournoyer]

Hi,

Leo Famulari <leo@famulari.name> writes:

> * gnu/packages/python-web.scm (python-websockets): Update to 12.0.

That's a major version bump that causes the rebuild of 414 dependents.
We'll want to see what QA thinks (breakage?).

-- 
Thanks,
Maxim

[bug#71640] yt-dlp: Update to 2024-05-27

[Leo Famulari]

On Sun, Jun 23, 2024 at 10:00:04PM -0400, Maxim Cournoyer wrote:
> Hi,
> 
> Leo Famulari <leo@famulari.name> writes:
> 
> > * gnu/packages/python-web.scm (python-websockets): Update to 12.0.
> 
> That's a major version bump that causes the rebuild of 414 dependents.
> We'll want to see what QA thinks (breakage?).

Right. According to QA, nothing is broken by this change:

https://qa.guix.gnu.org/issue/71640

Also, I checked if the 'streamlink' package still works, and it does.
That's an end-user application that uses python-websockets. Not sure if
my use of the program (streaming something from the web to VLC)
exercises websockets... but it does work.

[bug#71640] [PATCH 6/6] gnu: yt-dlp: Update to 2024.05.27.

[Leo Famulari]

* gnu/packages/video.scm (yt-dlp): Update to 2024.05.27.
[build-system]: Use PYPROJECT-BUILD-SYSTEM.
[arguments]: Remove the custom build phase 'fix-the-data-directories', because
There is no longer a setup.py file to be amended.
[inputs]: Add PYTHON-REQUESTS-NEXT and PYTHON-URLLIB3-NEXT.
[native-inputs]: Add PYTHON-HATCHLING.

Change-Id: I0c9e609817071346ca85d1f0966cf8994b2da2c5
---
 gnu/packages/video.scm | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/gnu/packages/video.scm b/gnu/packages/video.scm
index 76e4a8a02f..e3c20693fc 100644
--- a/gnu/packages/video.scm
+++ b/gnu/packages/video.scm
@@ -3137,7 +3137,7 @@ (define-public youtube-dl
 (define-public yt-dlp
   (package
     (name "yt-dlp")
-    (version "2023.10.13")
+    (version "2024.05.27")
     (source
      (origin
        (method git-fetch)
@@ -3146,8 +3146,8 @@ (define-public yt-dlp
              (commit version)))
        (file-name (git-file-name name version))
        (sha256
-        (base32 "1cy8cpqwq6yfsbrnln3qqp9lsjckn20m6w7b890ha7jahyir5m1n"))))
-    (build-system python-build-system)
+        (base32 "13j6vg0kxfw3hppq7gzbz2d72g415071gh5arkwzj902rh0c7777"))))
+    (build-system pyproject-build-system)
     (arguments
      `(#:tests? ,(not (%current-target-system))
        #:phases
@@ -3176,14 +3176,6 @@ (define-public yt-dlp
                        "PYTHON=python"
                        "yt-dlp"
                        "completions"))))
-         (add-before 'install 'fix-the-data-directories
-           (lambda* (#:key outputs #:allow-other-keys)
-             (let ((prefix (assoc-ref outputs "out")))
-               (substitute* "setup.py"
-                 (("'etc/")
-                  (string-append "'" prefix "/etc/"))
-                 (("'share/")
-                  (string-append "'" prefix "/share/"))))))
          (replace 'check
            (lambda* (#:key tests? #:allow-other-keys)
              (when tests?
@@ -3192,6 +3184,8 @@ (define-public yt-dlp
                   python-certifi
                   python-mutagen
                   python-pycryptodomex
+                  python-requests-next ; TODO Remove this special package
+                  python-urllib3-next  ; TODO Remove this one too
                   python-websockets))
     (native-inputs
      (append
@@ -3199,7 +3193,7 @@ (define-public yt-dlp
        (if (supported-package? pandoc)
          (list pandoc)
          '())
-       (list python-pytest zip)))
+       (list python-hatchling python-pytest zip)))
     (synopsis "Download videos from YouTube.com and other sites")
     (description
      "yt-dlp is a small command-line program to download videos from
-- 
2.41.0

[bug#71640] yt-dlp: Update to 2024-05-27

[Maxim Cournoyer]

Hi,

Leo Famulari <leo@famulari.name> writes:

> * gnu/packages/video.scm (yt-dlp): Update to 2024.05.27.
> [build-system]: Use PYPROJECT-BUILD-SYSTEM.
> [arguments]: Remove the custom build phase 'fix-the-data-directories', because
> There is no longer a setup.py file to be amended.
> [inputs]: Add PYTHON-REQUESTS-NEXT and PYTHON-URLLIB3-NEXT.
> [native-inputs]: Add PYTHON-HATCHLING.
>
> Change-Id: I0c9e609817071346ca85d1f0966cf8994b2da2c5
> ---
>  gnu/packages/video.scm | 18 ++++++------------
>  1 file changed, 6 insertions(+), 12 deletions(-)
>
> diff --git a/gnu/packages/video.scm b/gnu/packages/video.scm
> index 76e4a8a02f..e3c20693fc 100644
> --- a/gnu/packages/video.scm
> +++ b/gnu/packages/video.scm
> @@ -3137,7 +3137,7 @@ (define-public youtube-dl
>  (define-public yt-dlp
>    (package
>      (name "yt-dlp")
> -    (version "2023.10.13")
> +    (version "2024.05.27")
>      (source
>       (origin
>         (method git-fetch)
> @@ -3146,8 +3146,8 @@ (define-public yt-dlp
>               (commit version)))
>         (file-name (git-file-name name version))
>         (sha256
> -        (base32 "1cy8cpqwq6yfsbrnln3qqp9lsjckn20m6w7b890ha7jahyir5m1n"))))
> -    (build-system python-build-system)
> +        (base32 "13j6vg0kxfw3hppq7gzbz2d72g415071gh5arkwzj902rh0c7777"))))
> +    (build-system pyproject-build-system)
>      (arguments
>       `(#:tests? ,(not (%current-target-system))
>         #:phases
> @@ -3176,14 +3176,6 @@ (define-public yt-dlp
>                         "PYTHON=python"
>                         "yt-dlp"
>                         "completions"))))
> -         (add-before 'install 'fix-the-data-directories
> -           (lambda* (#:key outputs #:allow-other-keys)
> -             (let ((prefix (assoc-ref outputs "out")))
> -               (substitute* "setup.py"
> -                 (("'etc/")
> -                  (string-append "'" prefix "/etc/"))
> -                 (("'share/")
> -                  (string-append "'" prefix "/share/"))))))
>           (replace 'check
>             (lambda* (#:key tests? #:allow-other-keys)
>               (when tests?
> @@ -3192,6 +3184,8 @@ (define-public yt-dlp
>                    python-certifi
>                    python-mutagen
>                    python-pycryptodomex
> +                  python-requests-next ; TODO Remove this special package
> +                  python-urllib3-next  ; TODO Remove this one too

That's OK only because that's an application and these inputs are not
propagated.  I'm a bit worried the '-next' variants may be used by
packagers elsewhere, but I guess this means we should give the
python-team branch a spin.

Also, I see you haven't used inheritance anywhere; I guess that's to
make the transition to the latest package as painful as possible
(simply deleting the old definitions).

-- 
Thanks,
Maxim

[bug#71640] yt-dlp: Update to 2024-05-27

[Leo Famulari]

On Sun, Jun 23, 2024 at 09:58:13PM -0400, Maxim Cournoyer wrote:
> > +                  python-requests-next ; TODO Remove this special package
> > +                  python-urllib3-next  ; TODO Remove this one too
> 
> That's OK only because that's an application and these inputs are not
> propagated.  I'm a bit worried the '-next' variants may be used by
> packagers elsewhere, but I guess this means we should give the
> python-team branch a spin.

I could make the packages hidden. Is there a better approach?

> Also, I see you haven't used inheritance anywhere; I guess that's to
> make the transition to the latest package as painful as possible
> (simply deleting the old definitions).

For me, using inheritance is painful, and this is relatively painless.

[bug#71640] yt-dlp: Update to 2024-05-27

[Maxim Cournoyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> On Sun, Jun 23, 2024 at 09:58:13PM -0400, Maxim Cournoyer wrote:
>> > +                  python-requests-next ; TODO Remove this special package
>> > +                  python-urllib3-next  ; TODO Remove this one too
>> 
>> That's OK only because that's an application and these inputs are not
>> propagated.  I'm a bit worried the '-next' variants may be used by
>> packagers elsewhere, but I guess this means we should give the
>> python-team branch a spin.
>
> I could make the packages hidden. Is there a better approach?

Not really, as the problem with Python is that everything gets
propagated and you can't mix incompatible libraries together. I guess
that's something to keep in mind when reviewing Python packages.  It's
OK to use any version for *applications*, but it's not OK for libraries,
as these are expected to be compatible with the larger Python package
collection of Guix.

>> Also, I see you haven't used inheritance anywhere; I guess that's to
>> make the transition to the latest package as painful as possible
>> (simply deleting the old definitions).
>
> For me, using inheritance is painful, and this is relatively painless.

I meant painless above.  I disagree that inheritance is generally
painful, although it brings some complications (such as 'guix refresh'
not taking inheritance dependencies into account).

Anyway, LGTM :-).

-- 
Thanks,
Maxim

bug#71640: yt-dlp: Update to 2024-05-27

[Leo Famulari]

On Fri, Jun 28, 2024 at 08:27:00PM -0400, Maxim Cournoyer wrote:
> Anyway, LGTM :-).

Pushed as 9014b27480a1fafc643feb97db9c7efff74294ba

Let me know if you see the need for any follow-ups, and thanks for
reviewing!

[bug#71640] yt-dlp: Update to 2024-05-27

[Maxim Cournoyer]

Hi,

Leo Famulari <leo@famulari.name> writes:

> yt-dlp has diverged from youtube-dl sufficiently that package inheritance
> hinders package maintenance.
>
> * gnu/packages/video.scm (yt-dlp): Don't inherit from YOUTUBE-DL.
> [build-system, synopsis, license]: New fields.
> [arguments]: Adjust accordingly.
> [inputs]: Add FFMPEG.
>
> Change-Id: I14c4cfb6a75ba0421c62eac778072ab3e76c72a1

Reviewed-by: Maxim Cournoyer <maxim.cournoyer@gmail>

But we'll want to have some QA feedback before merging (or lots of
manual building).

-- 
Thanks,
Maxim

[bug#71640] Deprecation

[Andreas Enge]

Do you think we could/should remove the youtube-dl package and declare
yt-dlp as its successor (as a follow-up to this patch)?

Andreas

[bug#71640] Deprecation

[Leo Famulari]

On Mon, Jul 01, 2024 at 02:19:17PM +0200, Andreas Enge wrote:
> Do you think we could/should remove the youtube-dl package and declare
> yt-dlp as its successor (as a follow-up to this patch)?

Yes, that's a good idea. I'll do it.

[bug#71640] Deprecation

[Leo Famulari]

On Wed, Jul 03, 2024 at 08:36:47PM -0400, Leo Famulari wrote:
> On Mon, Jul 01, 2024 at 02:19:17PM +0200, Andreas Enge wrote:
> > Do you think we could/should remove the youtube-dl package and declare
> > yt-dlp as its successor (as a follow-up to this patch)?
> 
> Yes, that's a good idea. I'll do it.

Actually, I checked the youtube-dl Git repo, and it does see regular
activity. But they are no longer making releases.

The overall status of the youtube-dl project is not clear to me.

For that reason I'm going to leave it unchanged for now.