[NonGNU ELPA] Package suggestion: yeetube

[NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]


Hello,

I've been trying to learn elisp and made this simple niche 
package, to watch & download youtube content via Emacs using only 
free software.

Homepage: https://git.thanosapollo.com/yeetube/about/


Package about section:

This package provides the ability to scrape YouTube, with the 
results displayed in a proced-like buffer.

Package Functionality includes:

    Search Youtube for query
    Play video url, by default using mpv
    Bookmark/Save video url
    Download video using yt-dlp
    A minimal yt-dlp front-end, which is independent of the rest 
    YouTube functionality, to download multiple urls.


I'm submitting this package to learn more about the GNU Emacs 
ecosystem & gain any valuable input that you may provide through 
this process. 

Thank you,



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-elpa-packages-yeetube-New-package.patch --]
[-- Type: text/x-patch, Size: 832 bytes --]

From b94a05d965c96d96239fdb2ffb07482be565e93c Mon Sep 17 00:00:00 2001
From: Thanos Apollo <public@thanosapollo.com>
Date: Wed, 4 Oct 2023 10:21:18 +0300
Subject: [PATCH] elpa-packages (yeetube): New package

---
 elpa-packages | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/elpa-packages b/elpa-packages
index d49aebe08e..19fd2a3540 100644
--- a/elpa-packages
+++ b/elpa-packages
@@ -827,6 +827,10 @@
  (yasnippet-snippets	:url "https://github.com/AndreaCrotti/yasnippet-snippets"
   :ignored-files ("LICENSE" "report" "snippets.html"))
 
+ (yeetube    :url "https://git.thanosapollo.com/yeetube"
+             :ignored-files ("LICENSE")
+             :readme "README.md")
+
  (zenburn-theme		:url "https://github.com/bbatsov/zenburn-emacs"
   :news "CHANGELOG.md"
   :ignored-files ("LICENSE" "screenshots"))
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 45 bytes --]



-- 
Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Stefan Kangas]

Thanos Apollo <public@thanosapollo.com> writes:

> This package provides the ability to scrape YouTube, with the
> results displayed in a proced-like buffer.
>
> Package Functionality includes:
>
>     Search Youtube for query
>     Play video url, by default using mpv
>     Bookmark/Save video url
>     Download video using yt-dlp
>     A minimal yt-dlp front-end, which is independent of the rest
>     YouTube functionality, to download multiple urls.

I'm not sure how acceptable something like this is with regards to our
policies, so I'm copying in Richard.

From a very quick glance at the code:

- Is there anything more needed to add support for Individuous
  instances?  See: https://invidious.io/

- Any chance you could add an option like `vc-tor'?

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Stefan Kangas <stefankangas@gmail.com> writes:


[...]


>
> - Is there anything more needed to add support for Individuous
>   instances?  See: https://invidious.io/
>

I had an option to use a user specified instance until this commit 
https://git.thanosapollo.com/yeetube/commit/?id=5e3c2928b2c626cccede4284a3dc022f79f333e2 
~7 days ago

I remove it for the following reasons:
- Scraping a website that displays scraped content from youtube 
  seems pointless, since it's quite simple to parse youtube's json 
  & html (without using non-free js)
- Invidious has faced legal pressure recently 
  https://github.com/iv-org/invidious/issues/3872 and I don't feel 
  it's needed to bring this to an emacs package
- Depending on an external invidious instance is not ideal, nor 
  the projects goal.
- Maintaining both youtube & invidious scraping functionality is 
  not ideal either.

> - Any chance you could add an option like `vc-tor'?

Sure, I didn't know about it, if anyone can send me some tips on 
how to implement it or a patch I could work on it


-- 
Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Stefan Kangas]

Thanos Apollo <public@thanosapollo.com> writes:

>> - Is there anything more needed to add support for Individuous
>>   instances?  See: https://invidious.io/
>
> I had an option to use a user specified instance until this commit
> https://git.thanosapollo.com/yeetube/commit/?id=5e3c2928b2c626cccede4284a3dc022f79f333e2
> ~7 days ago
>
> I remove it for the following reasons:
> - Scraping a website that displays scraped content from youtube
>   seems pointless, since it's quite simple to parse youtube's json
>   & html (without using non-free js)

The point is that it wouldn't tell the Google servers about your
activities.

> - Invidious has faced legal pressure recently
>   https://github.com/iv-org/invidious/issues/3872 and I don't feel
>   it's needed to bring this to an emacs package

IANAL, but I think the risk should be minimal as we wouldn't be the ones
talking directly to YouTube's servers.

> - Depending on an external invidious instance is not ideal, nor
>   the projects goal.
> - Maintaining both youtube & invidious scraping functionality is
>   not ideal either.

Fair enough.

>> - Any chance you could add an option like `vc-tor'?
>
> Sure, I didn't know about it, if anyone can send me some tips on
> how to implement it or a patch I could work on it

Basically, you just need to add "torsocks" to the command "yt-dl ...",
so it becomes "torsocks yt-dl ...".

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Stefan Kangas <stefankangas@gmail.com> writes:


[...]
>>> - Any chance you could add an option like `vc-tor'?
>>
>> Sure, I didn't know about it, if anyone can send me some tips 
>> on
>> how to implement it or a patch I could work on it
>
> Basically, you just need to add "torsocks" to the command "yt-dl 
> ...",
> so it becomes "torsocks yt-dl ...".

This sounds nice, I will try it

-- 
Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  
  > I'm not sure how acceptable something like this is with regards to our
  > policies, so I'm copying in Richard.

Thanks for bringing me in.

  > >     Search Youtube for query

It must not use the actual youtube site, since that requires users to
run nonfree JS software.  However, it is ok to access those videos
using the invidio.us proxies.

  > >     Play video url, by default using mpv

Is there a specific reason to prefer mpv rather than vlc?
I don't know which is better, but if we are going to make Emacs
prefer one, let's find which one is better and why.

  > >     Download video using yt-dlp

Does this work on invidio.us?

  > >     A minimal yt-dlp front-end, which is independent of the rest

Can someone explain this concreely?

  > >     YouTube functionality, to download multiple urls.

Can someone explain this concreely?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Richard Stallman <rms@gnu.org> writes:

[...]
>   > >     Search Youtube for query
>
> It must not use the actual youtube site, since that requires 
> users to
> run nonfree JS software.  However, it is ok to access those 
> videos
> using the invidio.us proxies.
>

I'm not a web developer (or any kind of developer), but I don't 
think it uses any javascipt, apologies for my ignorance if that's 
not the case. I wanted this to be an 'Invidious' but as a text 
buffer for Emacs, without relying on the invidious project.


>   > >     Play video url, by default using mpv
>
> Is there a specific reason to prefer mpv rather than vlc?
> I don't know which is better, but if we are going to make Emacs
> prefer one, let's find which one is better and why.
>

It has a custom option to use any other media player. I'm just 
familiar with mpv.

For example you can use GNU/Emms like so:

```
(setf yeetube-player #'emms-play-url)
```

which you can setup to use vlc.


>   > >     Download video using yt-dlp
>
> Does this work on invidio.us?
>

Yes

>   > >     A minimal yt-dlp front-end, which is independent of 
>   > >     the rest
>
> Can someone explain this concreely?
>
>   > >     YouTube functionality, to download multiple urls.
>
> Can someone explain this concreely?

yt-dlp is not specific to youtube. You can use it download videos 
from news sites etc since it accepts any m3u/mpd url


If you are familiar with ffmpeg, it works similarly to this 
command (function wise):


```
ffmpeg -protocol_whitelist file,crypto,data,https,tls,tcp -stats 
-i <URL.m3u> -codec copy <FILE.mp4>
```

e.g `yeetube-download-videos` just prompts the user to input any 
URL they like and downloads them as NAME. It also includes custom 
options to export content as audio only format etc.

-- 

Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > >     Search Youtube for query
>
> It must not use the actual youtube site, since that requires users to
> run nonfree JS software.  However, it is ok to access those videos
> using the invidio.us proxies.

AFAIU, no JS is involved here. Only `url-retrieve-synchronously' +
parsing the retrieved URL.

>   > >     Play video url, by default using mpv
>
> Is there a specific reason to prefer mpv rather than vlc?
> I don't know which is better, but if we are going to make Emacs
> prefer one, let's find which one is better and why.

Both VLC and MPV support watching youtube videos via URL. (And not only
youtube). AFAIU, the difference is that VLC implements
reverse-engineered youtube API directly inside VLC, while MPV relies on
external tool - yt-dlp.

The advantage of VLC is that it never executes javascript, while yt-dlp
may do it for some websites, including youtube (see
https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/youtube.py#L3101)

>   > >     Download video using yt-dlp
>
> Does this work on invidio.us?

yt-dlp works on a large number of websites, including invidio instances.
I am not sure about VLC (I do not use VLC now).

>   > >     A minimal yt-dlp front-end, which is independent of the rest
>
> Can someone explain this concreely?

yt-dlp is a program (Unlicense license, compatible with GNU GPL) that is
able to _download_ videos from various websites, including websites that
actively try to prevent users from downloading videos (for example,
youtube and twitch).

From ethical perspective, yt-dlp (and thus mpv) has a disadvatage that
it may sometimes execute downloaded javascript to work around websites
that try to force users use javascript in order to watch videos.
However, the amount of executed javascript is much smaller compared to
using a proper browser - javascript is executed when website uses a
complex javascript algorithms, that cannot be reverse-engineered, to
generate access tokens to retrieve video stream.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Using tor & url

[Thanos Apollo]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]


Quite a late reply to a dead thread, but for anyone reading this in the
future and is interested in implementing tor support, it might be
useful.

Stefan Kangas <stefankangas@gmail.com> writes:

[...]
>
> From a very quick glance at the code:
>
> - Is there anything more needed to add support for Individuous
>   instances?  See: https://invidious.io/
>

(FYI this has been added)

> - Any chance you could add an option like `vc-tor'?

Is it recommended to use 'socks for this?

e.g using a macro like this one:

====
(defmacro yeetube-with-tor-socks (&rest body)
  `(let ((url-gateway-method 'socks)
         (socks-noproxy '("localhost"))
         (socks-server '("Default server" "127.0.0.1" 9050 5)))
     ,@body))
====

This was the easiest way I could find to route traffic using
#'url-retrieve-synchronously to tor.

A built in option like this for url to route all traffic through tor
might be possible?

Additionally, I found the performance of url-retrieve-synchronously to
improve when I set 'url-request-extra-headers' to something like this:

====

'(("Accept-Language" . "Accept-Language: en-US,en;q=0.9")
    ("Accept" . "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8")
    ("User-Agent" . "Mozilla/5.0 (Windows NT 10.0; rv:122.0) Gecko/20100101 Firefox/122.0"))

====

I'm using the user-agent from arkenfox user.js
  <https://github.com/arkenfox/user.js/>

I had url-privacy-level set to 'high, but despite that my user-agent was
"URL/Emacs Emacs/29.2 (X11; x86_64-pc-linux-gnu)". Changing this
improved the performance of #'url-retrieve-synchronously (maybe because
it's not a common user-agent and it required further verification?) &
makes websites that utilize user-agent blocking accessible.

Also, when it comes to 'url-mime-language-string', shouldn't that either
be set to "en" by default (instead of nil) or get a locale value using
something like getenv? I was located in Greece with locale set to en-US
but I was getting Spanish by default on most websites (still trying to
figure out why).

-- 
Information wants to be free.

Thanos Apollo
https://thanosapollo.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: [NonGNU ELPA] Package suggestion: yeetube

[Daniel Martín]

Thanos Apollo <public@thanosapollo.com> writes:

> Hello,
>
> I've been trying to learn elisp and made this simple niche package, to
> watch & download youtube content via Emacs using only free software.
>
> Homepage: https://git.thanosapollo.com/yeetube/about/
>

Thanks for writing this package, it looks interesting.  I see it
requires Emacs 29.1, but is there a reason it requires such a recent
version of Emacs?  I'd set the minimum version to something older, if
possible, so that users on older versions of Emacs can still use recent
packages like yours.

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Daniel Martín <mardani29@yahoo.es> writes:

> Thanos Apollo <public@thanosapollo.com> writes:
>
>> Hello,
>>
>> I've been trying to learn elisp and made this simple niche 
>> package, to
>> watch & download youtube content via Emacs using only free 
>> software.
>>
>> Homepage: https://git.thanosapollo.com/yeetube/about/
>>
>
> Thanks for writing this package, it looks interesting.

Thanks!

> I see it requires Emacs 29.1, but is there a reason it requires 
> such a recent
> version of Emacs?  I'd set the minimum version to something 
> older, if
> possible, so that users on older versions of Emacs can still use 
> recent
> packages like yours.

I only used `defvar-keymap` which requires 29.1. I will redo the 
keymap the 'old way' and set the minimum version to 27.2 (for 
cl-lib)


-- 
Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Philip Kaludercic]

Thanos Apollo <public@thanosapollo.com> writes:

> Daniel Martín <mardani29@yahoo.es> writes:
>
>> Thanos Apollo <public@thanosapollo.com> writes:
>>
>>> Hello,
>>>
>>> I've been trying to learn elisp and made this simple niche package,
>>> to
>>> watch & download youtube content via Emacs using only free
>>> software.
>>>
>>> Homepage: https://git.thanosapollo.com/yeetube/about/
>>>
>>
>> Thanks for writing this package, it looks interesting.
>
> Thanks!
>
>> I see it requires Emacs 29.1, but is there a reason it requires such
>> a recent
>> version of Emacs?  I'd set the minimum version to something older,
>> if
>> possible, so that users on older versions of Emacs can still use
>> recent
>> packages like yours.
>
> I only used `defvar-keymap` which requires 29.1. I will redo the
> keymap the 'old way' and set the minimum version to 27.2 (for cl-lib)

For defvar-keymap, you can also depend on the Compat package, that
defines a compatibility version for older Emacs releases.

RE: [External] : Re: [NonGNU ELPA] Package suggestion: yeetube

[Drew Adams]

> the Compat package, that defines a
> compatibility version for older Emacs releases.

How old?  Emacs 20?  What minimal version
does that package itself require?  Does
it, itself, need some compat support...?

Re: [External] : Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Drew Adams <drew.adams@oracle.com> writes:

> How old?  Emacs 20?  What minimal version
> does that package itself require?  Does
> it, itself, need some compat support...?

Philip will give you a more concrete answer, but according to the 
documentation:

```
Requires: ((emacs "24.4") (compat "29.1.4.2"))

   There is no need to depend on ‘emacs 24.4’ specifically.  One 
   can
choose any newer version, if features not provided by Compat 
necessitate
it, for example bug fixes or UI improvements.

   In any file where compatibility forms are used, a

     (require 'compat)

   should be added early on.  In packages which are part of Emacs 
   itself
and which want to take advantage of Compat, the ‘noerror’ flag 
should be
specified: ‘(require 'compat nil 'noerror)’.  In the future a 
minimal
version of Compat may be added to the Emacs core, such that the
‘noerror’ flag will not be necessary anymore.
```

Not sure, but maybe something like this is a valid use case?

``` emacs-lisp
(compat-call defvar-keymap yeetube-mode-map
	     :doc "Keymap for yeetube commands"
	     "RET" #'yeetube-play
             ...
```             


-- 
Thanos Apollo
https://thanosapollo.com

Re: [External] : Re: [NonGNU ELPA] Package suggestion: yeetube

[Philip Kaludercic]

Thanos Apollo <public@thanosapollo.com> writes:

> Drew Adams <drew.adams@oracle.com> writes:
>
>> How old?  Emacs 20?  What minimal version
>> does that package itself require?  Does
>> it, itself, need some compat support...?

Emacs 24.4 is the current minimum version.  This is currently the oldest
version that one can reasonably support without running into too many
edge-cases that are beyond the scope of the library, while also being
distributed in most supported, popular distributions.

> Philip will give you a more concrete answer, but according to the
> documentation:
>
> ```
> Requires: ((emacs "24.4") (compat "29.1.4.2"))
>
>   There is no need to depend on ‘emacs 24.4’ specifically.  One    can
> choose any newer version, if features not provided by Compat
> necessitate
> it, for example bug fixes or UI improvements.
>
>   In any file where compatibility forms are used, a
>
>     (require 'compat)
>
>   should be added early on.  In packages which are part of Emacs
>   itself
> and which want to take advantage of Compat, the ‘noerror’ flag should
> be
> specified: ‘(require 'compat nil 'noerror)’.  In the future a minimal
> version of Compat may be added to the Emacs core, such that the
> ‘noerror’ flag will not be necessary anymore.
> ```
>
> Not sure, but maybe something like this is a valid use case?
>
> ``` emacs-lisp
> (compat-call defvar-keymap yeetube-mode-map
> 	     :doc "Keymap for yeetube commands"
> 	     "RET" #'yeetube-play
>             ...

There is no need to use compat-call, if the function/macro was
introduced in a more recent version than the minimum version of Emacs
you wish to support.  In other words, the point of `compat-call' is to
ensure the compatibility version of function is used, where the
signature was updated at some point (e.g. IIRC in Emacs 26.1 `assoc'
gained an optional TESTFN argument), but to avoid the complexity of
advising the default function with an updated signature, the API of
Compat allows/requires for developers to opt-in to new signatures.

> ```             -- Thanos Apollo
> https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Philip Kaludercic <philipk@posteo.net> writes:


[...]
>
> For defvar-keymap, you can also depend on the Compat package, 
> that
> defines a compatibility version for older Emacs releases.

Oh! Didn't know about compat, looks really useful. I already 
pushed a new commit fixing the issue, but I'll be playing around 
with it on the weekend, thanks! 


-- 
Thanos Apollo
https://thanosapollo.com

Re: [NonGNU ELPA] Package suggestion: yeetube

[Emanuel Berg]

Thanos Apollo wrote:

> I've been trying to learn elisp and made this simple niche
> package, to watch & download youtube content via Emacs using
> only free software.
>
> Homepage: https://git.thanosapollo.com/yeetube/about/

Nice!

I did

  (checkdoc-current-buffer t)

and also byte compiled with this command

   $ emacs -Q \
           --batch \
           --eval "(setq load-path (append load-path (list \".\")))" \
           -f batch-byte-compile *.el

to discover the following stuff you can improve:

*** yeetube-buffer.el: checkdoc-current-buffer
yeetube-buffer.el:185: Arguments occur in the doc string out
of order

In yeetube-buffer-create:
yeetube-buffer.el:209:8: Warning: ‘backward-delete-char’ is
for interactive use only; use ‘delete-char’ instead.

In yeetube-download-videos:
yeetube.el:250:2: Warning: docstring has wrong usage of
unescaped single quotes (use \=' or different quoting such as
`...')

In end of data:
yeetube.el:275:9: Warning: the function
‘yeetube-change-download-audio-format’ is not known to
be defined.

yeetube.el:274:9: Warning: the function
‘yeetube-change-download-directory’ is not known to
be defined.

PS. If you are into automated downloading, check out these files

  https://dataswamp.org/~incal/conf/.zsh/dl
  https://dataswamp.org/~incal/conf/.zsh/yt-dlp
  https://dataswamp.org/~incal/#mpv

It doesn't have an interface tho like yours, it just
downloads bulk and then you remove stuff you don't like
from mpv.

GLHF!

-- 
underground experts united
https://dataswamp.org/~incal

Re: [NonGNU ELPA] Package suggestion: yeetube

[Thanos Apollo]

Emanuel Berg <incal@dataswamp.org> writes:

> Thanos Apollo wrote:
>
>> I've been trying to learn elisp and made this simple niche
>> package, to watch & download youtube content via Emacs using
>> only free software.
>>
>> Homepage: https://git.thanosapollo.com/yeetube/about/
>
> Nice!
>
> I did
>
>   (checkdoc-current-buffer t)
>
> and also byte compiled with this command
>
>    $ emacs -Q \
>            --batch \
>            --eval "(setq load-path (append load-path (list 
>            \".\")))" \
>            -f batch-byte-compile *.el
>
> to discover the following stuff you can improve:
>

[...]



Fixed! thanks for pointing these out!

>
> PS. If you are into automated downloading, check out these files
>
>   https://dataswamp.org/~incal/conf/.zsh/dl
>   https://dataswamp.org/~incal/conf/.zsh/yt-dlp
>   https://dataswamp.org/~incal/#mpv
>
> It doesn't have an interface tho like yours, it just
> downloads bulk and then you remove stuff you don't like
> from mpv.
>
> GLHF!

Nice! I'll be using them


-- 

Thanos Apollo
https://thanosapollo.com