Unicode characters display issue

Unicode characters display issue

[Thanos Apollo]

I'm a medical student with little to no programming
experience/knowledge, apologies if this email is improperly formatted.

I just have an issue with a package I'm working on. It's used to scrape
video search engines like youtube, watch & download videos etc from
Emacs.

https://git.sr.ht/~thanosapollo/yeetube.el


My issue is when using 'url-retrieve-synchronously', creates a buffer
that does not automatically "transform" unicode characters. If you
copy-paste them into a browser everything is fine. Is this supposed to
be the default behavior? Am I doing something wrong?

Is the only way to fix this issue to use something like
'toggle-enable-multibyte-characters'?


-- 
Thanos
https://thanosapollo.com

Re: Unicode characters display issue

[Eli Zaretskii]

> From: Thanos Apollo <public@thanosapollo.com>
> Date: Sun, 02 Jul 2023 13:22:33 +0300
> 
> 
> I'm a medical student with little to no programming
> experience/knowledge, apologies if this email is improperly formatted.
> 
> I just have an issue with a package I'm working on. It's used to scrape
> video search engines like youtube, watch & download videos etc from
> Emacs.
> 
> https://git.sr.ht/~thanosapollo/yeetube.el
> 
> 
> My issue is when using 'url-retrieve-synchronously', creates a buffer
> that does not automatically "transform" unicode characters. If you
> copy-paste them into a browser everything is fine. Is this supposed to
> be the default behavior? Am I doing something wrong?
> 
> Is the only way to fix this issue to use something like
> 'toggle-enable-multibyte-characters'?

It's hard to answer your questions without knowing what you did,
exactly.  Can you come up with a simplified Lisp program that behaves
like you describe, so that we could study it and see what you are
doing wrong?

Re: Unicode characters display issue

[Thanos Apollo]

Eli Zaretskii <eliz@gnu.org> writes:

> It's hard to answer your questions without knowing what you did,
> exactly.  Can you come up with a simplified Lisp program that behaves
> like you describe, so that we could study it and see what you are
> doing wrong?

Thanks for the reply!

``` 
(switch-to-buffer (url-retrieve-synchronously
"https://www.youtube.com/results?search_query=greece+survivor" t t))
```

After evaluating the above code and searching forward for "videoid"
you will be presented with unicode character sequences.


-- 
Thanos
https://thanosapollo.com

Re: Unicode characters display issue

[Eli Zaretskii]

> From: Thanos Apollo <public@thanosapollo.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Jul 2023 14:53:59 +0300
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It's hard to answer your questions without knowing what you did,
> > exactly.  Can you come up with a simplified Lisp program that behaves
> > like you describe, so that we could study it and see what you are
> > doing wrong?
> 
> Thanks for the reply!
> 
> ``` 
> (switch-to-buffer (url-retrieve-synchronously
> "https://www.youtube.com/results?search_query=greece+survivor" t t))
> ```
> 
> After evaluating the above code and searching forward for "videoid"
> you will be presented with unicode character sequences.

This is a unibyte buffer: "M-: enable-multibyte-characters RET"
reports nil.  So what you see are raw bytes of the encoded text.  You
need to decode it, e.g., by using decode-coding-region.  The header
says "Content-Type: text/html; charset=utf-8", so the coding-system to
use in decoding is utf-8.  For example:

  (decode-coding-region (point-min) (point-max) 'utf-8
                        (get-buffer-create "decoded"))