bug#54921: eww/url mishandles IDN over proxy

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

I have configured Privoxy as the http/https proxy.  When visiting a URL
with an IDN address url-retrieve uses the original domain name for the
proxy connection, not the IDNA encoded name, causing Privoxy to return a
404 page.  I'm not sure if that is a bug in Privoxy or in eww.  I could
not find any RFC on that topic.

(eww "https://www.pimmelgate-süd.de/") returns the Privoxy 404 page,
whereas (eww "https://xn--pimmelgate-sd-7ob.de/") returns the actual web
page.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> I have configured Privoxy as the http/https proxy.  When visiting a URL
> with an IDN address url-retrieve uses the original domain name for the
> proxy connection, not the IDNA encoded name, causing Privoxy to return a
> 404 page.  I'm not sure if that is a bug in Privoxy or in eww.  I could
> not find any RFC on that topic.

What does your proxy setup in Emacs look like?  Is this via url-proxy
and socks.el?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 14 2022, Lars Ingebrigtsen wrote:

> What does your proxy setup in Emacs look like?

I have set https_proxy=http://localhost:8118/ in the environment, which
is picked up by url to set url-proxy-services.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> I have set https_proxy=http://localhost:8118/ in the environment, which
> is picked up by url to set url-proxy-services.

Hm, right...

I'm not at all familiar with this code, so here's a stab in the dark.
Does this fix the problem?

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..22e003d22d 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1405,10 +1405,12 @@ url-https-proxy-connect
               (and proxy-auth
                    (concat "Proxy-Authorization: " proxy-auth "\r\n")))
             "\r\n")
-    (url-host url-current-object)
+    (url-http--encode-string
+     (puny-encode-domain (url-host url-current-object)))
     (or (url-port url-current-object)
         url-https-default-port)
-    (url-host url-current-object))))
+    (url-http--encode-string
+     (puny-encode-domain (url-host url-current-object))))))
 
 (defun url-https-proxy-after-change-function (_st _nd _length)
   (let* ((process-buffer (current-buffer))


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 14 2022, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> I have set https_proxy=http://localhost:8118/ in the environment, which
>> is picked up by url to set url-proxy-services.
>
> Hm, right...
>
> I'm not at all familiar with this code, so here's a stab in the dark.
> Does this fix the problem?
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index daeba17031..22e003d22d 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -1405,10 +1405,12 @@ url-https-proxy-connect
>                (and proxy-auth
>                     (concat "Proxy-Authorization: " proxy-auth "\r\n")))
>              "\r\n")
> -    (url-host url-current-object)
> +    (url-http--encode-string
> +     (puny-encode-domain (url-host url-current-object)))
>      (or (url-port url-current-object)
>          url-https-default-port)
> -    (url-host url-current-object))))
> +    (url-http--encode-string
> +     (puny-encode-domain (url-host url-current-object))))))

url-http--encode-string shouldn't be needed, since puny encoding is
always ASCII.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> url-http--encode-string shouldn't be needed, since puny encoding is
> always ASCII.

Just cargo-culting the other calls to punycode in URL.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 14 2022, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> url-http--encode-string shouldn't be needed, since puny encoding is
>> always ASCII.
>
> Just cargo-culting the other calls to punycode in URL.

Unlike url-http-create-request, url-https-proxy-connect doesn't have
that weird multibyte test, and can handle any ASCII string.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Unlike url-http-create-request, url-https-proxy-connect doesn't have
> that weird multibyte test, and can handle any ASCII string.

OK.  But did the patch fix the problem you're seeing?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

error in process filter: url-https-proxy-after-change-function: Error: (error Non-ASCII hostname www.pimmelgate-süd.de detected, please use puny-encode-domain)

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> error in process filter: url-https-proxy-after-change-function: Error:
> (error Non-ASCII hostname www.pimmelgate-süd.de detected, please use
> puny-encode-domain)

Try:

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..06e7dbf756 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1405,10 +1405,12 @@ url-https-proxy-connect
               (and proxy-auth
                    (concat "Proxy-Authorization: " proxy-auth "\r\n")))
             "\r\n")
-    (url-host url-current-object)
+    (url-http--encode-string
+     (puny-encode-domain (url-host url-current-object)))
     (or (url-port url-current-object)
         url-https-default-port)
-    (url-host url-current-object))))
+    (url-http--encode-string
+     (puny-encode-domain (url-host url-current-object))))))
 
 (defun url-https-proxy-after-change-function (_st _nd _length)
   (let* ((process-buffer (current-buffer))
@@ -1430,12 +1432,12 @@ url-https-proxy-after-change-function
             (condition-case e
                 (let ((tls-connection (gnutls-negotiate
                                        :process proc
-                                       :hostname (url-host url-current-object)
+                                       :hostname (puny-encode-domain (url-host url-current-object))
                                        :verify-error nil)))
                   ;; check certificate validity
                   (setq tls-connection
                         (nsm-verify-connection tls-connection
-                                               (url-host url-current-object)
+                                               (puny-encode-domain (url-host url-current-object))
                                                (url-port url-current-object)))
                   (with-current-buffer process-buffer (erase-buffer))
                   (set-process-buffer tls-connection process-buffer)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

That works, but still fails with http://www.pimmelgate-süd.de/.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> That works, but still fails with http://www.pimmelgate-süd.de/.

OK, I've pushed the https proxy fix to master, then.

Where does the plain-HTTP proxy stuff live, then?  Hm...  It could be
here, I guess.  Can you try the following patch?

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 96a4742956..678ee05ae5 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,7 +195,8 @@ url-http-find-free-connection
 	(unwind-protect
             (let ((proc (url-open-stream host buf
                                          (if url-using-proxy
-                                             (url-host url-using-proxy)
+                                             (puny-encode-domain
+                                              (url-host url-using-proxy))
                                            host)
                                          (if url-using-proxy
                                              (url-port url-using-proxy)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 14 2022, Lars Ingebrigtsen wrote:

> +                                             (puny-encode-domain
> +                                              (url-host url-using-proxy))

That's the address of the proxy, not the original URL.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> +                                             (puny-encode-domain
>> +                                              (url-host url-using-proxy))
>
> That's the address of the proxy, not the original URL.

So it is.  Then I have no idea where the Host in the http case is coming
from.  I'd have thought that the proxy would just look at the Host
header (which should be puny-encoded), but I don't know much about http
proxying.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 14 2022, Lars Ingebrigtsen wrote:

> So it is.  Then I have no idea where the Host in the http case is coming
> from.  I'd have thought that the proxy would just look at the Host
> header (which should be puny-encoded), but I don't know much about http
> proxying.

The proxy just receives the full URL in the GET request.  Something like
this, but puny-encode-domain must only be called with the domain name.

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..c59613da62 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -332,7 +332,8 @@ url-http-create-request
               (if (and using-proxy
                        ;; Bug#35969.
                        (not (equal "https" (url-type url-http-target-url))))
-                  (url-recreate-url url-http-target-url) real-fname))
+                  (puny-encode-domain (url-recreate-url url-http-target-url))
+                real-fname))
              " HTTP/" url-http-version "\r\n"
              ;; Version of MIME we speak
              "MIME-Version: 1.0\r\n"

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> The proxy just receives the full URL in the GET request.  Something like
> this, but puny-encode-domain must only be called with the domain name.

Right.  Does the following untested patch work?

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 96a4742956..b5bcd123c7 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -332,7 +332,10 @@ url-http-create-request
               (if (and using-proxy
                        ;; Bug#35969.
                        (not (equal "https" (url-type url-http-target-url))))
-                  (url-recreate-url url-http-target-url) real-fname))
+                  (let ((url (copy-sequence url-http-target-url)))
+                    (setf (url-host url) (puny-encode-domain (url-host url)))
+                    (url-recreate-url url))
+                real-fname))
              " HTTP/" url-http-version "\r\n"
              ;; Version of MIME we speak
              "MIME-Version: 1.0\r\n"


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 15 2022, Lars Ingebrigtsen wrote:

> Right.  Does the following untested patch work?

I think it would be better to expand the url struct to also store the
IDNA host name.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> Right.  Does the following untested patch work?
>
> I think it would be better to expand the url struct to also store the
> IDNA host name.

And then have separate functions to reconstruct the URL with the IDNA or
the original domain name, depending on what we want?  I don't think that
makes that much sense.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54921: eww/url mishandles IDN over proxy

[Andreas Schwab]

On Apr 15 2022, Lars Ingebrigtsen wrote:

> And then have separate functions to reconstruct the URL with the IDNA or
> the original domain name, depending on what we want?  I don't think that
> makes that much sense.

Just add an optional argument.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#54921: eww/url mishandles IDN over proxy

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> And then have separate functions to reconstruct the URL with the IDNA or
>> the original domain name, depending on what we want?  I don't think that
>> makes that much sense.
>
> Just add an optional argument.

I don't think that is much of an improvement -- url-recreate-url is
pretty well-defined as it is.

So I've pushed the proposed patch, which I assume fixed the problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no