bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]


`url-generic-parse-url' from url-parse does not implement correctly
relative path handling as described in the RFC3986 and subsections of
section 5.2.

URL listed in 5.4.1 and 5.4.2 does not work such as:

      "../../../g"    =  "http://a/g"

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> URL listed in 5.4.1 and 5.4.2 does not work such as:
>
>       "../../../g"    =  "http://a/g"

Where does the "a" come from?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

On Sun, Sep 11 2011, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> URL listed in 5.4.1 and 5.4.2 does not work such as:
>>
>>       "../../../g"    =  "http://a/g"
>
> Where does the "a" come from?

Read:

       "http://a/../../../g"    =  "http://a/g"

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Andreas Schwab]

Julien Danjou <julien@danjou.info> writes:

> Read:
>
>        "http://a/../../../g"    =  "http://a/g"

ELISP> (url-generic-parse-url "http://a/../../../g")
[cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t]

How is that wrong?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Sun, Sep 11 2011, Andreas Schwab wrote:

> ELISP> (url-generic-parse-url "http://a/../../../g")
> [cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t]
>
> How is that wrong?

Just tested with telnet:

GET /url/ HTTP/1.1
Host: domain.com

HTTP/1.1 200 OK



GET /../../../url/ HTTP/1.1
Host: domain.com

HTTP/1.1 400 Bad Request


-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

>> Julien Danjou <julien@danjou.info> writes:
>>
>>> URL listed in 5.4.1 and 5.4.2 does not work such as:
>>>
>>>       "../../../g"    =  "http://a/g"
>>
>> Where does the "a" come from?
>
> Read:
>
>        "http://a/../../../g"    =  "http://a/g"

5.2 talks about relative URLs, though, so "http://a/../../../g" seems
irrelevant...

5.4.1 also talks about relative URLs, and how to resolve them if you
have a base address.  But `url-generic-parse-url' does not, as far as I
can tell, deal with the concept of "relative URLs and bases", but only
parses full, non-relative URLs.

And for a full URL, this looks correct to me:

(url-generic-parse-url "http://a/../../../g")
=>
[cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t nil]

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Andreas Schwab]

Julien Danjou <julien@danjou.info> writes:

> On Sun, Sep 11 2011, Andreas Schwab wrote:
>
>> ELISP> (url-generic-parse-url "http://a/../../../g")
>> [cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t]
>>
>> How is that wrong?
>
> Just tested with telnet:
>
> GET /url/ HTTP/1.1
> Host: domain.com
>
> HTTP/1.1 200 OK
>
>
>
> GET /../../../url/ HTTP/1.1
> Host: domain.com
>
> HTTP/1.1 400 Bad Request

What are you trying to prove?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

On Sun, Sep 11 2011, Andreas Schwab wrote:

> What are you trying to prove?

That your example:

ELISP> (url-generic-parse-url "http://a/../../../g")
[cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t]

will not work.

Could you give a try, maybe you'll understand?

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Andreas Schwab]

Julien Danjou <julien@danjou.info> writes:

> On Sun, Sep 11 2011, Andreas Schwab wrote:
>
>> What are you trying to prove?
>
> That your example:
>
> ELISP> (url-generic-parse-url "http://a/../../../g")
> [cl-struct-url "http" nil nil "a" 80 "/../../../g" nil nil t]
>
> will not work.

In which way does it not work?  There is the scheme "http", the host
"a", the port 80 and the local part "/../../../g".  Everything is fully
correct according to the rules of a URI.  How the local part is
interpreted is only defined by the remote host.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]

On Sun, Sep 11 2011, Andreas Schwab wrote:
> In which way does it not work?  There is the scheme "http", the host
> "a", the port 80 and the local part "/../../../g".  Everything is fully
> correct according to the rules of a URI.  How the local part is
> interpreted is only defined by the remote host.

Men, this is getting me crazy. Let me rephrase the whole thing.

You got an URL of the form:

    http://a/../../../foobar.png

If you use Firefox, Chromium, wget, or whatever to retrieve it, the
program will act according to RFC3986 and transform that URL to:

    http://a/foobar.png

`url' from Emacs will not, and will fail to retrieve the image.


Now I may be mistaken about where in the code the bug is, but there's a
bug: the `url' functions are unable to fetch such an URL, whereas any
other tool is able to.

In Lisp:

(switch-to-buffer (url-retrieve-synchronously
  "http://www.gnu.org/graphics/t-desktop-4-small.jpg"))
=> That works

(switch-to-buffer (url-retrieve-synchronously
  "http://www.gnu.org/../graphics/t-desktop-4-small.jpg"))
=> Show a 400 bad request

% wget http://www.gnu.org/../graphics/t-desktop-4-small.jpg
--2011-09-11 17:52:56--  http://www.gnu.org/graphics/t-desktop-4-small.jpg
Resolving www.gnu.org (www.gnu.org)... 140.186.70.148
Connecting to www.gnu.org (www.gnu.org)|140.186.70.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30195 (29K) [image/jpeg]
Saving to: `t-desktop-4-small.jpg'

100%[=======================================================================================================================================================================================================================================>] 30,195      66.5K/s   in 0.4s    

2011-09-11 17:52:56 (66.5 KB/s) - `t-desktop-4-small.jpg' saved [30195/30195]

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Men, this is getting me crazy. Let me rephrase the whole thing.
>
> You got an URL of the form:
>
>     http://a/../../../foobar.png
>
> If you use Firefox, Chromium, wget, or whatever to retrieve it, the
> program will act according to RFC3986 and transform that URL to:
>
>     http://a/foobar.png

Where in RFC3986 does it say that you're supposed to do that?  5.2
(etc.) only talks about relative URLs.

You may be totally correct that url.el should do this, ../-stripping in
absolute URLs, but does the RFC actually say so?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Andreas Schwab]

Julien Danjou <julien@danjou.info> writes:

> You got an URL of the form:
>
>     http://a/../../../foobar.png

This is a URI, not a relative-ref, so the rules for . and .. don't apply.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

On Sun, Sep 11 2011, Lars Magne Ingebrigtsen wrote:

> Where in RFC3986 does it say that you're supposed to do that?  5.2
> (etc.) only talks about relative URLs.
> 
> You may be totally correct that url.el should do this, ../-stripping in
> absolute URLs, but does the RFC actually say so?

It seems to me that it's what 5.4 talks about.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> It seems to me that it's what 5.4 talks about.

This is how 5.4 starts:

5.4.  Reference Resolution Examples

   Within a representation with a well defined base URI of

      http://a/b/c/d;p?q

   a relative reference is transformed to its target URI as follows.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Sun, Sep 11 2011, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> It seems to me that it's what 5.4 talks about.
>
> This is how 5.4 starts:
>
> 5.4.  Reference Resolution Examples
>
>    Within a representation with a well defined base URI of
>
>       http://a/b/c/d;p?q
>
>    a relative reference is transformed to its target URI as follows.

Then go to 5.4.2:

   Although the following abnormal examples are unlikely to occur in
   normal practice, all URI parsers should be capable of resolving them
   consistently.  Each example uses the same base as that above.

   Parsers must be careful in handling cases where there are more ".."
   segments in a relative-path reference than there are hierarchical
   levels in the base URI's path.  Note that the ".." syntax cannot be
   used to change the authority component of a URI.


-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Then go to 5.4.2:
>
>    Although the following abnormal examples are unlikely to occur in
>    normal practice, all URI parsers should be capable of resolving them
>    consistently.  Each example uses the same base as that above.
>
>    Parsers must be careful in handling cases where there are more ".."
>    segments in a relative-path reference than there are hierarchical
>    levels in the base URI's path.  Note that the ".." syntax cannot be
>    used to change the authority component of a URI.

Still talking about "base" and "relative".  Nothing about parsing a full
URL.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

However, as a practical issue, it might make sense to just do what
Firefox does, which is probably "best practise".  If for no other reason
than that Firefox does it. 

It will strip "../" and "./" from the front of the local part of all
URLs, according to tcpdump.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I'm sorry, I am missing why this does not apply to the example I gave.

http://a/../../b is not a relative URL.  :-)

The entire thing is about how to glue a base URL, like "http://a/b/c"
together with a relative URL, like "../../d".  If you're not gluing URLs
together, then the section does not apply.  At all.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

On Sun, Sep 11 2011, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> Then go to 5.4.2:
>>
>>    Although the following abnormal examples are unlikely to occur in
>>    normal practice, all URI parsers should be capable of resolving them
>>    consistently.  Each example uses the same base as that above.
>>
>>    Parsers must be careful in handling cases where there are more ".."
>>    segments in a relative-path reference than there are hierarchical
>>    levels in the base URI's path.  Note that the ".." syntax cannot be
>>    used to change the authority component of a URI.
>
> Still talking about "base" and "relative".  Nothing about parsing a full
> URL.

I'm sorry, I am missing why this does not apply to the example I gave.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Sun, Sep 11 2011, Lars Magne Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> I'm sorry, I am missing why this does not apply to the example I gave.
>
> http://a/../../b is not a relative URL.  :-)
>
> The entire thing is about how to glue a base URL, like "http://a/b/c"
> together with a relative URL, like "../../d".  If you're not gluing URLs
> together, then the section does not apply.  At all.

Oh ok. But if you consider the base URL being http://a/, you clearly
match this case. I admit we are not doing relative anyhow, but I don't
think absolute and relative are supposed to have different behaviour in
such a case.

And it seems that 5.2.4 clearly explains the algo in our (absolute) case,
don't you think?


-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Magne Ingebrigtsen]

However, as a practical issue, it might make sense to just do what
Firefox does, which is probably "best practise".  If for no other reason
than that Firefox does it. 

It will strip "../" and "./" from the front of the local part of all
URLs, according to tcpdump.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> However, as a practical issue, it might make sense to just do what
> Firefox does, which is probably "best practise".  If for no other reason
> than that Firefox does it. 
>
> It will strip "../" and "./" from the front of the local part of all
> URLs, according to tcpdump.

Hey, I wonder what Alain's new URL parsing function does here.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#8622: 24.0.50; url-parse does not implement RFC3986 5.2

[Lars Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> However, as a practical issue, it might make sense to just do what
> Firefox does, which is probably "best practise".  If for no other reason
> than that Firefox does it. 
>
> It will strip "../" and "./" from the front of the local part of all
> URLs, according to tcpdump.

I've checked Chromium, too, and it does the same.

I think it might make sense for eww to strip the "../" things, and leave
url-parse at it is, since it should be standards-compliant, and the
standards don't seem to say anything about this.

I'll tweak eww.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no