Is there something like `file-name-concat', but for urls?

Is there something like `file-name-concat', but for urls?

[Marcin Borkowski]

Hi all,

given e.g. "https", "example.com" (or "example.com/") and "page", I want
to get "https://example.com/page".  Is there a built-in Elisp function
to do that?

TIA,


-- 
Marcin Borkowski
http://mbork.pl

Re: Is there something like `file-name-concat', but for urls?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Date: Mon, 24 Jul 2023 07:14:23 +0200
> 
> given e.g. "https", "example.com" (or "example.com/") and "page", I want
> to get "https://example.com/page".  Is there a built-in Elisp function
> to do that?

url-expand-file-name ?

Re: Is there something like `file-name-concat', but for urls?

[Marcin Borkowski]

On 2023-07-24, at 13:58, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Date: Mon, 24 Jul 2023 07:14:23 +0200
>> 
>> given e.g. "https", "example.com" (or "example.com/") and "page", I want
>> to get "https://example.com/page".  Is there a built-in Elisp function
>> to do that?
>
> url-expand-file-name ?

Thanks, that looks close enough!

-- 
Marcin Borkowski
http://mbork.pl

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

Marcin Borkowski wrote:

> given e.g. "https", "example.com" (or "example.com/") and
> "page", I want to get "https://example.com/page". Is there
> a built-in Elisp function to do that?

Do it yourself Marcin :) Even more so because you already did
something like that, with the filename normalizer, I remember!

If so, you can use the correct URL terminology, which is

1. scheme:
2. //server
3. [:port]
4. [/path]
5. [/program?q=argument]
6. [#fragment]

See this page,

  https://dataswamp.org/~incal/distsys/url.html

BTW thanks for letting me realize I should modernize that, it
is so old https isn't even included :$

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

On Tue, Jul 25, 2023 at 06:21:49PM +0200, Emanuel Berg wrote:
> Marcin Borkowski wrote:
> 
> > given e.g. "https", "example.com" (or "example.com/") and
> > "page", I want to get "https://example.com/page". Is there
> > a built-in Elisp function to do that?
> 
> Do it yourself Marcin :) Even more so because you already did
> something like that, with the filename normalizer, I remember!
> 
> If so, you can use the correct URL terminology, which is
> 
> 1. scheme:
> 2. //server
> 3. [:port]
> 4. [/path]
> 5. [/program?q=argument]
> 6. [#fragment]
> 
> See this page,
> 
>   https://dataswamp.org/~incal/distsys/url.html
> 
> BTW thanks for letting me realize I should modernize that, it
> is so old https isn't even included :$

Plus URL encoding. Plus deciding in fringe cases whether encoding
is in order or not (and perhaps avoiding those cases where it would
be in order but there are botched implementations around so better
don't (ISTR there was a security hole where the reverse proxy
charged with securing access had a different idea than the server
behind it). So RFC 3986 plus perhaps RFC 8820 plus a couple plus
"Real Life" (TM).

I mean: it sure is fun, and I've caught myself doing this too, but
some working library with the most egregious botches already fixed
and where new fixes can go to seems to be a good idea, in general :)

That doesn't mean one shouldn't try: you sure understand better
how that lib works once you have.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

tomas wrote:

>> BTW thanks for letting me realize I should modernize that,
>> it is so old https isn't even included [...]
>
> Plus URL encoding. Plus deciding in fringe cases whether
> encoding is in order or not [...]

How the URL can look is defined by a grammar, this is
Computer Science theory of Compiler Design. Because, if we get
the standard definition, expressed as a grammar, we can not
only use the correct terminology from there, we can also
verify if the result URL is valid or not.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

On Tue, Jul 25, 2023 at 08:33:12PM +0200, Emanuel Berg wrote:
> tomas wrote:
> 
> >> BTW thanks for letting me realize I should modernize that,
> >> it is so old https isn't even included [...]
> >
> > Plus URL encoding. Plus deciding in fringe cases whether
> > encoding is in order or not [...]
> 
> How the URL can look is defined by a grammar

Have you read RFC 3986?

> this is
> Computer Science theory of Compiler Design. Because, if we get
> the standard definition, expressed as a grammar, we can not
> only use the correct terminology from there, we can also
> verify if the result URL is valid or not.

No. This is, alas, Computer Tinkering Practice, where RFCs try
as well as they can, to codify existing practice. Some try to
even set up a grammar, often in some EBNF dialect -- but practice
and existing implementations often emerge long before that
grammar was written, shaped by negligence, inexperience, not
seldomly by economical or political interests.

When people, at last, meet at a table to hammer out an RFC,
often they are sent in by companies trying to defend the stakes
they have rammed in the ground.

Once the RFC is out, if, say, Microsoft or Google is doing
something not according to it, you can shout at them until you
are blue in the face. It won't help, anyway.

Sometimes, standards bodies just give up and declare something
as a "living standard", which means "whatever we come up with
this week", as happens to HTML5.

This is, alas, reality, where our "industry" is dominated by
a handful of actors.

Enjoy, nevertheless.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

tomas wrote:

>> this is Computer Science theory of Compiler Design.
>> Because, if we get the standard definition, expressed as
>> a grammar, we can not only use the correct terminology from
>> there, we can also verify if the result URL is valid
>> or not.
>
> No. This is, alas, Computer Tinkering Practice [...]

Compiling is at the heart of Computer Science.

If anything it is _more_ classic CS than today's Master
programs that are often embedded systems, HCI and such much
more modern disciplines.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[Marcin Borkowski]

On 2023-07-25, at 19:24, tomas@tuxteam.de wrote:

> I mean: it sure is fun, and I've caught myself doing this too, but
> some working library with the most egregious botches already fixed
> and where new fixes can go to seems to be a good idea, in general :)

Exactly.  I don't want to spend to much time on this...

Thanks,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

Marcin Borkowski wrote:

>> I mean: it sure is fun, and I've caught myself doing this
>> too, but some working library with the most egregious
>> botches already fixed and where new fixes can go to seems
>> to be a good idea, in general :)
>
> Exactly. I don't want to spend to much time on this...

But there are standard tools for this task, so no need to
re-implement anything manually ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

>>> I mean: it sure is fun, and I've caught myself doing this
>>> too, but some working library with the most egregious
>>> botches already fixed and where new fixes can go to seems
>>> to be a good idea, in general :)
>>
>> Exactly. I don't want to spend to much time on this...
>
> But there are standard tools for this task, so no need to
> re-implement anything manually ...

Here, I think this will do:

  https://elpa.gnu.org/packages/peg.html

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[Yuri Khan]

On Tue, 25 Jul 2023 at 23:39, Emanuel Berg <incal@dataswamp.org> wrote:

> If so, you can use the correct URL terminology, which is
>
> 1. scheme:
> 2. //server

It’s host rather than server, and you left out the user and password.

> 3. [:port]
> 4. [/path]
> 5. [/program?q=argument]

No such thing as program in URIs. It’s all part of the path. Also,
query does not have to be structured as k1=v1&k2=v2&….

> 6. [#fragment]


But yes, the correct answer to the OP would be ‘url-split’ and
‘url-unsplit’ if only such functions were defined.

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

Yuri Khan wrote:

> It's host rather than server, and you left out the user and
> password

Well, it is obviously not an exhaustive list but I agree host
is better so let's change that, thanks.

Maybe there is an RFC publication or W3C standard somewhere
with a formal and complete grammar ...

> But yes, the correct answer to the OP would be `url-split'
> and `url-unsplit' if only such functions were defined.

Let's find a definition first so the arguments get their
correct names, I thought that was it but maybe I got that set
from some textbook or random internet source rather than from
an official source then ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[Yuri Khan]

On Wed, 26 Jul 2023 at 02:28, Emanuel Berg <incal@dataswamp.org> wrote:

> Maybe there is an RFC publication or W3C standard somewhere
> with a formal and complete grammar ...

Of course there is, it’s called RFC 3986.

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

Yuri Khan wrote:

> On Wed, 26 Jul 2023 at 02:28, Emanuel Berg <incal@dataswamp.org> wrote:
>
>> Maybe there is an RFC publication or W3C standard somewhere
>> with a formal and complete grammar ...
>
> Of course there is, it’s called RFC 3986.

^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?

https://datatracker.ietf.org/doc/html/rfc3986#appendix-B

Case closed ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Is there something like `file-name-concat', but for urls?

[Emanuel Berg]

Here is how to do it in Emacs.

If you read the docstring for `url-generic-parse-url', you see
that it is based on RFC 3986, even.

Don't mention it ...

;;; -*- lexical-binding: t -*-
;;
;; this file:
;;   https://dataswamp.org/~incal/emacs-init/misc.el

(require 'cl-lib)
(require 'url-parse)

(defun valid-url (url)
  (let ((data (url-generic-parse-url url)))
    (when (cl-struct-slot-value 'url 'type data)
      t) ))

;; (valid-url "https://dataswamp.org/~incal/") ; t
;; (valid-url "dataswamp.org/~incal/")         ; nil

-- 
underground experts united
https://dataswamp.org/~incal