[PATCH] Add support for custom user agent strings in url.el

[PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I added support to url.el for a custom user agent string by defining a
variable url-custom-user-agent which contains a user-defined string. The
variable is nil by default, in which case the function
url-http-user-agent-string reutrns the default user agent.
(Patch was created on a git repository)

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ac2e140..1b63a25 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -214,11 +214,15 @@ request.")
          (and (listp url-privacy-level)
               (memq 'agent url-privacy-level)))
       ""
-    (format "User-Agent: %sURL/%s\r\n"
-           (if url-package-name
-               (concat url-package-name "/" url-package-version " ")
-             "")
-           url-version)))
+    
+    (if url-custom-user-agent
+       (format "User-Agent: %s\n" url-custom-user-agent)
+      
+      (format "User-Agent: %sURL/%s\r\n"
+             (if url-package-name
+                 (concat url-package-name "/" url-package-version " ")
+               "")
+             url-version))))
 
 (defun url-http-create-request (&optional ref-url)
   "Create an HTTP request for `url-http-target-url', referred to by REF-URL."
diff --git a/lisp/url/url.el b/lisp/url/url.el
index cbbcfd4..c54a94e 100644
--- a/lisp/url/url.el
+++ b/lisp/url/url.el
@@ -51,6 +51,10 @@
   :type 'directory
   :group 'url)
 
+(defvar url-custom-user-agent nil
+  "User Agent used by url-http-create-request to fetch pages. If nil,
+the default user agent will be used.")
+
 (defun url-do-setup ()
   "Setup the URL package.
 This is to avoid conflict with user settings if URL is dumped with

- -- 
Vibhav Pant
vibhavp@gmail.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCgAGBQJS24fqAAoJEO86onTybWtcUdkP/2s5MGg3/zY/Xg8KmSxAU20x
Ki0UamZHp5xNgr6UcN0SIseVGTTMBfJrwWL70LnslvvzRcPysRK0v+sOL4PjNcsT
nkOU8BjsTt4HMFi48LPqhr+OacAFrMVT0E3vo0rtYoQSSw7tNgEprLUr3i2Yc1vY
II23YllVdIqb6Ys6arYuMb5V5vP83GfnV9hm1jNSt3TGefSMsgDVggBRiw7NBk8Q
pJqZ14rblX68V+b2cmw8JAzDkq7mWZ+s6CU/Ic4KI3a37k3kvoDwXwU8kCMVpP7Q
aBK9VzSLlDzzLD+TZI7I7K5n6dA4He4BwwhFgQuUwOmoI5nF3W7y1gSOCL4sBOLF
5tqlRZFR+RJUDGabV6PoSux2K61sYegMN0NzzyrUCTPwqb9ISMNBfGfL0lp/QFuh
ENxHmTnyO/J+FTIFAQ4Dbv4RYn5xxivgPp4UmYG3vin7b3E/+2jsOKdeQO/V5iBA
85XUKEdxH5JUergw0A9rsBvWkRw/iy89+edqg1hNJkWD8sSY+4H3QXGly63mQzdc
51QWCwAJ1ShPif8RI9yD1Eyvctoasv+WM42NakCZ7bDUOwuHqJfAAEVbQeg9ijn9
EynM7n5CmAAap6vrjdeNWJvxPWPXv32UHSsstOAN7MPgd9Cw6gZvfYY4DITHqWbe
BykRZe2zqv4iIlgL87Rg
=8VAL
-----END PGP SIGNATURE-----

Re: [PATCH] Add support for custom user agent strings in url.el

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 2953 bytes --]

Seems to me that there should be just a defcustom `url-user-agent' and it
should be used always. By default it should have whatever value it
currently has and users will be able to customise it.


On 19 January 2014 10:10, Vibhav Pant <vibhavp@gmail.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> I added support to url.el for a custom user agent string by defining a
> variable url-custom-user-agent which contains a user-defined string. The
> variable is nil by default, in which case the function
> url-http-user-agent-string reutrns the default user agent.
> (Patch was created on a git repository)
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index ac2e140..1b63a25 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -214,11 +214,15 @@ request.")
>           (and (listp url-privacy-level)
>                (memq 'agent url-privacy-level)))
>        ""
> -    (format "User-Agent: %sURL/%s\r\n"
> -           (if url-package-name
> -               (concat url-package-name "/" url-package-version " ")
> -             "")
> -           url-version)))
> +
> +    (if url-custom-user-agent
> +       (format "User-Agent: %s\n" url-custom-user-agent)
> +
> +      (format "User-Agent: %sURL/%s\r\n"
> +             (if url-package-name
> +                 (concat url-package-name "/" url-package-version " ")
> +               "")
> +             url-version))))
>
>  (defun url-http-create-request (&optional ref-url)
>    "Create an HTTP request for `url-http-target-url', referred to by
> REF-URL."
> diff --git a/lisp/url/url.el b/lisp/url/url.el
> index cbbcfd4..c54a94e 100644
> --- a/lisp/url/url.el
> +++ b/lisp/url/url.el
> @@ -51,6 +51,10 @@
>    :type 'directory
>    :group 'url)
>
> +(defvar url-custom-user-agent nil
> +  "User Agent used by url-http-create-request to fetch pages. If nil,
> +the default user agent will be used.")
> +
>  (defun url-do-setup ()
>    "Setup the URL package.
>  This is to avoid conflict with user settings if URL is dumped with
>
> - --
> Vibhav Pant
> vibhavp@gmail.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBCgAGBQJS24fqAAoJEO86onTybWtcUdkP/2s5MGg3/zY/Xg8KmSxAU20x
> Ki0UamZHp5xNgr6UcN0SIseVGTTMBfJrwWL70LnslvvzRcPysRK0v+sOL4PjNcsT
> nkOU8BjsTt4HMFi48LPqhr+OacAFrMVT0E3vo0rtYoQSSw7tNgEprLUr3i2Yc1vY
> II23YllVdIqb6Ys6arYuMb5V5vP83GfnV9hm1jNSt3TGefSMsgDVggBRiw7NBk8Q
> pJqZ14rblX68V+b2cmw8JAzDkq7mWZ+s6CU/Ic4KI3a37k3kvoDwXwU8kCMVpP7Q
> aBK9VzSLlDzzLD+TZI7I7K5n6dA4He4BwwhFgQuUwOmoI5nF3W7y1gSOCL4sBOLF
> 5tqlRZFR+RJUDGabV6PoSux2K61sYegMN0NzzyrUCTPwqb9ISMNBfGfL0lp/QFuh
> ENxHmTnyO/J+FTIFAQ4Dbv4RYn5xxivgPp4UmYG3vin7b3E/+2jsOKdeQO/V5iBA
> 85XUKEdxH5JUergw0A9rsBvWkRw/iy89+edqg1hNJkWD8sSY+4H3QXGly63mQzdc
> 51QWCwAJ1ShPif8RI9yD1Eyvctoasv+WM42NakCZ7bDUOwuHqJfAAEVbQeg9ijn9
> EynM7n5CmAAap6vrjdeNWJvxPWPXv32UHSsstOAN7MPgd9Cw6gZvfYY4DITHqWbe
> BykRZe2zqv4iIlgL87Rg
> =8VAL
> -----END PGP SIGNATURE-----
>
>

[-- Attachment #2: Type: text/html, Size: 3750 bytes --]

Re: [PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

Bozhidar Batsov writes:

> Seems to me that there should be just a defcustom `url-user-agent' and it
> should be used always. By default it should have whatever value it
> currently has and users will be able to customise it.
The function url-http-user-agent-string generates a user agent by
concatinating url-package-name and url-package-version. So the defcustom
would have to be a function or a sexp, making things a little confusing
for new users. 

-- 
Vibhav Pant
vibhavp@gmail.com

Re: [PATCH] Add support for custom user agent strings in url.el

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

The default of the defcustom can also be the result of a function call
(like `format', etc). Basically everything that returns a string will do.


On 19 January 2014 10:36, Vibhav Pant <vibhavp@gmail.com> wrote:

>
> Bozhidar Batsov writes:
>
> > Seems to me that there should be just a defcustom `url-user-agent' and it
> > should be used always. By default it should have whatever value it
> > currently has and users will be able to customise it.
> The function url-http-user-agent-string generates a user agent by
> concatinating url-package-name and url-package-version. So the defcustom
> would have to be a function or a sexp, making things a little confusing
> for new users.
>
> --
> Vibhav Pant
> vibhavp@gmail.com
>

[-- Attachment #2: Type: text/html, Size: 1206 bytes --]

Re: [PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

Here's the fixed patch. I created a defcustom for the user-agent in
url-http.el.

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ac2e140..6a0ec13 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -132,6 +132,15 @@ request.")
     (507 insufficient-storage            "Insufficient storage"))
   "The HTTP return codes and their text.")

+(defcustom url-user-agent (format "User-Agent: %sURL/%s\r\n"
+                                 (if url-package-name
+                                     (concat url-package-name "/"
+                                             url-package-version " ")
+                                   "") url-version)
+  "User Agent used by the URL package."
+  :type 'string
+  :group 'url)
+
 ;(eval-when-compile
 ;; These are all macros so that they are hidden from external sight
 ;; when the file is byte-compiled.
@@ -214,11 +223,7 @@ request.")
          (and (listp url-privacy-level)
               (memq 'agent url-privacy-level)))
       ""
-    (format "User-Agent: %sURL/%s\r\n"
-           (if url-package-name
-               (concat url-package-name "/" url-package-version " ")
-             "")
-           url-version)))
+    url-user-agent))

 (defun url-http-create-request (&optional ref-url)
   "Create an HTTP request for `url-http-target-url', referred to by
REF-URL."



On Sun, Jan 19, 2014 at 2:42 PM, Bozhidar Batsov <bozhidar@batsov.com>wrote:

> The default of the defcustom can also be the result of a function call
> (like `format', etc). Basically everything that returns a string will do.
>
>
> On 19 January 2014 10:36, Vibhav Pant <vibhavp@gmail.com> wrote:
>
>>
>> Bozhidar Batsov writes:
>>
>> > Seems to me that there should be just a defcustom `url-user-agent' and
>> it
>> > should be used always. By default it should have whatever value it
>> > currently has and users will be able to customise it.
>> The function url-http-user-agent-string generates a user agent by
>> concatinating url-package-name and url-package-version. So the defcustom
>> would have to be a function or a sexp, making things a little confusing
>> for new users.
>>
>> --
>> Vibhav Pant
>> vibhavp@gmail.com
>>
>
>


-- 
Vibhav Pant
vibhavp@gmail.com

[-- Attachment #2: Type: text/html, Size: 3402 bytes --]

Re: [PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

By the way, the patch will still respect the paranoid option, where no user
agent string is sent.


On Sun, Jan 19, 2014 at 3:15 PM, Vibhav Pant <vibhavp@ubuntu.com> wrote:

> Here's the fixed patch. I created a defcustom for the user-agent in
> url-http.el.
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index ac2e140..6a0ec13 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -132,6 +132,15 @@ request.")
>      (507 insufficient-storage            "Insufficient storage"))
>    "The HTTP return codes and their text.")
>
> +(defcustom url-user-agent (format "User-Agent: %sURL/%s\r\n"
>
> +                                 (if url-package-name
> +                                     (concat url-package-name "/"
> +                                             url-package-version " ")
> +                                   "") url-version)
> +  "User Agent used by the URL package."
> +  :type 'string
> +  :group 'url)
> +
>  ;(eval-when-compile
>  ;; These are all macros so that they are hidden from external sight
>  ;; when the file is byte-compiled.
> @@ -214,11 +223,7 @@ request.")
>
>           (and (listp url-privacy-level)
>                (memq 'agent url-privacy-level)))
>        ""
> -    (format "User-Agent: %sURL/%s\r\n"
> -           (if url-package-name
> -               (concat url-package-name "/" url-package-version " ")
> -             "")
> -           url-version)))
> +    url-user-agent))
>
>
>  (defun url-http-create-request (&optional ref-url)
>    "Create an HTTP request for `url-http-target-url', referred to by
> REF-URL."
>
>
>
> On Sun, Jan 19, 2014 at 2:42 PM, Bozhidar Batsov <bozhidar@batsov.com>wrote:
>
>> The default of the defcustom can also be the result of a function call
>> (like `format', etc). Basically everything that returns a string will do.
>>
>>
>> On 19 January 2014 10:36, Vibhav Pant <vibhavp@gmail.com> wrote:
>>
>>>
>>> Bozhidar Batsov writes:
>>>
>>> > Seems to me that there should be just a defcustom `url-user-agent' and
>>> it
>>> > should be used always. By default it should have whatever value it
>>> > currently has and users will be able to customise it.
>>> The function url-http-user-agent-string generates a user agent by
>>> concatinating url-package-name and url-package-version. So the defcustom
>>> would have to be a function or a sexp, making things a little confusing
>>> for new users.
>>>
>>> --
>>> Vibhav Pant
>>> vibhavp@gmail.com
>>>
>>
>>
>
>
> --
> Vibhav Pant
> vibhavp@gmail.com
>



-- 
Vibhav Pant
vibhavp@ubuntu.com
"0x2B | ~ 0x2B (Hamlet, Shakespeare)"

[-- Attachment #2: Type: text/html, Size: 4242 bytes --]

Re: [PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

[-- Attachment #1: Type: text/plain, Size: 126 bytes --]

And here's the attached patch file for those whose email clients
cannot view it correctly.

-- 
Vibhav Pant
vibhavp@gmail.com

[-- Attachment #2: url-custom-user-agent.patch --]
[-- Type: text/x-patch, Size: 1289 bytes --]

commit f380c64e2b3a6c892c8f2821b0d902cf5ca9d63d
Author: Vibhav Pant <vibhavp@gmail.com>
Date:   Sun Jan 19 13:45:28 2014 +0000

    Add support for custom user agent strings.

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index ac2e140..6a0ec13 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -132,6 +132,15 @@ request.")
     (507 insufficient-storage            "Insufficient storage"))
   "The HTTP return codes and their text.")
 
+(defcustom url-user-agent (format "User-Agent: %sURL/%s\r\n"
+				  (if url-package-name
+				      (concat url-package-name "/"
+					      url-package-version " ")
+				    "") url-version)
+  "User Agent used by the URL package."
+  :type 'string
+  :group 'url)
+
 ;(eval-when-compile
 ;; These are all macros so that they are hidden from external sight
 ;; when the file is byte-compiled.
@@ -214,11 +223,7 @@ request.")
 	  (and (listp url-privacy-level)
 	       (memq 'agent url-privacy-level)))
       ""
-    (format "User-Agent: %sURL/%s\r\n"
-	    (if url-package-name
-		(concat url-package-name "/" url-package-version " ")
-	      "")
-	    url-version)))
+    url-user-agent))
 
 (defun url-http-create-request (&optional ref-url)
   "Create an HTTP request for `url-http-target-url', referred to by REF-URL."

Re: [PATCH] Add support for custom user agent strings in url.el

[Stefan Monnier]

> Here's the fixed patch. I created a defcustom for the user-agent in
> url-http.el.

Looks good, thank you.  Could someone remember to apply it when we
reopen the trunk?


        Stefan

Re: [PATCH] Add support for custom user agent strings in url.el

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Here's the fixed patch. I created a defcustom for the user-agent in
>> url-http.el.
>
> Looks good, thank you.  Could someone remember to apply it when we
> reopen the trunk?

Vibhav, if you send the patch to debbugs and mark it as "pending", then
debbugs should remind us.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: [PATCH] Add support for custom user agent strings in url.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The default "user agent" string should ideally be one that many users'
browsers will send, so as to give little information to web sites.

    >  (defun url-http-create-request (&optional ref-url)
    >    "Create an HTTP request for `url-http-target-url', referred to by
    > REF-URL."

That looks like a doc string bug -- the first line should stand on its
own.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: [PATCH] Add support for custom user agent strings in url.el

[Rüdiger Sonderfeld]

On Monday 20 January 2014 04:13:38 Richard Stallman wrote:
> The default "user agent" string should ideally be one that many users'
> browsers will send, so as to give little information to web sites.

It should at least do so for the `paranoid' setting.  Not sending a User-agent 
at all is probably far easier to trace than sending a commonly used User-
agent.  Although the other headers might be a give away.

The `defcustom' could provide the user with a few default suggestions for 
popular User-agent strings.

Regards,
Rüdiger

Re: [PATCH] Add support for custom user agent strings in url.el

[Rüdiger Sonderfeld]

On Sunday 19 January 2014 15:28:45 Vibhav Pant wrote:
> By the way, the patch will still respect the paranoid option, where no user
> agent string is sent.

Looks good to me.  But maybe the docstring should inform the user about the 
different behaviour for the `paranoid' setting.

Regards,
Rüdiger

Re: [PATCH] Add support for custom user agent strings in url.el

[Vibhav Pant]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

"Rüdiger Sonderfeld" <ruediger@c-plusplus.de> wrote:
>On Sunday 19 January 2014 15:28:45 Vibhav Pant wrote:
>> By the way, the patch will still respect the paranoid option, where
>no user
>> agent string is sent.
>
>Looks good to me.  But maybe the docstring should inform the user about
>the
>different behaviour for the `paranoid' setting.
>
>Regards,
>Rüdiger

The docstring for the paranoid option already does that. We could add a "link" to the paranoid option if the docstring allows us to do that.
- --
Vibhav Pant
vibhavp@gmail.com

Sent from my Android device with K-9 Mail. Please excuse my brevity.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.0.9

iQI9BAEBCAAnBQJS3SyyIBxWaWJoYXYgUGFudCA8dmliaGF2cEBnbWFpbC5jb20+
AAoJEO86onTybWtcEPMP/iu6WMI1br6REinsyHIuOuCjfCKczUOKFhjcHQ1VkQOf
4QdkOl09qN+q2oRAQATrpaJMq1PzE/wFRNwgTJK3/m7hTr0IkPy9sF19fENkkVy+
dOFd0QucS1GQoQw7l8axSJDsWvLEHbT1wDUUbVPDGkrkQbJ8isZakGg2LsxwSXZA
BzfHvXJ4cus31qvQLQFBXS4wEFPtHj0mR+gVjQ8UhRrp9vT3sYlM1fGKG9kj7GyQ
dFKKKQSHjUO6wnCSYXLoXjiwNqtxEa9WUeAHtGxJYjnlraNYNcq0fhlqpOf0L6m4
oYzIw6Jd29tjsdZfu78nVSsR5faU86Q7jBdbfLi5Sz6o3N7yIYx48g7vk6vxkHPm
5wtYVLHbyqro37lYOyYDD/mKkdA2WfmzlK3uT4vXPtiIsWPTuBRU/IQF1uUxEby4
dMpNCO4Az+d8EPuAas82J9rC9lPoc/M8r4LIIK2CnNMIkgtz1IVgmaHy7lz4lkxr
dD4s+/M1WLCBC0rCPSMnaAWB8GQD9PkqWrT2DH1T75tXlAA5uJZL86zatnUYCmN0
U8EYSou9KVMEYWBkm6k3vPG5NgO2IU8OTxBQa1CZ2OEyHGwGT7/Ia8t0v1jLHpH+
SwKj57u6E+0w5EMYck9i5PS96ymFQGvXQF/S9d8Nv1Ntr0aO3g5QEKKUnSFIxfde
=VCAr
-----END PGP SIGNATURE-----