From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: =?UTF-8?Q?Cl=c3=a9ment_Pit--Claudel?= Newsgroups: gmane.emacs.devel Subject: Re: Help understanding the URL code Date: Thu, 7 Apr 2016 12:02:56 +0100 Message-ID: <57063E60.4040408@gmail.com> References: <5703DBB2.1030903@gmail.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="sHCRVXMeHiC4QWgIAxNt1pXuv3NMIAqOI" X-Trace: ger.gmane.org 1460027018 6168 80.91.229.3 (7 Apr 2016 11:03:38 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 7 Apr 2016 11:03:38 +0000 (UTC) Cc: Emacs developers To: Lars Magne Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Apr 07 13:03:32 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1ao7il-0000Su-TB for ged-emacs-devel@m.gmane.org; Thu, 07 Apr 2016 13:03:32 +0200 Original-Received: from localhost ([::1]:48950 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao7il-0004Pz-3Q for ged-emacs-devel@m.gmane.org; Thu, 07 Apr 2016 07:03:31 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:37152) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao7iI-0004Ib-L8 for emacs-devel@gnu.org; Thu, 07 Apr 2016 07:03:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ao7iF-0007Fn-DW for emacs-devel@gnu.org; Thu, 07 Apr 2016 07:03:02 -0400 Original-Received: from mout.kundenserver.de ([212.227.17.24]:60408) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao7iF-0007FO-3X for emacs-devel@gnu.org; Thu, 07 Apr 2016 07:02:59 -0400 Original-Received: from [146.179.198.83] ([146.179.198.83]) by mrelayeu.kundenserver.de (mreue101) with ESMTPSA (Nemesis) id 0MFcKD-1b0RVs3s61-00Eccg; Thu, 07 Apr 2016 13:02:57 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 In-Reply-To: X-Provags-ID: V03:K0:/f92wMFqSFlXE9rqWVMe3GKBdekj7wc3aJlV3Qb43RY9wvA8dzX 3oRa9fn64WdYw6ppRole0vuPMOttapirZjZBXVaBFLuqoyOJQBv4bNk0X7Hsy1NjAl5TH1L L+R0yrJy7yKmmQCM+TGgbpY+eoozr28uwNXcpsxuSvDXMiPIN0ngy5QS7VA0FKbn7OS2P+L gMT2/E2uWL+2/wYvgvy5Q== X-UI-Out-Filterresults: notjunk:1;V01:K0:aJHH4D/xKdI=:B4Iqetja9KHPZnyRaG7aDw YZwSMc+dC2qiIYHB7tMOQqjEJaiB6MvoI9bH5h8NdpLC9JAReCXJ8c9IgIOQgfjE8+9YRyuWP UxKyGNDFocn93Tri1zTOd5+keq53yYmHCGUc1m/snTJDLu0FT6PvBi84LiDxbUpbHVd/P/urQ OR4OAJNqHwgY+0MPXCoRkcR41IroCnWRxKA7/g6FdUvsivQDMG6G80jD1MP5ZryLuAfD8z7k8 ExqHv7sdSGhtpG7g1FyH1IiUQavSu8ekneqvaCkIVu13Uj+SBmRnnQP3mvJ2JY+/Rodh0Vh8Q WqLSl3pe0WJX8SaafnKqVdyO0w94Fy00sGVnh0TE40tya07kFeCoBfDzqrOas6ujqcZJ5HKQL Sscanpo39P/4oaqT1uA26r16UvAsLJd+lxmjEYpVyHo51f1d0+kOEIMicxv/1wxynZJMVU//p 1hfAK9gFRtIVvMlXXOXXzkUh/rNdj7KHcCR789Yfvl6LrZUiS4ychk3c1caZAsN+XD7MtKap4 mke/wzMeijJuAonHdqUNIzSrz7gOqEBEesQ6nf5hc8blxENJoJ+iI1MyE58eorFmVgwcOhan6 uFzWYpJ/btzxVhjq5rWSSlqcT9npww1uLc9N8hEGEyKwbK+gOdR2z+pwuGE8gzB2XYcSFcaqR OqOKO4pCCO1aBcl07+368awaVjO7zR0FEbrjykIKdyN6xXg3W+xIw5JGqe1FwvjIlqks= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.24 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:202784 Archived-At: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --sHCRVXMeHiC4QWgIAxNt1pXuv3NMIAqOI Content-Type: multipart/mixed; boundary="3S6GdHgt2AcMwmqljj3D6Dtc5IgXGvJxh" From: =?UTF-8?Q?Cl=c3=a9ment_Pit--Claudel?= To: Lars Magne Ingebrigtsen Cc: Emacs developers Message-ID: <57063E60.4040408@gmail.com> Subject: Re: Help understanding the URL code References: <5703DBB2.1030903@gmail.com> In-Reply-To: --3S6GdHgt2AcMwmqljj3D6Dtc5IgXGvJxh Content-Type: multipart/mixed; boundary="------------040003010600020108010509" This is a multi-part message in MIME format. --------------040003010600020108010509 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Lars, On 04/06/2016 12:34 PM, Lars Magne Ingebrigtsen wrote: > Your patch looks good Thanks for the review! > but perhaps this part could be changed a bit: > (...) > It should be possible to not have a User-Agent header at all. nil as a= > value to `url-user-agent' would be the obvious choice, and `default' > could be the value to signal using `url-http--user-agent-default-string= '. You're right; it doesn't hurt to have that option locally too, in additio= n to tweaking the privacy level. Please find an updated patch attached :) (I would also welcome comments a= bout the commit message format; should I link to this discussion?) Cheers, Cl=E9ment. --------------040003010600020108010509 Content-Type: text/x-diff; name="0001-Take-Emacs-and-package-versions-into-account-in-URL-.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename*0="0001-Take-Emacs-and-package-versions-into-account-in-URL-.pa"; filename*1="tch" =46rom 7552bcd037f5f2b8ce080be73adc5209e4a4fb28 Mon Sep 17 00:00:00 2001 From: =3D?UTF-8?q?Cl=3DC3=3DA9ment=3D20Pit--Claudel?=3D Date: Thu, 7 Apr 2016 11:31:13 +0100 Subject: [PATCH] Take Emacs and package versions into account in URL's User-Agent string * url-vars.el (url-privacy-level): Allow `emacs' in list of information not to send. (url-user-agent): Add nil and `default' options; do not pre-compute value. * url-http.el (url-http-user-agent-string): Compute User-Agent string dynamically. (url-http--user-agent-default-string): New function. The original code took `url-package-name' and `url-package-version' into account only when url-vars.el was loaded; the new code takes them into account in all cases, allowing users to let-bind them. It also adds the current Emacs version to the User-Agent string. --- lisp/url/url-http.el | 35 ++++++++++++++++++++++++++++------- lisp/url/url-vars.el | 25 ++++++++++++++++--------- 2 files changed, 44 insertions(+), 16 deletions(-) diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el index 5832e92..4bdd3b5 100644 --- a/lisp/url/url-http.el +++ b/lisp/url/url-http.el @@ -211,15 +211,36 @@ url-http-find-free-connection (if connection (url-http-mark-connection-as-busy host port connection)))) =20 +(defun url-http--user-agent-default-string () + "Compute a default User-Agent string based on `url-privacy-level'." + (let ((package-info (when url-package-name + (format "%s/%s" url-package-name url-package-ver= sion))) + (emacs-info (unless (and (listp url-privacy-level) + (memq 'emacs url-privacy-level)) + (format "GNUEmacs/%s" emacs-version))) + (os-info (unless (and (listp url-privacy-level) + (memq 'os url-privacy-level)) + (format "(%s; %s)" url-system-type url-os-type))) + (url-info (format "URL/%s" url-version))) + (string-join (delq nil (list package-info url-info + emacs-info os-info)) + " "))) + ;; Building an HTTP request (defun url-http-user-agent-string () - (if (or (eq url-privacy-level 'paranoid) - (and (listp url-privacy-level) - (memq 'agent url-privacy-level))) - "" - (if (functionp url-user-agent) - (funcall url-user-agent) - url-user-agent))) + "Compute a User-Agent string. +The string is based on `url-privacy-level' and `url-user-agent'." + (let* ((hide-ua + (or (eq url-privacy-level 'paranoid) + (and (listp url-privacy-level) + (memq 'agent url-privacy-level)))) + (ua-string + (and (not hide-ua) + (cond + ((functionp url-user-agent) (funcall url-user-agent)) + ((stringp url-user-agent) url-user-agent) + ((eq url-user-agent 'default) (url-http--user-agent-defa= ult-string)))))) + (if ua-string (format "User-Agent: %s\r\n" (string-trim ua-string)) = ""))) =20 (defun url-http-create-request (&optional ref-url) "Create an HTTP request for `url-http-target-url', referred to by REF-= URL." diff --git a/lisp/url/url-vars.el b/lisp/url/url-vars.el index 960a04a..97dac9c 100644 --- a/lisp/url/url-vars.el +++ b/lisp/url/url-vars.el @@ -116,6 +116,7 @@ url-privacy-level Valid symbols are: email -- the email address os -- the operating system info +emacs -- the version of Emacs lastloc -- the last location agent -- do not send the User-Agent string cookies -- never accept HTTP cookies @@ -143,6 +144,7 @@ url-privacy-level (checklist :tag "Custom" (const :tag "Email address" :value email) (const :tag "Operating system" :value os) + (const :tag "Emacs version" :value emacs) (const :tag "Last location" :value lastloc) (const :tag "Browser identification" :value agent) (const :tag "No cookies" :value cookie))) @@ -357,15 +359,20 @@ url-gateway-method (const :tag "Direct connection" :value native)) :group 'url-hairy) =20 -(defcustom url-user-agent (format "User-Agent: %sURL/%s\r\n" - (if url-package-name - (concat url-package-name "/" - url-package-version " ") - "") url-version) - "User Agent used by the URL package for HTTP/HTTPS requests -Should be a string or a function of no arguments returning a string." - :type '(choice (string :tag "A static User-Agent string") - (function :tag "Call a function to get the User-Agent s= tring")) +(defcustom url-user-agent 'default + "User Agent used by the URL package for HTTP/HTTPS requests. +Should be one of: +* A string (not including the \"User-Agent:\" prefix) +* A function of no arguments, returning a string +* `default' (to compute a value according to `url-privacy-level') +* nil (to omit the User-Agent header entirely)" + :type + '(choice + (string :tag "A static User-Agent string") + (function :tag "Call a function to get the User-Agent string") + (const :tag "No User-Agent at all" :value nil) + (const :tag "An string auto-generated according to `url-privacy-leve= l'" + :value default)) :version "25.1" :group 'url) =20 --=20 2.8.1 --------------040003010600020108010509-- --3S6GdHgt2AcMwmqljj3D6Dtc5IgXGvJxh-- --sHCRVXMeHiC4QWgIAxNt1pXuv3NMIAqOI Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJXBj5gAAoJEPqg+cTm90wjQtwP/iVMJcUVzHotJvNPSCgqhpEI /+3pfMtQTleI0CH4VVktD7fl7eVhDVT5Gz4j7NVagY/ybg9RFFNeN2p+MXzo83ks UIvHUiNw31RH3fmYV2AUpwLJls76HCcBWHV7V6oxN8kcAuyfYAf8Orkw845FGg++ 9kzhuRIsl7V8LDlNpgK4ohpQh+M5v5biEkJAhIsw4EMo1sq7PVKWjoXP+vd4h2tv l+ih3AAJv+PFwBS/iLZcy5Ovwjp/vA6+ne6r5aZfde3OLu30z7xrRjYhRh2DC2wL lIGqBqPMz2eCoOibx/pP24Dv+ried3xqwT4MI6s7ArZ5bQEhbvFvYLkb+utt5MKR TfufARp94Cgl9ht7YcnLc96+4LdyhNIE2kL1FPssjhDys94tInOMG1EtOMcOyu4s A9SSnbM4t3f+QDILO+4BSFg/1KMlOHUnEZVCpTYbwmr10BzMfch3vk4bb7jFcwlW E6YjAi1m3SqoE+DMA8MRsV6eXjryaw3n6lJXIKHw8FQUYuqFhQK1dSODVV9xy158 WT4mt4fV+NUp698z5PNeDeySpLqyDFRiRmCYOYBYPS9e0j1LABMJXy4OyIXERvXs 5MuIHwPIfPPOSfBFjVYJt0DkX6VFFoyl8aGeTnqrQzUIpNZQzBAfVzX6M91UaXgP iUkKB/I+WAYfbsuy9ZcB =dkGF -----END PGP SIGNATURE----- --sHCRVXMeHiC4QWgIAxNt1pXuv3NMIAqOI--