bug#18812: 24.4; url.el user agent does not include Emacs version

bug#18812: 24.4; url.el user agent does not include Emacs version

[Phil Hagelberg]

The url.el functions for making HTTP requests set the User-Agent header
to "URL/Emacs" which is not very descriptive. It would be helpful to
include the Emacs version, at least.



In GNU Emacs 24.4.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.10)
 of 2014-10-20 on enigma
Windowing system distributor `The X.Org Foundation', version 11.0.11204000
System Description:	Debian GNU/Linux 7.7 (wheezy)

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  magit-auto-revert-mode: t
  diff-auto-refine-mode: t
  recentf-mode: t
  hi-lock-mode: t
  shell-dirtrack-mode: t
  elisp-slime-nav-mode: t
  paredit-mode: t
  whitespace-mode: t
  idle-highlight-mode: t
  hl-line-mode: t
  page-break-lines-mode: t
  eldoc-mode: t
  winner-mode: t
  show-paren-mode: t
  ido-hacks-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
F C-x C-f r e a <return> M-< C-x ' C-s m a r m a l 
M-< C-v C-l C-n C-e C-b C-b C-b C-SPC M-b M-b M-b M-b 
M-b M-b M-b M-b M-w C-v C-v C-x C-f <backspace> <backspace> 
. e m <return> e l <return> r e c <return> c i d <return> 
y C-x k <return> C-n C-n C-x k <return> C-x k <return> 
C-x b C-n <return> C-x C-f C-g C-c C-c C-x C-g C-n 
TAB TAB k y F - r F P P C-x k <return> C-x b C-g C-x 
o C-c g C-x k <return> C-h v u s e r - a g e C-g C-h 
f u r l - r e t <return> C-x ' C-e C-x b h e l p <return> 
C-e M-b <return> C-x ' M-< C-s u s e r - a g e n t 
C-a C-s u r l / e m a C-a M-x u r l - r e t C-g C-x 
b C-g C-x b s c a <backspace> r a <return> ( u r l 
- r e t r M-/ - s M-/ SPC " h t t p : / / h t t p b 
i n . o r g / h e a d e r s C-e C-x C-e C-x b h t t 
p b i <return> C-x b C-n <return> M-< C-s U R L / <backspace> 
C-s C-s C-s C-s C-s C-s C-s C-s C-s C-s C-a C-s a g 
e n t C-s C-a C-x b <return> C-x b h t t p b <return> 
C-x b <return> C-x b <return> C-s U s e r C-a C-v C-v 
M-x r e p o r t - e <return>

Recent messages:
Quit
Type "q" in help window to restore its previous buffer, C-M-v to scroll help.
Mark set
Mark saved where search started [2 times]
Quit [2 times]
Using try-expand-dabbrev-all-buffers [2 times]
Contacting host: httpbin.org:80
#<buffer  *http httpbin.org:80*>
Mark set
Mark saved where search started [3 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail url-http url-gw url-cache
url-auth eieio-opt speedbar sb-image ezimage dframe smerge-mode
yaml-mode sgml-mode texinfo gitignore-mode conf-mode pcmpl-unix flyspell
ispell hippie-exp mule-util magit-key-mode magit view grep epa epg
diff-mode autorevert filenotify git-rebase-mode git-commit-mode server
log-edit message rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log markdown-mode noutline outline
recentf tree-widget wid-edit network-stream starttls tls cider
cider-mode cider-repl cider-eldoc pkg-info find-func epl hi-lock
misearch multi-isearch vc-git clojure-test-mode rx cider-interaction
compile arc-mode archive-mode cider-stacktrace easy-mmode cider-client
nrepl-client cider-util ewoc dash which-func clojure-mode imenu inf-lisp
tramp tramp-compat tramp-loaddefs trampver shell format-spec
elisp-slime-nav etags paredit whitespace idle-highlight-mode thingatpt
hl-line page-break-lines em-unix em-term term disp-table ehelp em-script
em-prompt em-ls em-hist em-pred em-glob em-dirs em-cmpl em-basic
em-banner em-alias pcomplete comint ansi-color esh-var esh-io esh-cmd
esh-opt esh-ext esh-proc esh-arg eldoc esh-groups eshell esh-module
esh-mode esh-util winner ring scpaste htmlize url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source eieio byte-opt eieio-core gnus-util mm-util
mail-prsvr password-cache url-vars mailcap find-file-in-project
better-defaults paren edmacro kmacro saveplace ido-hacks advice smex ido
magit-autoloads .loaddefs el-get el-get-autoloads el-get-list-packages
el-get-dependencies el-get-build el-get-status pp el-get-methods
el-get-fossil el-get-svn el-get-pacman el-get-github-zip
el-get-github-tar el-get-http-zip el-get-http-tar el-get-hg el-get-go
el-get-git-svn el-get-fink el-get-emacswiki el-get-http el-get-notify
help-mode easymenu el-get-emacsmirror el-get-github el-get-git
el-get-elpa package epg-config el-get-darcs el-get-cvs el-get-bzr
el-get-brew el-get-builtin el-get-apt-get el-get-recipes derived
el-get-byte-compile el-get-custom el-get-core cl-macs autoload help-fns
lisp-mnt bytecomp byte-compile cconv dired cl gv cl-loaddefs cl-lib
time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind gfilenotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

Memory information:
((conses 16 289204 29504)
 (symbols 48 34079 0)
 (miscs 40 2049 2541)
 (strings 32 71001 6561)
 (string-bytes 1 1923368)
 (vectors 16 35186)
 (vector-slots 8 1476900 204832)
 (floats 8 265 689)
 (intervals 56 5110 54)
 (buffers 960 84)
 (heap 1024 53550 2663))

bug#18812: 24.4; url.el user agent does not include Emacs version

[Stefan Monnier]

> The url.el functions for making HTTP requests set the User-Agent header
> to "URL/Emacs" which is not very descriptive.  It would be helpful to
> include the Emacs version, at least.

Other people feel like it's already too descriptive.  Why should all
Emacs users constantly tell the whole world exactly which Emacs version
they're running?


        Stefan

bug#18812: 24.4; url.el user agent does not include Emacs version

[Ted Zlatanov]

On Thu, 23 Oct 2014 21:19:21 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The url.el functions for making HTTP requests set the User-Agent header
>> to "URL/Emacs" which is not very descriptive.  It would be helpful to
>> include the Emacs version, at least.

SM> Other people feel like it's already too descriptive.  Why should all
SM> Emacs users constantly tell the whole world exactly which Emacs version
SM> they're running?

Web browsers do.  It helps web developers and is not particularly
sensitive if only the major.minor version is used (without too much detail).

Ted

bug#18812: 24.4; url.el user agent does not include Emacs version

[Stefan Monnier]

> Web browsers do.

I know, and I don't like it either.

> It helps web developers

I think it only helps them in the sense "helps to dig themselves
further into a hole, instead of backing up towards the light" ;-)

> and is not particularly sensitive if only the major.minor version is
> used (without too much detail).

It wouldn't be the end of the world, indeed.
But I still haven't heard of any concrete reason why it would be
beneficial.  The only argument I've seen so far amounts to a handwavy
"would be helpful", which I find rather unconvincing.


        Stefan

bug#18812: 24.4; url.el user agent does not include Emacs version

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But I still haven't heard of any concrete reason why it would be
> beneficial.  The only argument I've seen so far amounts to a handwavy
> "would be helpful", which I find rather unconvincing.

Personally, I am curious about the new signing features in
package.el. However, since these don't work in versions of Emacs older
than 24.4, I thought it would be interesting to examine the access logs
of Marmalade to see how fast the adoption of 24.4 is going.

-Phil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

bug#18812: 24.4; url.el user agent does not include Emacs version

[Stefan Monnier]

> Personally, I am curious about the new signing features in
> package.el. However, since these don't work in versions of Emacs older
> than 24.4, I thought it would be interesting to examine the access logs
> of Marmalade to see how fast the adoption of 24.4 is going.

You don't need the version number for that: just check (failed?)
accesses to the *.sig files, since Emacs-24.4 will automatically look
for them by default.

The advantage is that this will also account for the 24.4 users who
don't have GPG installed (since for those, package.el won't look for
.sig files).


        Stefan

bug#18812: 24.4; url.el user agent does not include Emacs version

[Ivan Shmakov]

>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:
>>>>> On Thu, 23 Oct 2014 21:19:21 -0400 Stefan Monnier wrote:

 PH> The url.el functions for making HTTP requests set the User-Agent
 PH> header to "URL/Emacs" which is not very descriptive.  It would be
 PH> helpful to include the Emacs version, at least.

 SM> Other people feel like it's already too descriptive.  Why should
 SM> all Emacs users constantly tell the whole world exactly which Emacs
 SM> version they're running?

 TZ> Web browsers do.  It helps web developers and is not particularly
 TZ> sensitive if only the major.minor version is used (without too much
 TZ> detail).

	Web browsers (and HTTP/HTTPS clients in general) also provide a
	way for the user to override the default User-Agent: value, and
	I believe that Emacs should allow for that, too.

	While somewhat tangential to the problem as reported, some of
	the issues I’ve seen with the default User-Agent: settings are:

	• Lynx’ default User-Agent: includes ‘libwww’ as a substring;
	  apparently, some Web sites mistake that for the well-known
	  Perl library of the same name, and thus reject the requests;
	  the solution is to either manually remove the respective
	  stanza, or to disable User-Agent: altogether;

	• other sites are known to disallow HTTP requests /without/
	  User-Agent:, however;

	• GNU Wget default identification is also not unknown to be
	  rejected; per my experience, using something like “tegW/1.15”
	  instead (via --user-agent=) tends to resolve the issue;

	• moreover, some sites actually serve different content
	  depending on User-Agent:, – apparently for “SEO” purposes; for
	  instance, a PDF file is served for a request from a party
	  identifying itself as a search engine, while regular browsers
	  get a Web page with a (presumably non-free) JavaScript-based
	  PDF reader set up for that file instead.

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

bug#18812: 24.4; url.el user agent does not include Emacs version

[Phil Hagelberg]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> You don't need the version number for that: just check (failed?)
> accesses to the *.sig files, since Emacs-24.4 will automatically look
> for them by default.

I can imagine other sites that would want to know this information, but
that works for me.

-Phil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

bug#18812: 24.4; url.el user agent does not include Emacs version

[Lars Ingebrigtsen]

Phil Hagelberg <phil@hagelb.org> writes:

> The url.el functions for making HTTP requests set the User-Agent header
> to "URL/Emacs" which is not very descriptive. It would be helpful to
> include the Emacs version, at least.

You can now customise this with the `url-user-agent' variable.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no