bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Winston Weinert]

I noticed EWW sends header "Accept: */*", which appears to prevent
HTTP servers that provide multiple content types from picking a sensible
response content type.

More information about the Accept header with a list of RFCs at the
bottom of the page can be found here [1]. For comparison lynx sends a
more specific header, which indicates It'd prefer common web content
types before anything else [2]. I experienced this particular bug when
browsing [3], which uses the Accept header to determine when to show the
built-in API Explorer. In a web browser it shows a HTML form to interact
with the API, however actual API clients use "Accept: application/json"
or similar to get the desired format.


Regards,
Winston Weinert


[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept
[2]: "Accept: text/html, text/plain, text/sgml, text/css, 
application/xhtml+xml, */*;q=0.01"
[3]: https://restframework.herokuapp.com/


In GNU Emacs 26.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll 
bars)
  of 2018-09-08 built on snowcrash
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description:	Gentoo Base System release 2.4.1

Recent messages:
C-c C-g is undefined
Quit
Contacting host: duckduckgo.com:443
ImageMagick error: insufficient image data in file `/tmp/foo.ico' @ 
error/icon.c/ReadICONImage/375
Contacting host: localhost:8000
Making completion list... [3 times]
Note: file is write protected
Mark saved where search started
Quit
delete-backward-char: Text is read-only

Configured using:
  'configure --prefix=/usr --build=x86_64-pc-linux-gnu
  --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
  --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
  --localstatedir=/var/lib --disable-silent-rules
  --docdir=/usr/share/doc/emacs-26.1-r3
  --htmldir=/usr/share/doc/emacs-26.1-r3/html --libdir=/usr/lib64
  --program-suffix=-emacs-26 --infodir=/usr/share/info/emacs-26
  --localstatedir=/var
  --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
  --without-compress-install --without-hesiod --without-pop
  --with-file-notification=inotify --enable-acl --with-dbus
  --without-modules --without-gameuser --without-gpm --without-kerberos
  --without-kerberos5 --with-lcms2 --with-xml2 --without-mailutils
  --without-selinux --with-gnutls --without-libsystemd --with-threads
  --without-wide-int --with-zlib --with-sound=alsa --with-x --without-ns
  --without-gconf --without-gsettings --with-toolkit-scroll-bars
  --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xpm
  --with-imagemagick --with-xft --without-cairo --without-libotf
  --without-m17n-flt --with-x-toolkit=lucid --with-xaw3d 'CFLAGS=-O2
  -pipe -march=bdver2' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS NOTIFY ACL
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 THREADS
LCMS2

Important settings:
   value of $LANG: en_US.utf8
   locale-coding-system: utf-8-unix

Major mode: Outline

Minor modes in effect:
   TeX-PDF-mode: t
   global-magit-file-mode: t
   diff-auto-refine-mode: t
   magit-auto-revert-mode: t
   global-git-commit-mode: t
   async-bytecomp-package-mode: t
   override-global-mode: t
   winum-mode: t
   shell-dirtrack-mode: t
   recentf-mode: t
   global-edit-server-edit-mode: t
   mode-line-bell-mode: t
   editorconfig-mode: t
   electric-pair-mode: t
   image-diredx-async-mode: t
   show-paren-mode: t
   cl-old-struct-compat-mode: t
   tooltip-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   buffer-read-only: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim hides 
~/.emacs.d/custom/emacs-eclim/eclim
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-java-run hides 
~/.emacs.d/custom/emacs-eclim/eclim-java-run
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-macros hides 
~/.emacs.d/custom/emacs-eclim/eclim-macros
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-debug hides 
~/.emacs.d/custom/emacs-eclim/eclim-debug
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-completion hides 
~/.emacs.d/custom/emacs-eclim/eclim-completion
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-common hides 
~/.emacs.d/custom/emacs-eclim/eclim-common
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-problems hides 
~/.emacs.d/custom/emacs-eclim/eclim-problems
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-maven hides 
~/.emacs.d/custom/emacs-eclim/eclim-maven
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclimd hides 
~/.emacs.d/custom/emacs-eclim/eclimd
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-ant hides 
~/.emacs.d/custom/emacs-eclim/eclim-ant
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-pkg hides 
~/.emacs.d/custom/emacs-eclim/eclim-pkg
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-project hides 
~/.emacs.d/custom/emacs-eclim/eclim-project
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-scala hides 
~/.emacs.d/custom/emacs-eclim/eclim-scala
/home/winston/.emacs.d/elpa/eclim-20180928.523/eclim-java hides 
~/.emacs.d/custom/emacs-eclim/eclim-java
~/.emacs.d/custom/emacs-eclim/company-emacs-eclim hides 
/home/winston/.emacs.d/elpa/company-emacs-eclim-20180911.1121/company-emacs-eclim
~/code/cyberpunk-theme.el/cyberpunk-theme hides 
/home/winston/.emacs.d/elpa/cyberpunk-theme-20180609.509/cyberpunk-theme

Features:
(shadow sort mail-extr emacsbug rng-xsd xsd-regexp rng-cmpct rng-nxml
rng-valid nxml-mode nxml-outln nxml-rap sgml-mode tabify man scheme
yaml-mode term ehelp sml-mode conf-mode gist gh-gist gh-oauth gh-api
logito gh-cache pcache gh-auth gh-url gh-profile timezone eieio-base
cus-edit url-file url-dired eww mm-url url-queue helm-dash helm
helm-source helm-multi-match helm-lib eclim-scala eclim-debug eclimd gud
eclim-java-run eclim-ant eclim-maven eclim-project eclim-java
eclim-problems yasnippet eclim-common popup arc-mode archive-mode
eclim-macros package-recipe-mode package-build-badges package-build
package-recipe skeleton ibuf-macs grep ibuf-ext ibuffer ibuffer-loaddefs
racket-unicode-input-method quail racket-debug racket-mode
racket-bug-report racket-collection racket-stepper racket-logger
racket-profile racket-imenu racket-edit racket-complete shr svg dom
racket-repl racket-common racket-indent racket-font-lock racket-ppss
racket-keywords-and-builtins tq racket-util racket-custom sh-script smie
executable tar-mode autoload radix-tree lisp-mnt mm-archive
network-stream starttls url-cache tmm haskell-doc inf-haskell
haskell-decl-scan haskell haskell-completions haskell-load
haskell-commands highlight-uses-mode haskell-modules haskell-sandbox
haskell-navigate-imports haskell-repl haskell-svg haskell-collapse
hideshow haskell-debug haskell-interactive-mode
haskell-presentation-mode haskell-compile haskell-hoogle haskell-process
haskell-session vc-git haskell-mode haskell-cabal haskell-utils
haskell-font-lock haskell-indentation haskell-string
haskell-sort-imports haskell-lexeme haskell-align-imports haskell-compat
haskell-complete-module haskell-ghc-support etags xref project dabbrev
haskell-customize ssh-config-mode misearch multi-isearch preview
prv-emacs tex-buf font-latex latex tex-ispell tex-style tex dbus xml
tex-mode latexenc org-table ispell org-rmail org-mhe org-irc org-info
org-gnus nnir gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
gnus-win org-docview org-bibtex bibtex org-bbdb org-w3m linum whitespace
editorconfig-core editorconfig-core-handle editorconfig-fnmatch
smart-mode-line-dark-theme smart-mode-line rich-minority cyberpunk-theme
magithub magithub-dash magithub-notification magithub-orgs
magithub-issue-tricks magithub-issue-post magithub-edit-mode
magithub-repo magithub-ci magithub-issue magithub-label magithub-user
magithub-core magithub-faces magithub-settings parse-time markdown-mode
color bug-reference ghub+ apiwrap apropos s magit-bookmark
magit-obsolete magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-collab ghub-graphql treepy
graphql pcase ghub url-http tls gnutls url-gw nsm url-auth url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap let-alist json map magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
diff-mode magit-core magit-autorevert autorevert filenotify
magit-process magit-margin magit-mode git-commit magit-git magit-section
magit-utils crm subr-x magit-popup log-edit pcvs-util add-log
with-editor async-bytecomp async server use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
use-package-core cl-extra winum dash python-mode which-func imenu shell
hippie-exp flymake-proc flymake warnings compile cc-cmds cc-engine
cc-vars cc-defs web-mode disp-table dashboard dashboard-widgets recentf
tree-widget page-break-lines org-agenda bookmark pp smex edit-server
keychain-environment help-mode dired+ image-file dired-x dired-aux
buffer-move windmove mode-line-bell editorconfig ox-twbs ox-odt rng-loc
rng-uri rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox profiler org-element avl-tree generator org org-macro
org-footnote org-pcomplete pcomplete org-list org-faces org-entities
noutline outline easy-mmode org-version ob-emacs-lisp ob ob-tangle
org-src ob-ref ob-lob ob-table ob-keys ob-exp ob-comint ob-core ob-eval
org-compat org-macs org-loaddefs find-func elec-pair irfc scribble
finder-inf edmacro kmacro gh-common marshal eieio-compat image-dired+
image-dired rx pydoc-info advice info-look info package url-handlers
url-parse url-vars wheatgrass-theme paren gnus nnheader wid-edit
cus-start cus-load site-gentoo mu4e desktop frameset mu4e-speedbar
speedbar sb-image ezimage dframe mu4e-main mu4e-context mu4e-view
cal-menu calendar cal-loaddefs thingatpt browse-url comint ansi-color
ring mu4e-headers mu4e-compose mu4e-draft mu4e-actions ido rfc2368
smtpmail auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
sendmail mu4e-mark mu4e-message html2text mu4e-proc mu4e-utils doc-view
jka-compr image-mode mu4e-lists mu4e-vars message rmc puny seq byte-opt
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader hl-line cl gv cl-loaddefs cl-lib
mu4e-meta preview-latex auto-loads tex-site mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting font-render-setting
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 2584773 148802)
  (symbols 48 78658 1)
  (miscs 40 5277 2890)
  (strings 32 395278 13798)
  (string-bytes 1 15855316)
  (vectors 16 114301)
  (vector-slots 8 2574679 85614)
  (floats 8 656 957)
  (intervals 56 273185 3106)
  (buffers 992 129))

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Federico Tedin]

> I noticed EWW sends header "Accept: */*", which appears to prevent
> HTTP servers that provide multiple content types from picking a sensible
> response content type.
>
> More information about the Accept header with a list of RFCs at the
> bottom of the page can be found here [1]. For comparison lynx sends a
> more specific header, which indicates It'd prefer common web content
> types before anything else [2]. I experienced this particular bug when
> browsing [3], which uses the Accept header to determine when to show the
> built-in API Explorer. In a web browser it shows a HTML form to interact
> with the API, however actual API clients use "Accept: application/json"
> or similar to get the desired format.

I've been looking into this problem. It seems that EWW uses
url-retrieve to retrieve HTTP contents, which then calls url-http.  In
url-http, the variable url-mime-accept-string is used as the value for
the HTTP Accept header (if it is set to nil, "*/*" is used
instead). By default, url-mime-accept-string is set to nil.

We could temporarily set the variable url-mime-accept-string with
another value when url-retrieve is called by EWW. I am not sure what
the default value should be, or if it should be configurable. Maybe
the same value used by Lynx would be a good starting point.

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Federico Tedin]

[-- Attachment #1: Type: text/plain, Size: 204 bytes --]

I've created a small patch which introduces a new variable, its value is
used as the value for the 'Accept' HTTP header in requests created by EWW.
As the default value I chose to use the same Lynx uses.

[-- Attachment #2: eww.patch --]
[-- Type: text/x-patch, Size: 2504 bytes --]

From b2fe312aa5232fa3cfc259feb3cf313ae85faa9b Mon Sep 17 00:00:00 2001
From: Federico Tedin <federicotedin@gmail.com>
Date: Sun, 4 Nov 2018 00:07:52 -0300
Subject: [PATCH 1/1] Adds variable 'eww-accept-content-types'

* lisp/net/eww.el (eww-accept-content-types): New variable.
  (eww): Use it (Bug#33002).
  (eww-reload): Use it.
* etc/NEWS: Mention 'eww-accept-content-types'.
---
 etc/NEWS        |  4 ++++
 lisp/net/eww.el | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 1020a2a0ea..f5203c6b11 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -511,6 +511,10 @@ and its value has been changed to Duck Duck Go.
 'shr-selected-link' face to give the user feedback that the command
 has been executed.
 
+*** New customizable variable 'eww-accept-content-types'
+This new variable controls the value of the 'Accept' header sent in
+HTTP requests.
+
 +++
 *** New option 'shr-discard-aria-hidden'.
 If set, shr will not render tags with attribute 'aria-hidden="true"'.
diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index 64cc1a51f6..f5a877f112 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -128,6 +128,13 @@ eww-use-external-browser-for-content-type
   :type '(choice (const :tag "Never" nil)
                  regexp))
 
+(defcustom eww-accept-content-types
+  "text/html, text/plain, text/sgml, text/css, application/xhtml+xml, */*;q=0.01"
+  "Value used for the HTTP Accept header."
+  :version "27.1"
+  :group 'eww
+  :type 'string)
+
 (defcustom eww-after-render-hook nil
   "A hook called after eww has finished rendering the buffer."
   :version "25.1"
@@ -282,8 +289,9 @@ eww
   (let ((inhibit-read-only t))
     (insert (format "Loading %s..." url))
     (goto-char (point-min)))
-  (url-retrieve url 'eww-render
-                (list url nil (current-buffer))))
+  (let ((url-mime-accept-string eww-accept-content-types))
+    (url-retrieve url 'eww-render
+                  (list url nil (current-buffer)))))
 
 (defun eww--dwim-expand-url (url)
   (setq url (string-trim url))
@@ -944,8 +952,9 @@ eww-reload
 	    (error "No current HTML data")
 	  (eww-display-html 'utf-8 url (plist-get eww-data :dom)
 			    (point) (current-buffer)))
-      (url-retrieve url 'eww-render
-		    (list url (point) (current-buffer) encode)))))
+      (let ((url-mime-accept-string eww-accept-content-types))
+        (url-retrieve url 'eww-render
+		      (list url (point) (current-buffer) encode))))))
 
 ;; Form support.
 
-- 
2.17.1

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Eli Zaretskii]

> From: Federico Tedin <federicotedin@gmail.com>
> Date: Sun, 4 Nov 2018 00:15:38 -0300
> Cc: 33002@debbugs.gnu.org
> 
> I've created a small patch which introduces a new variable, its value is
> used as the value for the 'Accept' HTTP header in requests created by EWW.
> As the default value I chose to use the same Lynx uses.

Does it really make sense to make this a customizable option?  How
would an average user choose a value for this option?

I think EWW should decide by itself what header to send.

Thanks.

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Federico Tedin]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

> Does it really make sense to make this a customizable option?  How
> would an average user choose a value for this option?
>
> I think EWW should decide by itself what header to send.

I agree, it is not something most people will want to change. If they
really want to change it though, they can just do (setq
eww-accept-content-types ...). I've changed the patch to use defvar
instead of defcustom.

[-- Attachment #2: eww.patch --]
[-- Type: text/x-patch, Size: 1858 bytes --]

From e6730701f320cf65ab02c8d550a442d3e4ea3c25 Mon Sep 17 00:00:00 2001
From: Federico Tedin <federicotedin@gmail.com>
Date: Sun, 4 Nov 2018 15:24:36 -0300
Subject: [PATCH 1/1] Adds variable 'eww-accept-content-types'

* lisp/net/eww.el (eww-accept-content-types): New variable.
  (eww): Use it (Bug#33002).
  (eww-reload): Use it.
---
 lisp/net/eww.el | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index 64cc1a51f6..0c3fe8f7b3 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -223,6 +223,10 @@ eww-prompt-history
 (defvar eww-local-regex "localhost"
   "When this regex is found in the URL, it's not a keyword but an address.")
 
+(defvar eww-accept-content-types
+  "text/html, text/plain, text/sgml, text/css, application/xhtml+xml, */*;q=0.01"
+  "Value used for the HTTP 'Accept' header.")
+
 (defvar eww-link-keymap
   (let ((map (copy-keymap shr-map)))
     (define-key map "\r" 'eww-follow-link)
@@ -282,8 +286,9 @@ eww
   (let ((inhibit-read-only t))
     (insert (format "Loading %s..." url))
     (goto-char (point-min)))
-  (url-retrieve url 'eww-render
-                (list url nil (current-buffer))))
+  (let ((url-mime-accept-string eww-accept-content-types))
+    (url-retrieve url 'eww-render
+                  (list url nil (current-buffer)))))
 
 (defun eww--dwim-expand-url (url)
   (setq url (string-trim url))
@@ -944,8 +949,9 @@ eww-reload
 	    (error "No current HTML data")
 	  (eww-display-html 'utf-8 url (plist-get eww-data :dom)
 			    (point) (current-buffer)))
-      (url-retrieve url 'eww-render
-		    (list url (point) (current-buffer) encode)))))
+      (let ((url-mime-accept-string eww-accept-content-types))
+        (url-retrieve url 'eww-render
+		      (list url (point) (current-buffer) encode))))))
 
 ;; Form support.
 
-- 
2.17.1

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Federico Tedin]

Eli, sorry to bother you. Just wanted to know if the last patch I sent
is OK or if it needs any further changes.

Thanks.

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Eli Zaretskii]

> From: Federico Tedin <federicotedin@gmail.com>
> Date: Tue, 13 Nov 2018 18:26:53 -0300
> Cc: John Smith <winston@ml1.net>, 33002@debbugs.gnu.org
> 
> Eli, sorry to bother you. Just wanted to know if the last patch I sent
> is OK or if it needs any further changes.

I still don't think that a variable is the right solution for this.
Perhaps Lars (CC'ed) or someone else has an opinion or a better
proposal?

bug#33002: 26.1; EWW sends unspecific "Accept: */*" header

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Federico Tedin <federicotedin@gmail.com>
>> Date: Tue, 13 Nov 2018 18:26:53 -0300
>> Cc: John Smith <winston@ml1.net>, 33002@debbugs.gnu.org
>> 
>> Eli, sorry to bother you. Just wanted to know if the last patch I sent
>> is OK or if it needs any further changes.
>
> I still don't think that a variable is the right solution for this.
> Perhaps Lars (CC'ed) or someone else has an opinion or a better
> proposal?

I think it makes sense to use a variable here -- if for no other reason
that the string is used two different places in the eww source code, and
avoiding repeating that long string is nice.  But leaving it
undocumented (in the manual) is fine, I think, as I don't see anybody
ever wanting to change it.

So I've applied Federico's patch to Emacs 27.1.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no