bug#22855: 24.5; `url-mime-accept-string' not set in asynchronous call

bug#22855: 24.5; `url-mime-accept-string' not set in asynchronous call

[Bjarte Johansen]

Hi,

Sending a http request asynchronously through url-retrieve I am not
(always) able to dynamically bind url-mime-accept-string so I can set
the Accept header. Setting url-request-extra-headers does not help as
url-http-create-request sets "Accept: */*" and overrides any headers I
would set in url-request-extra-headers.

Example code:

    (let ((url-debug t)
          (url-request-method "POST")
	  (url-request-extra-headers
	    `(("Content-Type" . "application/x-www-form-urlencoded")))
          (url-request-data
            "query=SELECT%20%3FC%20WHERE%20%7B%5B%5D%20a%20%3FC%7D%20LIMIT%201%0A")
          (url-mime-accept-string "text/csv"))
      (url-retrieve "http://live.dbpedia.org/sparql/"
                    #'(lambda (_)
                        (message (buffer-string)))))

This will sometimes return a csv file and sometimes an xml file (usually
the first time). In *URL-DEBUG* you can see that the Accept header is
not properly set.

I believe that it fails as url-http-create-request is called in the
asynchronous context and can therefore, sometimes, escape the let
binding.



In GNU Emacs 24.5.1 (x86_64-apple-darwin15.2.0, NS apple-appkit-1404.34)
of 2016-01-18 on reventlov.local
Windowing system distributor `Apple', version 10.3.1404
Configured using:
`configure --prefix=/usr/local/Cellar/emacs/24.5
--enable-locallisppath=/usr/local/share/emacs/site-lisp
--infodir=/usr/local/Cellar/emacs/24.5/share/info/emacs --with-xml2
--without-dbus --with-gnutls --with-ns --disable-ns-self-contained'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  helm-mode: t
  shell-dirtrack-mode: t
  smartparens-strict-mode: t
  smartparens-mode: t
  eldoc-mode: t
  elisp-slime-nav-mode: t
  highlight-symbol-nav-mode: t
  highlight-symbol-mode: t
  window-number-mode: t
  persp-mode: t
  projectile-global-mode: t
  projectile-mode: t
  global-company-mode: t
  company-mode: t
  show-paren-mode: t
  override-global-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Searching for url-http-create-request...
Mark set

Mark saved where search started
user-error: No further undo information [2 times]
Mark set
Saving file /usr/local/Cellar/emacs/24.5/share/emacs/24.5/lisp/url/url-http.el.gz...
compressing url-http.el.gz...done
Wrote /usr/local/Cellar/emacs/24.5/share/emacs/24.5/lisp/url/url-http.el.gz
Mark saved where search started
Quit

Features:
(shadow mail-extr emacsbug sendmail misearch multi-isearch jka-compr
eieio-opt speedbar sb-image ezimage dframe find-func view sort
mm-archive message rfc822 mml mml-sec mailabbrev gmm-utils mailheader
mm-decode mm-bodies mm-encode mail-utils network-stream starttls
url-cache sparql-mode derived url-handlers url-http tls url-auth
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw helm-command
helm-elisp helm-eval edebug image-file vc-git winner
exec-path-from-shell helm-mode helm-projectile helm-files rx image-dired
tramp tramp-compat tramp-loaddefs trampver shell pcomplete format-spec
dired-x dired-aux ffap helm-tags helm-bookmark helm-adaptive helm-info
bookmark pp helm-external helm-net browse-url xml url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse auth-source gnus-util mm-util mail-prsvr
password-cache url-vars mailcap helm-buffers helm-grep helm-regexp
helm-plugin helm-elscreen helm-utils helm-locate helm-help helm-types
helm helm-source eieio eieio-core helm-multi-match helm-lib dired
smartparens-config smartparens eldoc elisp-slime-nav help-mode
highlight-symbol finder-inf clojure-mode align imenu window-number
smart-mode-line-respectful-theme smart-mode-line mule-util rich-minority
saveplace persp-projectile cl-macs perspective cl gv byte-opt projectile
advice help-fns grep compile comint ansi-color ibuf-ext ibuffer dash
thingatpt company-oddmuse company-keywords company-etags etags ring
company-gtags company-dabbrev-code company-dabbrev company-files
company-capf company-cmake company-xcode company-clang company-semantic
company-eclim company-template company-css company-nxml company-bbdb
company pcase paren edmacro kmacro sanityinc-tomorrow-night-theme
color-theme-sanityinc-tomorrow use-package diminish bytecomp
byte-compile cl-extra cconv bind-key easy-mmode cl-loaddefs cl-lib
go-mode-autoloads slime-autoloads info easymenu package epg-config
time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel ns-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process cocoa ns multi-tty emacs)

Memory information:
((conses 16 309551 388578)
(symbols 48 37911 23)
(miscs 40 186 1720)
(strings 32 77048 171324)
(string-bytes 1 2244791)
(vectors 16 31697)
(vector-slots 8 604233 182119)
(floats 8 160 1012)
(intervals 56 3074 2395)
(buffers 960 35))

bug#22855: 24.5; `url-mime-accept-string' not set in asynchronous call

[Lars Ingebrigtsen]

Bjarte Johansen <bjarte.johansen@gmail.com> writes:

>     (let ((url-debug t)
>           (url-request-method "POST")
> 	  (url-request-extra-headers
> 	    `(("Content-Type" . "application/x-www-form-urlencoded")))
>           (url-request-data
>             "query=SELECT%20%3FC%20WHERE%20%7B%5B%5D%20a%20%3FC%7D%20LIMIT%201%0A")
>           (url-mime-accept-string "text/csv"))
>       (url-retrieve "http://live.dbpedia.org/sparql/"
>                     #'(lambda (_)
>                         (message (buffer-string)))))
>
> This will sometimes return a csv file and sometimes an xml file (usually
> the first time). In *URL-DEBUG* you can see that the Accept header is
> not properly set.

The reason it sometimes work is that URL caches connections.  If you 

(setq url-http-open-connections (make-hash-table :test 'equal :size 17))

before calling, then it'll fail every time.

And the `url-mime-accept-string' variable is just not meant to be used
as an "interface" variable -- it's supposed to be a constant value.

But that really makes no sense, so I'll get fixin'.

I really dislike the `url-retrieve' "interface", where you bind a
gazillion special variables instead of passing in values explicitly.
One of these ... years...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no