bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Damien Cassou]

I'm using Emacs 25 rc1 and org-caldav from
https://github.com/dengste/org-caldav. The function #'org-caldav-sync
(https://github.com/dengste/org-caldav/blob/master/org-caldav.el#L524)
raises an error when a local event with accented characters is sent to
the remote cardav server.

When I switch to Emacs 24, the synchronization works perfectly.



In GNU Emacs 25.1.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.20.6)
 of 2016-08-07 built on localhost
Windowing system distributor 'The X.Org Foundation', version 11.0.11803000
Configured using:
 'configure --disable-dependency-tracking
 --prefix=/nix/store/r602ppq51pa30vwj9xwr390dpqm44n24-emacs-25.1-rc1
 --with-x-toolkit=gtk3 --with-xft'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $EMACSLOADPATH: /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp:
  value of $LC_MONETARY: fr_FR.UTF-8
  value of $LC_NUMERIC: fr_FR.UTF-8
  value of $LC_TIME: fr_FR.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Messages

Minor modes in effect:
  shell-switcher-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  beacon-mode: t
  ace-window-display-mode: t
  helm-mode: t
  helm-descbinds-mode: t
  async-bytecomp-package-mode: t
  diff-auto-refine-mode: t
  global-paren-face-mode: t
  global-visible-mark-mode: t
  visible-mark-mode: t
  runner-run-in-background: t
  projectile-global-mode: t
  projectile-mode: t
  global-discover-mode: t
  discover-mode: t
  guide-key-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  drag-stuff-global-mode: t
  drag-stuff-mode: t
  shell-dirtrack-mode: t
  global-ethan-wspace-mode: t
  override-global-mode: t
  save-place-mode: t
  winner-mode: t
  show-paren-mode: t
  recentf-mode: t
  global-hl-line-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
UID:86d3c6f5-14e0-4b1c-9444-27d4271a47c5
DTSTART;VALUE=DATE:20160820
DTEND;VALUE=DATE:20160821
RRULE:FREQ=YEARLY;INTERVAL=1
SUMMARY:Anniversaire d’Ana¯s (2016)
DESCRIPTION:
CATEGORIES:schplaf
END:VEVENT
END:VCALENDAR


Load-path shadows:
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-maildir-fcc hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-maildir-fcc
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-tag hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-tag
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-mua hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-mua
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-hello hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-hello
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-address hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-address
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-show hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-show
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-query hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-query
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-print hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-print
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-jump hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-jump
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-wash hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-wash
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/coolj hides /home/cassou/.nix-profile/share/emacs/site-lisp/coolj
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-tree hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-tree
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-company hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-company
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-crypto hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-crypto
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-version hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-version
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-lib hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-lib
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-message hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-message
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch-parser hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch-parser
/nix/store/03bz4mi799mq4ji7rwfq0sjgnmw6hnm0-notmuch-0.22/share/emacs/site-lisp/notmuch hides /home/cassou/.nix-profile/share/emacs/site-lisp/notmuch
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.3.0/avy hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.4.0/avy
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.3.0/avy-pkg hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.4.0/avy-pkg
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.3.0/avy-autoloads hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/avy-0.4.0/avy-autoloads
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-2.2/bind-key hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-20150321/bind-key
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-2.2/bind-key-pkg hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-20150321/bind-key-pkg
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-2.2/bind-key-autoloads hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/bind-key-20150321/bind-key-autoloads
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-0.18.2/f hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-20151113/f
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-0.18.2/f-pkg hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-20151113/f-pkg
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-0.18.2/f-autoloads hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/f-20151113/f-autoloads
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/helm-1.9.9/helm-multi-match hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/helm-core-1.9.9/helm-multi-match
/run/current-system/sw/share/emacs/site-lisp/nix-mode hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/nix-mode-1.11.2/nix-mode
/home/cassou/.nix-profile/share/emacs/site-lisp/password-store hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/password-store-1.6.5/password-store
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.13.0/projectile hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.14.0/projectile
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.13.0/projectile-autoloads hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.14.0/projectile-autoloads
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.13.0/projectile-pkg hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/projectile-0.14.0/projectile-pkg
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-1.11.0/s hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-20151023/s
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-1.11.0/s-pkg hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-20151023/s-pkg
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-1.11.0/s-autoloads hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/s-20151023/s-autoloads
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/counsel-0.8.0/counsel hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/swiper-0.7.0/counsel
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/wgrep-20141017/wgrep-helm hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/wgrep-helm-2.1.10/wgrep-helm
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/org-vcard-0.1.0/styles/flat/functions hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/org-vcard-0.1.0/styles/tree/functions
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package/__DOT__dir-locals hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package-with-docs/__DOT__dir-locals
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package/__PROJECT-NAME__ hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package-with-docs/__PROJECT-NAME__
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package/test/__PROJECT-NAME__-tests hides /nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/skeletor-1.6.1/project-skeletons/elisp-package-with-docs/test/__PROJECT-NAME__-tests
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/site-start hides /nix/store/r602ppq51pa30vwj9xwr390dpqm44n24-emacs-25.1-rc1/share/emacs/site-lisp/site-start
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/let-alist-1.0.4/let-alist hides /nix/store/r602ppq51pa30vwj9xwr390dpqm44n24-emacs-25.1-rc1/share/emacs/25.1/lisp/emacs-lisp/let-alist
/nix/store/jzmx884q0xfhzcqlhx1zdppf7g7angyz-emacs-packages-deps/share/emacs/site-lisp/elpa/seq-2.16/seq hides /nix/store/r602ppq51pa30vwj9xwr390dpqm44n24-emacs-25.1-rc1/share/emacs/25.1/lisp/emacs-lisp/seq

Features:
(shadow emacsbug cal-move mm-archive htmlfontify ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util parse-time org-agenda tabify org-table
ox-beamer org-capture org-protocol url-cache org-caldav org-id url-dav
url-http url-auth url-gw url-handlers helm-command em-unix em-term term
disp-table ehelp em-script em-prompt em-ls em-hist em-pred em-glob
em-dirs em-cmpl em-basic em-banner em-alias shell-switcher rswitcher
network-stream nsm starttls tls gnutls mailalias mail-extr vdirel
org-vcard ox-twbs ox-latex ox-icalendar ox-html ox-ascii ox-publish ox
org-element org org-macro org-footnote org-pcomplete org-list org-faces
org-entities org-version ob-python ob-java ob-emacs-lisp ob-sh ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint
ob-core ob-eval org-compat org-macs org-loaddefs sort notmuch-jump
notmuch notmuch-message notmuch-maildir-fcc notmuch-hello notmuch-tree
notmuch-show notmuch-print notmuch-crypto notmuch-mua notmuch-address
notmuch-company notmuch-parser notmuch-wash coolj notmuch-query
goto-addr icalendar diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs notmuch-tag notmuch-lib notmuch-version mm-view mml-smime
smime dig smartscan nix-mode ido tramp-cache linum esh-var esh-io
esh-cmd esh-opt esh-ext esh-proc esh-arg esh-groups magit-blame
magit-stash magit-bisect magit-remote magit-commit epa magit-sequence
magit helm-eshell helm-elisp helm-eval edebug eshell esh-module esh-mode
esh-util magit-apply magit-wip magit-log magit-diff derived smerge-mode
magit-core magit-autorevert autorevert filenotify magit-process
magit-popup magit-mode magit-git crm magit-section magit-utils
git-commit log-edit pcvs-util add-log with-editor tramp-sh server
flyspell ispell beacon nameless auth-password-store beginend
flycheck-cask flycheck-package lisp-mnt flycheck json map subr-x
noutline outline ace-window password-store f s youtube helm-mode
helm-descbinds helm-config helm-easymenu async-bytecomp async
aggressive-indent git-timemachine vc-git diff-mode find-func paren-face
visible-mark git-auto-commit-mode paredit iso-transl profile smtpmail
sendmail cus-edit message rfc822 mml mml-sec epg mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader unify-opening helm-projectile helm-files
rx image-dired tramp tramp-compat tramp-loaddefs trampver ucs-normalize
format-spec ffap helm-tags helm-bookmark helm-adaptive helm-info
bookmark pp helm-external helm-net browse-url xml url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse auth-source gnus-util mm-util help-fns mail-prsvr
password-cache url-vars mailcap helm-buffers helm-grep wgrep-helm wgrep
helm-regexp helm-elscreen helm-utils helm-locate helm-help helm-types
helm helm-source eieio-compat eieio eieio-core helm-multi-match helm-lib
dired-imenu imenu runner dired-aux dired-x dired projectile grep
ibuf-ext ibuffer dash thingatpt discover makey guide-key face-remap
popwin undo-tree diff drag-stuff tex-mode compile shell pcomplete comint
ansi-color calc-prog calc-ext calc calc-loaddefs calc-macs ace-link avy
ethan-wspace edmacro kmacro zerodark-theme use-package diminish bind-key
easy-mmode finder-inf tex-site advice cl-seq cl-macs cl info package
epg-config seq seq-25 byte-opt gv bytecomp byte-compile cl-extra
help-mode cconv saveplace winner ring paren recentf tree-widget wid-edit
cl-loaddefs pcase cl-lib easymenu hl-line cus-start cus-load time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 711921 384227)
 (symbols 48 56735 83)
 (miscs 40 10218 12651)
 (strings 32 144440 74389)
 (string-bytes 1 6921745)
 (vectors 16 84308)
 (vector-slots 8 2061323 196674)
 (floats 8 666 2545)
 (intervals 56 19321 11810)
 (buffers 976 64)
 (heap 1024 78340 26129))

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 08/15/2016 09:58 PM, Damien Cassou wrote:
>
> I'm using Emacs 25 rc1 and org-caldav from
> https://github.com/dengste/org-caldav. The function #'org-caldav-sync
> (https://github.com/dengste/org-caldav/blob/master/org-caldav.el#L524)
> raises an error when a local event with accented characters is sent to
> the remote cardav server.

org-caldav probably binds url-request-data to a multibyte string. It 
should convert that to unibyte.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Damien Cassou]

Hi Dmitry,

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/15/2016 09:58 PM, Damien Cassou wrote:
>> I'm using Emacs 25 rc1 and org-caldav from
>> https://github.com/dengste/org-caldav. The function #'org-caldav-sync
>> (https://github.com/dengste/org-caldav/blob/master/org-caldav.el#L524)
>> raises an error when a local event with accented characters is sent to
>> the remote cardav server.
>
> org-caldav probably binds url-request-data to a multibyte string. It
> should convert that to unibyte.

thank you very much for your very fast answer. I hope that will help me
fix my problem. Can you please help me understand what you say by
answering these questions? Thanks.

- what will happen to my multibyte characters (such as ⇒) that I'm using
  for months and that worked well with Emacs 24?

- where is the unibyte conversion requirement discussed in the url
  library documentation?

- where is the behavior change discussed in the Emacs' changelog?

Thank you again.

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 08/16/2016 09:23 AM, Damien Cassou wrote:

> - what will happen to my multibyte characters (such as ⇒) that I'm using
>   for months and that worked well with Emacs 24?

Unibyte != ASCII. Use encode-coding-string.

> - where is the behavior change discussed in the Emacs' changelog?

We normally don't mention bug fixes in there. Binding url-request-data 
to a string with multibyte characters never actually worked properly (it 
resulted in a wrong Content-Length value). Some servers may have ignored 
that, but some may have even corrupted data silently.

 > - where is the unibyte conversion requirement discussed in the url
 >   library documentation?

Patch welcome?

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 16 Aug 2016 11:16:26 +0300
> 
> > - where is the behavior change discussed in the Emacs' changelog?
> 
> We normally don't mention bug fixes in there. Binding url-request-data 
> to a string with multibyte characters never actually worked properly (it 
> resulted in a wrong Content-Length value). Some servers may have ignored 
> that, but some may have even corrupted data silently.

I guess you are talking about NEWS, not ChangeLog.  It's tru we don't
describe bug-fixes in NEWS, but we do describe incompatible changes.
So I think it would be good to document this change there.  Could you
suggest some simple description of that?  (I don't understand the
details of the use case well enough to do that myself -- how is the
value of url-request-data related to the change we made in
url-http-create-request?)

Thanks.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 08/16/2016 06:01 PM, Eli Zaretskii wrote:

> I guess you are talking about NEWS, not ChangeLog.

Yup. That's what I gathered Damien to mean, considering we do mention 
this change in the generated ChangeLog.

> It's tru we don't
> describe bug-fixes in NEWS, but we do describe incompatible changes.
> So I think it would be good to document this change there.  Could you
> suggest some simple description of that?

Something like "url-request-data is not allowed to be bound to a 
multibyte value; if your request body is multibyte (which is the usual 
case, by the way), convert it with encode-coding-string".

Hmm... by the way, we could also mention that json-encode now keeps most 
multibyte characters as-is. Which is the reason we had to scramble to 
fix this problem with the URL package.

> (I don't understand the
> details of the use case well enough to do that myself -- how is the
> value of url-request-data related to the change we made in
> url-http-create-request?)

The related change is for the previous bug report (23750), and the 
commit a98aa02a that added the "Multibyte text in HTTP request" error.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Eli Zaretskii]

> Cc: damien@cassou.me, 24239@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 17 Aug 2016 04:32:24 +0300
> 
> > It's tru we don't
> > describe bug-fixes in NEWS, but we do describe incompatible changes.
> > So I think it would be good to document this change there.  Could you
> > suggest some simple description of that?
> 
> Something like "url-request-data is not allowed to be bound to a 
> multibyte value; if your request body is multibyte (which is the usual 
> case, by the way), convert it with encode-coding-string".

Thanks, I added something like that to NEWS.

> Hmm... by the way, we could also mention that json-encode now keeps most 
> multibyte characters as-is. Which is the reason we had to scramble to 
> fix this problem with the URL package.

If you can add this to NEWS in time for RC2, please do.

> > (I don't understand the
> > details of the use case well enough to do that myself -- how is the
> > value of url-request-data related to the change we made in
> > url-http-create-request?)
> 
> The related change is for the previous bug report (23750), and the 
> commit a98aa02a that added the "Multibyte text in HTTP request" error.

Yes, but how is url-request-data related?  And are there any other
variables that should be mentioned?

Thanks.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 08/17/2016 05:44 AM, Eli Zaretskii wrote:

>> The related change is for the previous bug report (23750), and the
>> commit a98aa02a that added the "Multibyte text in HTTP request" error.
>
> Yes, but how is url-request-data related?

Directly? The bug was about us setting Content-Length to a wrong value. 
Content-Length describes the size of the request body, which is assigned 
through url-request-data.

> And are there any other
> variables that should be mentioned?

url-http-extra-headers, probably. But its contents are less likely to be 
multibyte.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Eli Zaretskii]

> Cc: damien@cassou.me, 24239@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 17 Aug 2016 06:57:08 +0300
> 
> On 08/17/2016 05:44 AM, Eli Zaretskii wrote:
> 
> >> The related change is for the previous bug report (23750), and the
> >> commit a98aa02a that added the "Multibyte text in HTTP request" error.
> >
> > Yes, but how is url-request-data related?
> 
> Directly? The bug was about us setting Content-Length to a wrong value. 
> Content-Length describes the size of the request body, which is assigned 
> through url-request-data.

Yes, I know.  My problem is that I don't see how binding
url-request-data affects the payload that url-http-create-request
creates.  I'm probably missing something obvious, but what?

> > And are there any other
> > variables that should be mentioned?
> 
> url-http-extra-headers, probably. But its contents are less likely to be 
> multibyte.

OK, so I guess the entry is OK as it is.

Thanks.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 08/17/2016 06:17 PM, Eli Zaretskii wrote:

> Yes, I know.  My problem is that I don't see how binding
> url-request-data affects the payload that url-http-create-request
> creates.  I'm probably missing something obvious, but what?

url-http sets url-http-data to the value of url-request-data. And 
url-http-create-request references url-http-data in two places.

It's a bit confusing indeed.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Eli Zaretskii]

> Cc: damien@cassou.me, 24239@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 17 Aug 2016 21:02:42 +0300
> 
> url-http sets url-http-data to the value of url-request-data. And 
> url-http-create-request references url-http-data in two places.

Got it, thanks.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Damien Cassou]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 08/16/2016 09:23 AM, Damien Cassou wrote:
>
>> - what will happen to my multibyte characters (such as ⇒) that I'm using
>>   for months and that worked well with Emacs 24?
>
> Unibyte != ASCII. Use encode-coding-string.

sorry, I don't understand how to do that. org-caldav code is quite
simple. The two important functions are:

https://github.com/dengste/org-caldav/blob/master/org-caldav.el#L1246
https://github.com/dengste/org-caldav/blob/master/org-caldav.el#L315

    (defun org-caldav-put-event (buffer)
      "Add event in BUFFER to calendar.
     The filename will be derived from the UID."
      [...]
       (org-caldav-save-resource
          (concat (org-caldav-events-url) uid org-caldav-uuid-extension)
          (encode-coding-string (buffer-string) 'utf-8))))))

  (defun org-caldav-save-resource (url obj)
    "Save string OBJ as URL using WebDAV."
    (let* ((url-request-extra-headers
            '(("Content-type" . "text/calendar; charset=UTF-8")))
           (url-request-method "PUT")
           (url-request-data obj)
           (buffer (url-retrieve-synchronously url))
           result)
    [...]))

As you can see, encode-coding-string is already used to encode to utf-8.
Do I have to switch from utf-8 to iso-8859-*?

Sorry, I'm lost.

--
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Dmitry Gutov]

On 21.09.2016 13:27, Damien Cassou wrote:

> Do I have to switch from utf-8 to iso-8859-*?

Not at all (the strings must be unibyte, that's the only condition), the 
code above looks fine.

Are you using the release version? If not, you might want to upgrade. 
We've fixed a subtle bugs with cookies in the meantime.

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Damien Cassou]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 21.09.2016 13:27, Damien Cassou wrote:
>
>> Do I have to switch from utf-8 to iso-8859-*?
>
> Not at all (the strings must be unibyte, that's the only condition), the 
> code above looks fine.
>
> Are you using the release version? If not, you might want to upgrade. 
> We've fixed a subtle bugs with cookies in the meantime.

it seems Emacs 25.1 doesn't have the problem. It was indeed a bug after
all :-).

Thanks

-- 
Damien Cassou
http://damiencassou.seasidehosting.st

"Success is the ability to go from one failure to another without
losing enthusiasm." --Winston Churchill

bug#24239: 25.1; Potential regression: Multibyte text in HTTP request

[Noam Postavsky]

tags 24239 fixed
close 24239 25.1
quit

Damien Cassou <damien@cassou.me> writes:

> it seems Emacs 25.1 doesn't have the problem. It was indeed a bug after
> all :-).

I take it that means the bug should be closed.