bug#16543: 24.3; Newsticker retrieve method and https

bug#16543: 24.3; Newsticker retrieve method and https

[Cédric Chépied]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgment at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':


When the newsticker-url-list starts with an https url, newsticker tries to use
tls for all other urls.

start emacs -Q
go to buffer scratch
type: 
(setq newsticker-url-list (quote ( ("Archlinux"
"https://www.archlinux.org/feeds/news/" nil nil nil) ("Bonjour Madame"
"http://feeds2.feedburner.com/BonjourMadame" nil nil nil) )))
M-x eval-buffer
M-x newsticker-treeview

It hangs and last messages are:
Newsticker started!
Contacting host: www.emacswiki.org:80
Opening TLS connection to `www.archlinux.org'...
Opening TLS connection with `gnutls-cli --insecure -p 443 www.archlinux.org'...
Opening TLS connection to `feeds2.feedburner.com'...
Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com'...failed
Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com --protocols ssl3'...failed
Opening TLS connection with `openssl s_client -connect feeds2.feedburner.com:80 -no_ssl2 -ign_eof'...done


I found a "workaround":
start emacs -Q
go to buffer scratch
type: 
(setq newsticker-url-list (quote ( ("Bonjour Madame"
"http://feeds2.feedburner.com/BonjourMadame" nil nil nil) ("Archlinux"
"https://www.archlinux.org/feeds/news/" nil nil nil) )))
M-x eval-buffer
M-x newsticker-treeview

It works and I can say "Bonjour madame" (not really with emacs -Q because
pictures are not displayed). It only hang when https url is the first one. I did
not try with several https url.



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/24.3/etc/DEBUG.


In GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.6)
 of 2014-01-18 on mnt-storage-buildroots-staging-x86_64-eric
Windowing system distributor `The X.Org Foundation', version 11.0.11500000
Configured using:
 `configure '--prefix=/usr' '--sysconfdir=/etc' '--libexecdir=/usr/lib'
 '--localstatedir=/var' '--with-x-toolkit=gtk3' '--with-xft'
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector
 --param=ssp-buffer-size=4' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LANG: fr_FR.utf8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Folder

Minor modes in effect:
  global-srecode-minor-mode: t
  global-semantic-mru-bookmark-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-idle-scheduler-mode: t
  global-semantic-highlight-func-mode: t
  semantic-mode: t
  shell-dirtrack-mode: t
  global-auto-complete-mode: t
  diff-auto-refine-mode: t
  erc-smiley-mode: t
  show-paren-mode: t
  virtual-desktops-mode: t
  display-time-mode: t
  display-battery-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-x 1 M-x w l <return> <down> <return> <down> <down> 
<return> c y M-x e m a <tab> r e <M-backspace> <M-backspace> 
r e p o r t <tab> b <tab> <return>

Recent messages:
Loading msgdb for .~/.Mail/cedric.chepied@gmail.com/Buildroot...done
Checking folder diff...done
Creating msgdb...done
Loading msgdb for .~/.Mail/cedric.chepied@gmail.com/uClibc...done
Checking folder diff...done
Creating msgdb...done
All messages in cedric.chepied  are marked!
Starting new Ispell process [/usr/bin/aspell::default] ...
Type C-c C-x C-z to exit MIME mode, and type C-c C-x ? to get help.
Saving...done

Load-path shadows:
/usr/share/emacs/site-lisp/flim/hex-util hides /usr/share/emacs/24.3/lisp/hex-util
/usr/share/emacs/site-lisp/flim/md4 hides /usr/share/emacs/24.3/lisp/md4
/usr/share/emacs/site-lisp/flim/ntlm hides /usr/share/emacs/24.3/lisp/net/ntlm
/usr/share/emacs/site-lisp/flim/hmac-def hides /usr/share/emacs/24.3/lisp/net/hmac-def
/usr/share/emacs/site-lisp/flim/sasl hides /usr/share/emacs/24.3/lisp/net/sasl
/usr/share/emacs/site-lisp/flim/sasl-digest hides /usr/share/emacs/24.3/lisp/net/sasl-digest
/usr/share/emacs/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/24.3/lisp/net/sasl-ntlm
/usr/share/emacs/site-lisp/flim/sasl-cram hides /usr/share/emacs/24.3/lisp/net/sasl-cram
/usr/share/emacs/site-lisp/flim/hmac-md5 hides /usr/share/emacs/24.3/lisp/net/hmac-md5
/usr/share/emacs/site-lisp/semi/smime hides /usr/share/emacs/24.3/lisp/gnus/smime
/usr/share/emacs/site-lisp/semi/pgg-pgp5 hides /usr/share/emacs/24.3/lisp/obsolete/pgg-pgp5
/usr/share/emacs/site-lisp/semi/pgg-gpg hides /usr/share/emacs/24.3/lisp/obsolete/pgg-gpg
/usr/share/emacs/site-lisp/semi/pgg hides /usr/share/emacs/24.3/lisp/obsolete/pgg
/usr/share/emacs/site-lisp/semi/pgg-pgp hides /usr/share/emacs/24.3/lisp/obsolete/pgg-pgp
/usr/share/emacs/site-lisp/semi/pgg-def hides /usr/share/emacs/24.3/lisp/obsolete/pgg-def
/usr/share/emacs/site-lisp/semi/pgg-parse hides /usr/share/emacs/24.3/lisp/obsolete/pgg-parse

Features:
(shadow flyspell ispell wl-expire emacsbug message idna rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader elmo-maildir
modb-standard wl-mime mime-edit pgg-parse pccl pccl-20 pgg-def signature
mime-setup mail-mime-setup semi-setup mime-image wl-demo wl-draft eword-encode
wl-template sendmail elmo-net elmo-cache elmo-map elmo-dop wl-news wl-address
wl-thread wl-action wl-summary ps-print ps-def lpr wl-refile wl-message
elmo-mime mmelmo-buffer mmelmo-imap mmimap mime-parse mmbuffer mmgeneric
wl-highlight elmo-multi wl-folder wl wl-e21 wl-util elmo-flag elmo-localdir
wl-vars wl-version elmo elmo-signal elmo-msgdb modb modb-generic modb-entity
elmo-util elmo-date elmo-vars elmo-version luna mail-utils url-http mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums url-gw url-cache url-auth url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse url-vars mailcap hideshow chep-stickyfunc chep-tag-popup
chep-auto-complete semantic/analyze/complete semantic/bovine/gcc semantic/dep
srecode/mode semantic/senator semantic/decorate pulse srecode/insert
srecode/filters srecode/args srecode/find srecode/map srecode/srt-mode
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
srecode/template srecode/srt-wy semantic/wisent semantic/wisent/wisent
srecode/ctxt srecode/compile srecode/dictionary srecode/table srecode inversion
semantic/db-global semantic/symref/global semantic/symref semantic/db-find
semantic/db-ref cedet-global semantic/mru-bookmark semantic/db-mode semantic/db
eieio-base semantic/idle semantic/format semantic/tag-ls semantic/find
semantic/ctxt semantic/util-modes semantic/util semantic semantic/tag
semantic/lex semantic/fw mode-local cedet yasnippet readline-complete shell
pcomplete ac-etags auto-complete-config auto-complete popup smart-tabs-mode
magit-blame magit-key-mode magit view help-mode grep compile comint epa epg
epg-config diff-mode autorevert ansi-color git-rebase-mode git-commit-mode
log-edit pcvs-util add-log erc-goodies erc erc-backend erc-compat format-spec
thingatpt pp netrc chep-notification mime-play filename emu invisible inv-23
poem poem-e20 poem-e20_3 mime-view mime-conf calist semi-def mime eword-decode
mel path-util mime-def mcharset mcs-20 mcs-e20 pces pces-e20 pces-20 broken
pcustom poe std11 alist pym static apel-ver product mime-w3m w3m browse-url
doc-view jka-compr dired image-mode timezone w3m-hist w3m-fb bookmark-w3m
w3m-ems wid-edit w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util
ac-etags-autoloads auto-complete-autoloads edit-server-autoloads magit-autoloads
git-rebase-mode-autoloads git-commit-mode-autoloads mediawiki-autoloads
popup-autoloads finder-inf readline-complete-autoloads smart-tabs-mode-autoloads
w3m-autoloads info yasnippet-autoloads package cl-macs gv server edmacro kmacro
chep-mode-line php-mode etags cc-langs cl cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs speedbar sb-image ezimage
dframe xcscope ring paren uniquify warnings chep-strike chep-couleur
chep-retourne chep-pastebin google_search ifndef_fichier_h appt diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs newst-backend xml derived
newsticker-notify advice help-fns cl-lib advice-preload fmanager
chep-convert-audio chep-video chep-mpd ampc easymenu avl-tree network-stream
auth-source eieio byte-opt bytecomp byte-compile cconv gnus-util mm-util
mail-prsvr password-cache starttls tls chep-anchor easy-mmode virtual-desktops
ido time battery cus-start cus-load time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

bug#16543: 24.3; Newsticker retrieve method and https

[Ulf Jasper]

Cédric Chépied <cedric.chepied@gmail.com> writes:

> start emacs -Q
> go to buffer scratch
> type: 
> (setq newsticker-url-list (quote ( ("Archlinux"
> "https://www.archlinux.org/feeds/news/" nil nil nil) ("Bonjour Madame"
> "http://feeds2.feedburner.com/BonjourMadame" nil nil nil) )))
> M-x eval-buffer
> M-x newsticker-treeview
>
> It hangs and last messages are:
> Newsticker started!
> Contacting host: www.emacswiki.org:80
> Opening TLS connection to `www.archlinux.org'...
> Opening TLS connection with `gnutls-cli --insecure -p 443 www.archlinux.org'...
> Opening TLS connection to `feeds2.feedburner.com'...
> Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com'...failed
> Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com --protocols ssl3'...failed
> Opening TLS connection with `openssl s_client -connect feeds2.feedburner.com:80 -no_ssl2 -ign_eof'...done

I cannot reproduce that, neither with 24.3 nor with current trunk.  Do
you still see this problem?

bug#16543: 24.3; Newsticker retrieve method and https

[Cédric Chépied]

At Thu, 31 Jul 2014 20:16:18 +0200,
Ulf Jasper wrote:
> 
> Cédric Chépied <cedric.chepied@gmail.com> writes:
> 
> > start emacs -Q
> > go to buffer scratch
> > type: 
> > (setq newsticker-url-list (quote ( ("Archlinux"
> > "https://www.archlinux.org/feeds/news/" nil nil nil) ("Bonjour Madame"
> > "http://feeds2.feedburner.com/BonjourMadame" nil nil nil) )))
> > M-x eval-buffer
> > M-x newsticker-treeview
> >
> > It hangs and last messages are:
> > Newsticker started!
> > Contacting host: www.emacswiki.org:80
> > Opening TLS connection to `www.archlinux.org'...
> > Opening TLS connection with `gnutls-cli --insecure -p 443 www.archlinux.org'...
> > Opening TLS connection to `feeds2.feedburner.com'...
> > Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com'...failed
> > Opening TLS connection with `gnutls-cli --insecure -p 80 feeds2.feedburner.com --protocols ssl3'...failed
> > Opening TLS connection with `openssl s_client -connect feeds2.feedburner.com:80 -no_ssl2 -ign_eof'...done
> 
> I cannot reproduce that, neither with 24.3 nor with current trunk.  Do
> you still see this problem?

I cannot reproduce it anymore. I'm on master branch. I think you can close it.

Thanks,
-- 
Cédric Chépied
<cedric.chepied@gmail.com>

bug#16543: 24.3; Newsticker retrieve method and https

[Ulf Jasper]

Cédric Chépied <cedric.chepied@gmail.com> writes:

> I cannot reproduce it anymore. I'm on master branch. I think you can close it.

Closed.

bug#16543: Newsticker retrieve method and https

[Cédric Chépied]

Sorry but the bug as reappeared.

I'm on master branch. I still use the same recipe:

start emacs -Q
go to buffer scratch
type: 
(setq newsticker-url-list (quote ( ("Archlinux"
"https://www.archlinux.org/feeds/news/" nil nil nil) ("Bonjour Madame"
"http://feeds2.feedburner.com/BonjourMadame" nil nil nil) )))
M-x eval-buffer
M-x newsticker-treeview

With toggle-debug-on-quit:

Debugger entered--Lisp error: (quit)
  accept-process-output(#<process feeds2.feedburner.com> 1)
  open-tls-stream("feeds2.feedburner.com" #<buffer  *url-http-temp*-426647> "feeds2.feedburner.com" 80)
  network-stream-open-tls("feeds2.feedburner.com" #<buffer  *url-http-temp*-426647> "feeds2.feedburner.com" 80 (:type tls :nowait t))
  open-network-stream("feeds2.feedburner.com" #<buffer  *url-http-temp*-426647> "feeds2.feedburner.com" 80 :type tls :nowait t)
  byte-code("\306\211\x18\x19\n\307>\203$

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

Cédric Chépied <cedric.chepied@gmail.com> writes:

> Sorry but the bug as reappeared.

Reopened the bug.

> I'm on master branch. I still use the same recipe:
>
> start emacs -Q
> go to buffer scratch
> type: 
> (setq newsticker-url-list (quote ( ("Archlinux"
> "https://www.archlinux.org/feeds/news/" nil nil nil) ("Bonjour Madame"
> "http://feeds2.feedburner.com/BonjourMadame" nil nil nil) )))
> M-x eval-buffer
> M-x newsticker-treeview

Sorry, but I cannot reproduce that (revision 117928).

I will investigate, but please provide another url list (without
"Bonjour Madame") for which you see the problem.  Please also tell me
how you build Emacs (i.e. config switches).

Best,
Ulf

bug#16543: Newsticker retrieve method and https

[Cédric Chépied]

Hi,

At Tue, 23 Sep 2014 19:39:27 +0200,
Ulf Jasper wrote:
> Sorry, but I cannot reproduce that (revision 117928).

I'm on master branch, commit 7b6b99bbf30501a293f834e72ede829b4077a22a

commit 7b6b99bbf30501a293f834e72ede829b4077a22a
Author: Glenn Morris <rgm@gnu.org>
Date:   Tue Sep 23 06:21:30 2014 -0400

    Auto-commit of loaddefs files.
    
> I will investigate, but please provide another url list (without
> "Bonjour Madame") for which you see the problem.

No problem:
(setq newsticker-url-list (quote ( ("Archlinux"
"https://www.archlinux.org/feeds/news/" nil nil nil) ("LinuxFr"
"http://linuxfr.org/news.atom" nil nil nil))))

> Please also tell me
> how you build Emacs (i.e. config switches).

cd "$srcdir"/emacs
./autogen.sh
ac_cv_lib_gif_EGifPutExtensionLast=yes ./configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib \
    --localstatedir=/var --with-x-toolkit=lucid --with-xft --with-cc-fonts
make


Regards,
-- 
Cédric Chépied
<cedric.chepied@gmail.com>

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

Cédric Chépied <cedric.chepied@gmail.com> writes:

> No problem:
> (setq newsticker-url-list (quote ( ("Archlinux"
> "https://www.archlinux.org/feeds/news/" nil nil nil) ("LinuxFr"
> "http://linuxfr.org/news.atom" nil nil nil))))

Please try to evaluate this

    (mapc (lambda (u)
            (lexical-let ((msg (concat "Finished " u)))
              (url-retrieve u (lambda (&rest ignore)
                                (message msg)))))
          '("https://www.archlinux.org/feeds/news/"
            "http://linuxfr.org/news.atom"))
    
and tell me what you see in *Messages*.

bug#16543: Newsticker retrieve method and https

[Cédric Chépied]

At Wed, 24 Sep 2014 19:53:26 +0200,
Ulf Jasper wrote:
> Please try to evaluate this
> 
>     (mapc (lambda (u)
>             (lexical-let ((msg (concat "Finished " u)))
>               (url-retrieve u (lambda (&rest ignore)
>                                 (message msg)))))
>           '("https://www.archlinux.org/feeds/news/"
>             "http://linuxfr.org/news.atom"))
>     
> and tell me what you see in *Messages*.

With emacs -Q:
Symbol's function definition is void: lexical-let

With my configuration loaded:

Contacting host: www.archlinux.org:443
Opening TLS connection to `www.archlinux.org'...
Opening TLS connection with `gnutls-cli --insecure -p 443 www.archlinux.org'...done
Opening TLS connection to `www.archlinux.org'...done
Contacting host: linuxfr.org:80
Finished https://www.archlinux.org/feeds/news/
Finished http://linuxfr.org/news.atom

-- 
Cédric Chépied
<cedric.chepied@gmail.com>

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

> Symbol's function definition is void: lexical-let

My mistake.  Sorry.

After studying the stack trace and doing some tests I found out that the
problem occurs only if

  * `url-retrieve' is called from timers, 
  * the first url is of type https, the second of type http (as you mentioned)
  * gnutls-cli is used for https retrieval (libgnutls works fine)

Here is a recipe for reproducing the error.  I am preparing a fix now.

(defun debbug-16543-callback (status url)
  "Dummy callback method for url-retrieve which ignores STATUS, shows URL.
Show also the current value of `url-gateway-method'."
  (message "debbug-16543-callback url=%s url-gateway-method=%s"
           url url-gateway-method))

(defun debbug-16543-call-url-retreive (url)
  "Call `url-retrieve' for URL.
Forces the gnutls cli to be used."
  (let ((old-gnutls-available-p (symbol-function 'gnutls-available-p)))
    (unwind-protect
        (ignore-errors
          ;; force usage of command line tls
          (setf (symbol-function 'gnutls-available-p)
                #'(lambda () nil))
          ;; call url-retrieve for the url
          (url-retrieve url 'debbug-16543-callback (list url)))
      ;; restore original functions
      (setf (symbol-function 'gnutls-available-p)
            old-gnutls-available-p))))

(defun debbug-16543-check ()
  "Try to reproduce bug#16543.
If bug occurs then the linuxfr.org is fetched via tls which will
cause Emacs to hang."
  (interactive)
  (mapc (lambda (url)
          (run-at-time 0 nil 'debbug-16543-call-url-retreive url))
        '("https://www.archlinux.org/feeds/news/"
          "http://linuxfr.org/news.atom")))

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

Fixed in rev. 117970.

Please confirm that the problem really is gone.

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

Ulf Jasper <ulf.jasper@web.de> writes:

> Fixed in rev. 117970.

That should be 117972.

> Please confirm that the problem really is gone.

bug#16543: Newsticker retrieve method and https

[Cédric Chépied]

At Mon, 29 Sep 2014 10:19:44 +0200,
Ulf Jasper wrote:
> 
> Ulf Jasper <ulf.jasper@web.de> writes:
> 
> > Fixed in rev. 117970.
> 
> That should be 117972.
> 
> > Please confirm that the problem really is gone.

Fixed here :-)

Thank you
-- 
Cédric Chépied
<cedric.chepied@gmail.com>

bug#16543: Newsticker retrieve method and https

[Ulf Jasper]

Cédric Chépied <cedric.chepied@gmail.com> writes:

> Fixed here :-)

Thanks.  Closing again.