bug#28971: 24.4; Gnus TLS/SSL POP issue

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Emanuel Berg]

There was a discussion on
gmane.emacs.gnus.general about this bug or
problem. Best probably just to check it out
there. Otherwise I had it appended to a file
and made some edits to make it more clear -
here it is

Something happened to my Gnus. Or the
remote connection?

Now when I do `gnus-group-get-new-news' Gnus
hangs until I do C-g.

The broken POP processes pile up at
`list-processes'.

> Maybe confirm that POP access works from the
> command line first?
> https://unix.stackexchange.com/questions/201818/checking-pop-mail-account-using-terminal

Error according to socat, works with openssl.

test-zoho () { local server=pop.zoho.com local
port=995 socat - OPENSSL:${server}:${port} echo
"socat done" openssl s_client -quiet -connect
${server}:${port} echo "openssl done" }

$ test-zoho

2017/10/20 17:54:09 socat[29967]
E SSL_connect(): error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate
verify failed socat done depth=2 C = US, O =
"thawte, Inc.", OU = Certification Services
Division, OU = "(c) 2006 thawte, Inc. - For
authorized use only", CN = thawte Primary Root
CA verify return:1 depth=1 C = US, O = "thawte,
Inc.", OU = Domain Validated SSL, CN = thawte
DV SSL CA - G2 verify return:1 depth=0 CN =
*.zoho.com verify return:1 +OK POP3
mx.zohomail.com v1.0 server ready
<10427.1508514850321@mx.zohomail.com> user XYZ
+OK pass ABC +OK quit +OK mx.zohomail.com
Service closing transmission channel openssl
done

> Error according to socat, works with openssl.
> test-zoho () { local server=pop.zoho.com local port=995 socat - OPENSSL:${server}:${port} echo "socat done" openssl s_client -quiet -connect ${server}:${port} echo "openssl done" }
> $ test-zoho
> 2017/10/20 17:54:09 socat[29967] E SSL_connect(): error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed socat done depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA verify return:1 depth=1 C = US, O = "thawte, Inc.", OU = Domain Validated SSL, CN = thawte DV SSL CA - G2 verify return:1 depth=0 CN = *.zoho.com verify return:1 +OK POP3 mx.zohomail.com v1.0 server ready <10427.1508514850321@mx.zohomail.com> user XYZ +OK pass ABC +OK quit +OK mx.zohomail.com Service closing transmission channel openssl done

Unfortunately I don't know too much about tls,
and will be of limited help here, but my guess
is there's something slightly wrong with the
pop.zoho.com certificate, something that
openssl is letting slide, but socat and gnutls
aren't happy about.

> my guess is there's something slightly wrong
> with the pop.zoho.com certificate, something
> that openssl is letting slide, but socat and
> gnutls aren't happy about.

What exactly is openssl? Can I use that instead
of gnutls?

>> my guess is there's something slightly wrong
>> with the pop.zoho.com certificate, something
>> that openssl is letting slide, but socat and
>> gnutls aren't happy about.
> What exactly is openssl? Can I use that
> instead of gnutls?

My understanding (and again, I know very little
about these things) is that the two programs
are roughly equivalent: they handle creating
and checking TLS certificates, and creating and
handling TLS-encrypted connections to
remote servers.

I actually was just assuming you are using
gnutls, you can probably check the
libgnutls-version. You can also try setting
gnutls-log-level to 3 or 4 and fetching mail,
and see if anything interesting pops up in
*Messages*.

The bad news is that I tried this:

gnutls-cli --port 995 pop.zoho.com

And it connected just fine. You might try that,
and going through with the login procedure.

> the two programs are roughly equivalent: they
> handle creating and checking TLS
> certificates, and creating and handling
> TLS-encrypted connections to remote servers.

OK, then how do I specify openssl should be
used instead? Worth a shot... A search with
`apropos-value' and "gnutls" reveals

    starttls-gnutls-program "gnutls-cli"

    tls-program ("gnutls-cli --insecure -p %p
%h" "gnutls-cli --insecure -p %p %h --protocols
ssl3" "openssl s_client -connect %h:%p -no_ssl2
-ign_eof")

Neither set by me, but can be incorrect anyway,
of course. Eheh.

> I actually was just assuming you are using
> gnutls, you can probably check the
> libgnutls-version.

If you mean ... then:

    $ gnutls-cli --version gnutls-cli 3.3.8

> You can also try setting gnutls-log-level to
> 3 or 4 and fetching mail, and see if anything
> interesting pops up in *Messages*.

This does, over and over again:

    gnutls.c: [3] ASSERT: gnutls_buffers.c:547

    gnutls.c: [3] (Emacs) retry: Resource
temporarily unavailable, try again. gnutls.c:
[1] (Emacs) non-fatal error: Resource
temporarily unavailable, try again. gnutls.c:
[3] ASSERT: gnutls_buffers.c:1104

> The bad news is that I tried this:
> gnutls-cli --port 995 pop.zoho.com
> And it connected just fine. You might try
> that, and going through with the
> login procedure.

Login procedure works, the rest seems fine (?).

>> the two programs are roughly equivalent:
>> they handle creating and checking TLS
>> certificates, and creating and handling
>> TLS-encrypted connections to remote servers.
> OK, then how do I specify openssl should be
> used instead? Worth a shot... A search with
> `apropos-value' and "gnutls" reveals

>     starttls-gnutls-program "gnutls-cli"
> tls-program ("gnutls-cli --insecure -p %p %h"
> "gnutls-cli --insecure -p %p %h --protocols
> ssl3" "openssl s_client -connect %h:%p
> -no_ssl2 -ign_eof")

Maybe change this variable so that only the
"openssl" string is in there?

> If you mean ... then:
> $ gnutls-cli --version gnutls-cli 3.3.8

I meant that if the variable libgnutls-version
was non-nil, it probably meant you were using
gnutls. But the variables you found above are
more relevant.

>> You can also try setting gnutls-log-level to
>> 3 or 4 and fetching mail, and see if
>> anything interesting pops up in *Messages*.
>
> This does, over and over again:
>
> gnutls.c: [3] ASSERT: gnutls_buffers.c:547
>
> gnutls.c: [3] (Emacs) retry: Resource
> temporarily unavailable, try again.
> gnutls.c: [1] (Emacs) non-fatal error:
> Resource temporarily unavailable, try again.
> gnutls.c: [3] ASSERT: gnutls_buffers.c:1104
> The bad news is that I tried this:
>> gnutls-cli --port 995 pop.zoho.com

>> And it connected just fine. You might try
>> that, and going through with the login
>> >procedure.
>
> Login procedure works, the rest seems fine
> (?).

Another post on the topic:

I'm gonna use this thread to dump everything
related to this issue. At least then I have it
all in one place.

I've had the issue for several years off and
on, on several computers even with slightly
different OSs (Debian and Raspbian) so it is
starting to get embarrassing.

    Mail source (pop :user XYZ :password ABC
:server pop.zoho.com :port 995 :stream ssl)
failed: (error USER XYZ not valid)

and

    gnutls.c: [0] (Emacs) fatal error: The TLS
connection was non-properly terminated.




In GNU Emacs 24.4.1 (arm-unknown-linux-gnueabihf, GTK+ Version 3.14.5)
 of 2015-03-10 on bm-wb-01, modified by Debian
System Description:	Raspbian GNU/Linux 8.0 (jessie)

Configured using:
 `configure --build arm-linux-gnueabihf
 --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.4/site-lisp:/usr/share/emacs/site-lisp
 --build arm-linux-gnueabihf --prefix=/usr
 --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.4/site-lisp:/usr/share/emacs/site-lisp
 --with-x=yes --with-x-toolkit=gtk3
 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fstack-protector-strong -Wformat
 -Werror=format-security -Wall'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-z,relro'

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  show-paren-mode: t
  shell-dirtrack-mode: t
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-match-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-netsplit-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  erc-scrolltobottom-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-compression-mode: t
  auto-fill-function: do-auto-fill
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
C-n C-n C-n C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p 
C-p C-n C-n C-n C-n C-n C-n C-n C-p C-p RET C-p C-p 
ESC q C-n > > C-d C-d C-p ESC q C-n C-n C-n C-n C-n 
C-n C-n C-p C-p C-p C-p C-k C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p RET ESC q C-n C-f SPC C-b > C-p ESC 
q C-n C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f 
C-f C-n C-b C-f DEL RET RET > DEL C-n C-d C-d C-d C-p 
C-n C-d C-p > > DEL C-n C-b > SPC C-p C-b SPC C-p C-k 
RET C-p > C-n C-a ESC q C-e C-n C-n C-n C-a ESC q C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-p C-p 
C-p C-p C-p C-p C-p C-p ESC q C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-e RET RET A n o t 
h e r SPC p o s t SPC o n t SPC DEL DEL SPC t h e SPC 
t o p i o DEL c s DEL : C-x C-s C-o i C-o e i i SPC 
SPC C-o i C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-@ C-o C-k C-@ C-o k ESC w C-o i ESC 
x r e p o r t - b u g RET

Recent messages:
Starting new Ispell process [ispell::american-insane] ...
Spell-checking error-processed.txt using ispell with american-insane dictionary...
Personal dictionary saved.
Spell-checking error-processed.txt using ispell with american-insane dictionary...done
Mark set
Mark activated
C-o C-k is undefined
Mark activated
Saved text from "Something happened to my Gnus. Or the
re"

Load-path shadows:
/usr/share/emacs/24.4/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
./help hides /usr/share/emacs/24.4/lisp/help
~/.emacs.d/emacs-init/misc hides /usr/share/emacs/24.4/lisp/misc
/usr/share/emacs24/site-lisp/flim/md4 hides /usr/share/emacs/24.4/lisp/md4
/usr/share/emacs24/site-lisp/flim/hex-util hides /usr/share/emacs/24.4/lisp/hex-util
~/.emacs.d/lisp/abbrev hides /usr/share/emacs/24.4/lisp/abbrev
~/.emacs.d/emacs-init/gnus/server hides /usr/share/emacs/24.4/lisp/server
/usr/share/emacs24/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/24.4/lisp/textmodes/ispell
/usr/share/emacs24/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/24.4/lisp/textmodes/flyspell
/usr/share/emacs24/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/24.4/lisp/net/sasl-ntlm
/usr/share/emacs24/site-lisp/flim/sasl-cram hides /usr/share/emacs/24.4/lisp/net/sasl-cram
/usr/share/emacs24/site-lisp/flim/ntlm hides /usr/share/emacs/24.4/lisp/net/ntlm
/usr/share/emacs24/site-lisp/flim/sasl hides /usr/share/emacs/24.4/lisp/net/sasl
/usr/share/emacs24/site-lisp/flim/hmac-def hides /usr/share/emacs/24.4/lisp/net/hmac-def
/usr/share/emacs24/site-lisp/flim/hmac-md5 hides /usr/share/emacs/24.4/lisp/net/hmac-md5
/usr/share/emacs24/site-lisp/flim/sasl-digest hides /usr/share/emacs/24.4/lisp/net/sasl-digest

Features:
(shadow emacsbug make-mode latexenc flow-fill
sh-script smie executable w3m-cookie macros
mailalias mm-archive qp nnfolder gnus-bcklg
gnus-async gnus-dup gnus-ml mail-extr canlock
nnir eieio-opt speedbar sb-image ezimage dframe
find-func pop3 nndraft nnmh nnml netrc gnutls
network-stream starttls tls gnus-agent
nnvirtual nntp gnus-cache mule-util cl-macs gv
w3m-filter ffap paren xsel w3m-form
google-translate-core-ui ido
google-translate-core google-translate-tk url
url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util
url-parse url-vars json time-my tabs parse-time
tramp-sh term disp-table ehelp my-faces bibtex
t-mouse man tex-mode iterate-files isbn
slime-presentations slime-repl slime-parse
bridge nroff-mode sgml-mode cobol-mode ada-mode
which-func imenu align find-file checkdoc
cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs
guile summary gnus-score score-mode message-my
moggle mail-to-many mail smtpmail sendmail
global-keys yank-my wrap-search w3m-unisearch
w3m-search spell-new ispell sort-my sort slime
etags arc-mode archive-mode noutline outline
easy-mmode hyperspec shell-cli revert-buffer-my
lisp-new ielm linux-shell kill keys help-new
debug apropos help-mode dired-x compile-my
mode-line compile article gnus-cite dl
bookmarks w3m-bookmark file-write-to erc-my
fill-new erc-dcc quit gnus-my group
group-summary gnus-srvr gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig mailcap
gnus-sum nnoo gnus-group gnus-undo nnmail
mail-source gnus-start gnus-spec gnus-int
gnus-range gnus-win message rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus gnus-ems nnheader
mail-utils info slime-autoloads package
epg-config edit dired-my w3m-my w3m-tabs
cus-edit cus-start cus-load w3m-session w3m
browse-url doc-view jka-compr image-mode
timezone w3m-hist w3m-fb bookmark-w3m w3m-ems
w3m-ccl ccl w3m-favicon w3m-image w3m-proc
w3m-util files-my tramp tramp-compat
tramp-loaddefs trampver shell advice
sudo-user-path find-command window-new count
get-search-string caps-back buffer-menu
switch-to-buffer super scroll error dired close
buc switch-to-buffer-regexp subr-x align-new
erc-list erc-menu easymenu erc-join erc-ring
erc-networks erc-pcomplete pcomplete comint
ansi-color ring erc-track erc-match erc-button
wid-edit cl-loaddefs cl-lib erc-fill erc-stamp
erc-netsplit erc-goodies erc erc-backend
erc-compat format-spec auth-source eieio
byte-opt bytecomp byte-compile cconv eieio-core
gnus-util time-date mm-util help-fns mail-prsvr
password-cache thingatpt pp my-string
search-regexp-in-files w3m-load tooltip
electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer nadvice loaddefs button faces
cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable
backquote make-network-process dbusbind
gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit
x multi-tty emacs)

Memory information:
((conses 8 549625 85419)
 (symbols 24 51306 0)
 (miscs 20 815 4121)
 (strings 16 137215 11969)
 (string-bytes 1 4128153)
 (vectors 8 52408)
 (vector-slots 4 1667499 44276)
 (floats 8 422 723)
 (intervals 28 10530 1390)
 (buffers 512 190)
 (heap 1024 39263 71394))

-- 
underground experts united
http://user.it.uu.se/~embe8573

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Robert Pluim]

Emanuel Berg <moasen@zoho.com> writes:

> This does, over and over again:
>
>     gnutls.c: [3] ASSERT: gnutls_buffers.c:547
>
>     gnutls.c: [3] (Emacs) retry: Resource
> temporarily unavailable, try again. gnutls.c:
> [1] (Emacs) non-fatal error: Resource
> temporarily unavailable, try again. gnutls.c:
> [3] ASSERT: gnutls_buffers.c:1104
>
>> The bad news is that I tried this:
>> gnutls-cli --port 995 pop.zoho.com
>> And it connected just fine. You might try
>> that, and going through with the
>> login procedure.
>

You're using the builtin GNUTLS support. There have been a number of
fixes to it over the years, so it could be worth trying emacs-26. If
that's not possible, the alternative is to switch to using the
external gnutls-cli command instead. Something like

(defun gnutls-available-p ()
       nil)

before you load gnus should do the trick.

Regards

Robert

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Emanuel Berg]

Robert Pluim wrote:

> You're using the builtin GNUTLS support.
> There have been a number of fixes to it over
> the years, so it could be worth trying
> emacs-26. If that's not possible, the
> alternative is to switch to using the
> external gnutls-cli command instead.
> Something like
>
> (defun gnutls-available-p () nil)

OK, I'll try both suggestions and get back to
you on gmane.emacs.gnus.general !

-- 
underground experts united
http://user.it.uu.se/~embe8573

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Emanuel Berg]

Robert Pluim wrote:

> (defun gnutls-available-p () nil)

This doesn't seem to have solved the problem

    Opening TLS connection to `pop.zoho.com'...
    Opening TLS connection with `gnutls-cli --insecure -p 995 pop.zoho.com'...done
    Opening TLS connection to `pop.zoho.com'...done
    Mail source (pop :user ABC :password XYZ :server pop.zoho.com :port 995 :stream ssl) failed: (error USER ABC not valid)

-- 
underground experts united
http://user.it.uu.se/~embe8573

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Noam Postavsky]

Emanuel Berg <moasen@zoho.com> writes:

>> (defun gnutls-available-p () nil)
>
> This doesn't seem to have solved the problem
>
>     Opening TLS connection to `pop.zoho.com'...
>     Opening TLS connection with `gnutls-cli --insecure -p 995 pop.zoho.com'...done
>     Opening TLS connection to `pop.zoho.com'...done
>     Mail source (pop :user ABC :password XYZ :server pop.zoho.com :port 995 :stream ssl) failed: (error USER ABC not valid)

You may also need to customize `tls-program' to use the `openssl'
command.  Although I think external-program based tls is kind of flaky,
so I don't how much you can really expect from it.

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Robert Pluim]

Emanuel Berg <moasen@zoho.com> writes:

> Robert Pluim wrote:
>
>> (defun gnutls-available-p () nil)
>
> This doesn't seem to have solved the problem
>
>     Opening TLS connection to `pop.zoho.com'...
>     Opening TLS connection with `gnutls-cli --insecure -p 995 pop.zoho.com'...done
>     Opening TLS connection to `pop.zoho.com'...done
>     Mail source (pop :user ABC :password XYZ :server pop.zoho.com :port 995 :stream ssl) failed: (error USER ABC not valid)

That looks like your username/password is being rejected by the remote
POP server. There may be a buffer somewhere showing exactly what the
server said.

Robert

bug#28971: 24.4; Gnus TLS/SSL POP issue

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Emanuel Berg <moasen@zoho.com> writes:
>
>> Robert Pluim wrote:
>>
>>> (defun gnutls-available-p () nil)
>>
>> This doesn't seem to have solved the problem
>>
>>     Opening TLS connection to `pop.zoho.com'...
>>     Opening TLS connection with `gnutls-cli --insecure -p 995 pop.zoho.com'...done
>>     Opening TLS connection to `pop.zoho.com'...done
>>     Mail source (pop :user ABC :password XYZ :server pop.zoho.com :port 995 :stream ssl) failed: (error USER ABC not valid)
>
> That looks like your username/password is being rejected by the remote
> POP server. There may be a buffer somewhere showing exactly what the
> server said.

This doesn't seem to be an Emacs issue, so I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no