bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgment at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':

I am using gnutls within the vm (viewmail) mail reader (vm-version
8.2.0b) to read email over IMAP-SSL under nt Emacs 24.2.50.1 in
Windows 7

Emacs gives me the following error in the message line before immediately
crashing the entire emacs process:
GnuTLS error: #<proces IMAP over SSL>, -64

The Windows crash both then gives me the following error:
Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:		emacs.exe
  Application Version:	24.2.50.0
  Application Timestamp:		5037d090
  Fault Module Name:			libgnutls-28.dll
  Fault Module Version:			0.0.0.0
  Fault Module Timestamp:		501d8392
  Exception Code:				c0000005
  Exception Offset:				0000657e
  OS Version:					6.1.7601.2.1.0.768.3
  Locale ID:					1033
  Additional Information 1:		0a9e
  Additional Information 2:		0a9e372d3b4ad19135b953a78882e789
  Additional Information 3:		0a9e
  Additional Information 4:		0a9e372d3b4ad19135b953a78882e789

I do not believe this is a problem with vm (viewmail) since all works
well when I use an external program (stunnel) to create the SSL
tunnel. The problem only happens when I use the internal GNUtls
capability to create the SSL tunnel.

Even so, it is surprising to me that the entire Emacs app crashes - I
have beena heavy Emacs user since 1984 and I don't believe I have ever
had it crash! (kudos to emacs for reliability)


Debugging wasn't easy since I needed to 'catch' it right before the
crash and set a breakpoint there. It turns out that it crashes when
evaluating the declared c-function "gnutls-boot" which is an argument
to "gnutls-message-maybe" called from the function gnutls-negotiate in
gnutls.el

  (gnutls-boot process type params)
where:
		process = #<process IMAP over SSL>
		type    = gnutls-x509pki
		params = (:priority "NORMAL" :hostname "imap.mydomain.org" :loglevel 0 :min-prime-bits 256 :trustfiles ("/usr/ssl/certs/ca-bundle.crt") :crlfiles nil :keylist nil :verify-flags nil :verify-error nil :verify-hostname-error nil :callbacks nil)
  
  (gnutls-boot gnutls-x509pki

Here is what I got from the Emacs debugger:

* (gnutls-boot process type params)
* (setq ret (gnutls-boot process type params))
* (gnutls-message-maybe (setq ret (gnutls-boot process type params)) "boot: %s" params)
  (let* ((type (or type (quote gnutls-x509pki))) (trustfiles (or trustfiles (delq nil (mapcar (lambda (f) (and f ... f)) (if (functionp gnutls-trustfiles) (funcall gnutls-trustfiles) gnutls-trustfiles))))) (priority-string (or priority-string (cond ((eq type (quote gnutls-anon)) "NORMAL:+ANON-DH:!ARCFOUR-128") ((eq type (quote gnutls-x509pki)) (if gnutls-algorithm-priority (upcase gnutls-algorithm-priority) "NORMAL"))))) (min-prime-bits (or min-prime-bits gnutls-min-prime-bits)) (params (\` (:priority (\, priority-string) :hostname (\, hostname) :loglevel (\, gnutls-log-level) :min-prime-bits (\, min-prime-bits) :trustfiles (\, trustfiles) :crlfiles (\, crlfiles) :keylist (\, keylist) :verify-flags (\, :min-prime-bits (\, min-prime-bits) :trustfiles (\, trustfiles) :crlfiles (\, crlfiles) :keylist (\, keylist) :verify-flags (\, verify-flags) :verify-error (\, verify-error) :verify-hostname-error (\, verify-hostname-error) :callbacks nil))) ret) (edebug) (gnutls-message-maybe (setq ret (gnutls-boot process type params)) "boot: %s" params) (when (gnutls-errorp ret) (signal (quote gnutls-error) (list process ret))) process))
  (cl--block-wrapper (catch (quote --cl-block-gnutls-negotiate--) (let* ((type (or type (quote gnutls-x509pki))) (trustfiles (or trustfiles (delq nil (mapcar ... ...)))) (priority-string (or priority-string (cond (... "NORMAL:+ANON-DH:!ARCFOUR-128") (... ...)))) (min-prime-bits (or min-prime-bits gnutls-min-prime-bits)) (params (\` (:priority (\, priority-string) :hostname (\, hostname) :loglevel (\, gnutls-log-level) :min-prime-bits (\, min-prime-bits) :trustfiles (\, trustfiles) :crlfiles (\, crlfiles) :keylist (\, keylist) :verify-flags (\, verify-flags) :verify-error (\, verify-error) :verify-hostname-error (\, verify-hostname-error) :callbacks nil))) ret) (edebug) (gnutls-message-maybe (setq ret (gnutls-boot process type params)) "boot: %s" params) (when (gnutls-errorp ret) (signal (quote gnutls-error) (list process ret))) process)))
  (cl-block gnutls-negotiate (let* ((type (or type (quote gnutls-x509pki))) (trustfiles (or trustfiles (delq nil (mapcar (lambda ... ...) (if ... ... gnutls-trustfiles))))) (priority-string (or priority-string (cond ((eq type ...) "NORMA spec)))) (hostname (car (cdr (memq (quote :hostname) spec)))) (priority-string (car (cdr (memq (quote :priority-string) spec)))) (trustfiles (car (cdr (memq (quote :trustfiles) spec)))) (crlfiles (car (cdr (memq (quote :crlfiles) spec)))) (keylist (car (cdr (memq (quote :keylist) spec)))) (min-prime-bits (car (cdr (memq (quote :min-prime-bits) spec)))) (verify-flags (car (cdr (memq (quote :verify-flags) spec)))) (verify-error (car (cdr (memq (quote :verify-error) spec)))) (verify-hostname-error (car (cdr (memq (quote :verify-hostname-error) spec))))) (cl-block gnutls-negotiate (let* ((type (or  gnutls-negotiate(:process #<process IMAP over SSL> :type gnutls-x509pki :hostname "imap.mydomain.org")
  open-gnutls-stream("IMAP over SSL" #<buffer trace of IMAP over SSL session to imap.mydomain.org at 14:12:31> "imap.mydomain.org" 993)
  funcall(open-gnutls-stream "IMAP over SSL" #<buffer trace of IMAP over SSL session to ko  (  (  (with-  network-stream-open-tls("IMAP over SSL" #<buffer trace of IMAP over SSL session to imap.mydomain.org at 14:12:31> "imap.mydomain.org" 993 (:type tls))
  funcall(network-stream-open-tls "IMAP over SSL" #<buffer trace of IMAP over SSL session   (unwind-pr  (let ((work-buffer (or buffer (generate-new-buffer " *stream buffer*"))) (fun (cond ((and (e  (if (and (not return-list) (or (eq  (let ((type (plist-get parameters :type)) (return-list (plist-get parameters :return-list))) (if (and (not return-list) (or (eq type (quote plain)) (an  open-network-stream("IMAP over SSL" #<buffer trace of IMAP over SSL session to imap.mydomain.org at 14:12:31> "  byte-code("203+L:+ANON-DH:!ARCFOUR-128") ((eq type ...) (if gnutls-algorithm-priority ... "NORMAL"))))) (min-prime-bits (or min-prime-bits gnutls-min-prime-bits)) (params (\` (:priority (\, priority-string) :hostname (\, hostname) :loglevel (\, gnutls-log-level) :min-prime-bits (\, min-prime-bits) :trustfiles (\, trustfiles) :crlfiles (\, crlfiles) :keylist (\, keylist) :verify-flags (\, verify-flags) :verify-error (\, verify-error) :verify-hostname-error (\, verify-hostname-error) :callbacks nil))) ret) (edebug) (gnutls-message-maybe (setq ret (gnutls-boot process type params)) "boot: %s" params) (when (gnutls-errorp ret) (signal (quote gnutls-error) (list process ret))) process))
  (let* ((process (car (cdr (memq (quote :process) spec)))) (type (car (cdr (memq (quote :type) verify-flags) :verify-error (\, verify-error) :verify-hostname-error (\, verify-hostname-error) :callbacks nil))) ret) (edebug) (gnutls-message-maybe (setq ret (gnutls-boot process type params)) "boot: %s" params) (when (gnutls-errorp ret) (signal (quote gnutls-error) (list process ret))) process)
  (catch (quote --cl-block-gnutls-negotiate--) (let* ((type (or type (quote gnutls-x509pki))) (trustfiles (or trustfiles (delq nil (mapcar (lambda ... ...) (if ... ... gnutls-trustfiles))))) (priority-string (or priority-string (cond ((eq type ...) "NORMAL:+ANON-DH:!ARCFOUR-128") ((eq type ...) (if gnutls-algorithm-priority ... "NORMAL"))))) (min-prime-bits (or min-prime-bits gnutls-min-prime-bits)) (params (\` (:priority (\, priority-string) :hostname (\, hostname) :loglevel (\, gnutls-log-level)

----------------------------------

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
c:/MyPrograms/Emacs/etc/DEBUG.


In GNU Emacs 24.2.50.1 (i386-mingw-nt6.1.7601)
 of 2012-08-24 on YAMALOK
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --with-gcc (4.7) --cflags -m32 -O2 -g0 -march=prescott
 -mtune=prescott -pipe -IC:/gnuwin32/emacs/include
 -IC:/gnuwin32/emacs/lib -IC:/gnuwin32/src -IC:/gnutls/include
 -IC:/gnutls/lib -IC:/gnutls/bin -IC:/libxml2/include -IC:/libxml2/lib
 -IC:/libxml2/bin --ldflags '

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: VM Summary

Minor modes in effect:
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <escape> x r e p <tab> o r <tab> <return> 
s t u n n e l SPC <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> SPC <backspace> <backspace> 
<down-mouse-1> <mouse-1> C-h f g n u t <tab> <tab> 
<tab> a v a i <tab> <return> <help-echo> <help-echo> 
<down-mouse-1> <mouse-2> C-h C-g <escape> x C-g <C-down-mouse-1> 
<drag-mouse-1> <escape> x v m <return> C-g C-h f v 
m <backspace> m <backspace> m - - s t <tab> u <tab> 
<tab> C-g C-h v v m - s t <tab> u <tab> p <tab> <return> 
C-p <escape> x e m a c s = r <backspace> <backspace> 
- r e p <tab> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
r e p o <tab> r t - e m <tab> <return>

Recent messages:
c:/myname/bin/qp-decode.exe exited non-zero (code 123)
Parsing BBDB... (frobnicating...done)
inbox: Checking for new mail...
Decrypting c:/myname/emacs/.authinfo.gpg...0%
Opening input file: Can't decrypt, Cancelled; Exit
Making completion list... [2 times]
Quit
Making completion list...
Type "q" in help window to restore its previous buffer.
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort pp vm-save tapestry vm-crypto vm-imap jjk-vm dired
vm-mime-display-internal-application vm-ps-print bbdb-vm vm-autoload
bbdb-snarf mail-extr bbdb-autoloads bbdb-hooks bbdb-com cl cl-lib vcard
vm-vcard vm-pine smtpmail bbdb timezone vm-rfaddons vm-menu vm-window
vm-toolbar vm-folder vm-mime vm-undo vm-virtual vm-summary-faces
vm-summary vm-mouse vm-page vm-motion vm-minibuf vm-message vm-misc
vm-macro vm-autoloads vm-vars vm-version vm find-func emacsbug message
idna format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mail-utils help-mode easymenu jjk-comments jjk-load
ps-print ps-def lpr jjk-print ibm-keymaps jjk-frames jjk-hooks
jjk-keymaps ehelp electric uniquify warnings arc-mode archive-mode
jjk-lib epa-file epa derived epg epg-config advice help-fns
advice-preload auth-source eieio byte-opt bytecomp byte-compile cconv
macroexp gnus-util mm-util mail-prsvr password-cache cygwin-mount
ange-ftp comint ansi-color ring server time time-date tooltip ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 disp-table ls-lisp w32-win
w32-vars tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer button faces cus-face files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty emacs)

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Glenn Morris]

> In GNU Emacs 24.2.50.1 (i386-mingw-nt6.1.7601)
>  of 2012-08-24 on YAMALOK

There's zero reason to use that version instead of the 24.3 release.
Please upgrade to 24.3 and let us know if there is still a problem.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Glenn Morris wrote at about 15:38:20 -0400 on Friday, October 18, 2013:
 > 
 > > In GNU Emacs 24.2.50.1 (i386-mingw-nt6.1.7601)
 > >  of 2012-08-24 on YAMALOK
 > 
 > There's zero reason to use that version instead of the 24.3 release.
 > Please upgrade to 24.3 and let us know if there is still a problem.

Well perhaps not zero reason... 24.2.50.1 is the latest version available on
the sourceforge site and it is the only packaged executable version of
emacs for Windows I found that had the gnutls dll's already included...

That being said, I downloaded 24.3 from the gnu site and tried it with
the latest gnutls-3.0.9  dll's from the suggested
http://sourceforge.net/projects/ezwinports/files/ site.

And SAME crash of emacs!

Not sure what else I can do to troubleshoot this further...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 

>   Fault Module Name:			libgnutls-28.dll
...
> That being said, I downloaded 24.3 from the gnu site and tried it with
> the latest gnutls-3.0.9  dll's from the suggested
> http://sourceforge.net/projects/ezwinports/files/ site.

> And SAME crash of emacs!

Could you please try with the latest 3.x from
ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?

The `gnutls-boot' function is pretty inoffensive so let's make sure the
basics are covered before we dig into the C code.

Thanks
Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 > On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 > 
 > >   Fault Module Name:			libgnutls-28.dll
 > ...
 > > That being said, I downloaded 24.3 from the gnu site and tried it with
 > > the latest gnutls-3.0.9  dll's from the suggested
 > > http://sourceforge.net/projects/ezwinports/files/ site.
 > 
 > > And SAME crash of emacs!
 > 
 > Could you please try with the latest 3.x from
 > ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 > 
Same crash with 3.2.4 as with 3.0.9

 > The `gnutls-boot' function is pretty inoffensive so let's make sure the
 > basics are covered before we dig into the C code.
 > 
 > Thanks
 > Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 

> Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
>> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
>> 
>> >   Fault Module Name:			libgnutls-28.dll
>> ...
>> > That being said, I downloaded 24.3 from the gnu site and tried it with
>> > the latest gnutls-3.0.9  dll's from the suggested
>> > http://sourceforge.net/projects/ezwinports/files/ site.
>> 
>> > And SAME crash of emacs!
>> 
>> Could you please try with the latest 3.x from
>> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
>> 
> Same crash with 3.2.4 as with 3.0.9

Sorry, I don't know how you can be sure of the library loaded in
Windows.  Did you put the DLL in the same directory?

>> The `gnutls-boot' function is pretty inoffensive so let's make sure the
>> basics are covered before we dig into the C code.

Are you able to debug the Emacs executable?  I don't know if the build
you're using includes the necessary symbols, but a backtrace with at
least function names would be extremely helpful to figure this out.

Does the problem occur to any server or just the one you listed?  You
can test with

 (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")

It's strange that there's been no other reports of this issue.  Do you
have access to any other systems where you can test?

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 09:27:06 -0400 on Tuesday, October 22, 2013:
 > On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
 > 
 > > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 > >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 > >> 
 > >> >   Fault Module Name:			libgnutls-28.dll
 > >> ...
 > >> > That being said, I downloaded 24.3 from the gnu site and tried it with
 > >> > the latest gnutls-3.0.9  dll's from the suggested
 > >> > http://sourceforge.net/projects/ezwinports/files/ site.
 > >> 
 > >> > And SAME crash of emacs!
 > >> 
 > >> Could you please try with the latest 3.x from
 > >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 > >> 
 > > Same crash with 3.2.4 as with 3.0.9
 > 
 > Sorry, I don't know how you can be sure of the library loaded in
 > Windows.  Did you put the DLL in the same directory?

The dll's are in a directory that lies within my system PATH.
I know they are loaded because (gnutls-available-p) only proves true
when I have all the related dll's in the PATH.

 > 
 > >> The `gnutls-boot' function is pretty inoffensive so let's make sure the
 > >> basics are covered before we dig into the C code.
 > 
 > Are you able to debug the Emacs executable?  I don't know if the build
 > you're using includes the necessary symbols, but a backtrace with at
 > least function names would be extremely helpful to figure this out.

I just used edebug... for the trace I submitted...
 > 
 > Does the problem occur to any server or just the one you listed?  You
 > can test with
 > 
 >  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
 > 

This gave the same crash!

 > It's strange that there's been no other reports of this issue.  Do you
 > have access to any other systems where you can test?

Could it have anything to do with 64-bit Win7?
Could there be any conflicts with cygwin x86_64 that I have installed
(though I purposely didn't install cygwin gnutls) -- but could other
dll's be in conflict?

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

emacs@kosowsky.org wrote at about 11:23:59 -0400 on Tuesday, October 22, 2013:
 > Ted Zlatanov wrote at about 09:27:06 -0400 on Tuesday, October 22, 2013:
 >  > On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
 >  > 
 >  > > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 >  > >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 >  > >> 
 >  > >> >   Fault Module Name:			libgnutls-28.dll
 >  > >> ...
 >  > >> > That being said, I downloaded 24.3 from the gnu site and tried it with
 >  > >> > the latest gnutls-3.0.9  dll's from the suggested
 >  > >> > http://sourceforge.net/projects/ezwinports/files/ site.
 >  > >> 
 >  > >> > And SAME crash of emacs!
 >  > >> 
 >  > >> Could you please try with the latest 3.x from
 >  > >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 >  > >> 
 >  > > Same crash with 3.2.4 as with 3.0.9
 >  > 
 >  > Sorry, I don't know how you can be sure of the library loaded in
 >  > Windows.  Did you put the DLL in the same directory?
 > 
 > The dll's are in a directory that lies within my system PATH.
 > I know they are loaded because (gnutls-available-p) only proves true
 > when I have all the related dll's in the PATH.
 > 
 >  > 
 >  > >> The `gnutls-boot' function is pretty inoffensive so let's make sure the
 >  > >> basics are covered before we dig into the C code.
 >  > 
 >  > Are you able to debug the Emacs executable?  I don't know if the build
 >  > you're using includes the necessary symbols, but a backtrace with at
 >  > least function names would be extremely helpful to figure this out.
 > 
 > I just used edebug... for the trace I submitted...
 >  > 
 >  > Does the problem occur to any server or just the one you listed?  You
 >  > can test with
 >  > 
 >  >  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
 >  > 
 > 
 > This gave the same crash!
 > 
 >  > It's strange that there's been no other reports of this issue.  Do you
 >  > have access to any other systems where you can test?
 > 
 > Could it have anything to do with 64-bit Win7?
 > Could there be any conflicts with cygwin x86_64 that I have installed
 > (though I purposely didn't install cygwin gnutls) -- but could other
 > dll's be in conflict?

OK - I answered my own question.
There seems to be an incompatibility with the cygwin-mount library...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> Date: Tue, 22 Oct 2013 09:27:06 -0400
> Cc: 15648@debbugs.gnu.org
> 
> On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
> 
> > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
> >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
> >> 
> >> >   Fault Module Name:			libgnutls-28.dll
> >> ...
> >> > That being said, I downloaded 24.3 from the gnu site and tried it with
> >> > the latest gnutls-3.0.9  dll's from the suggested
> >> > http://sourceforge.net/projects/ezwinports/files/ site.
> >> 
> >> > And SAME crash of emacs!
> >> 
> >> Could you please try with the latest 3.x from
> >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
> >> 
> > Same crash with 3.2.4 as with 3.0.9
> 
> Sorry, I don't know how you can be sure of the library loaded in
> Windows.

Like this:

  M-: (get 'gnutls :loaded-from) RET

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

emacs@kosowsky.org wrote at about 11:41:09 -0400 on Tuesday, October 22, 2013:
 > emacs@kosowsky.org wrote at about 11:23:59 -0400 on Tuesday, October 22, 2013:
 >  > Ted Zlatanov wrote at about 09:27:06 -0400 on Tuesday, October 22, 2013:
 >  >  > On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
 >  >  > 
 >  >  > > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 >  >  > >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 >  >  > >> 
 >  >  > >> >   Fault Module Name:			libgnutls-28.dll
 >  >  > >> ...
 >  >  > >> > That being said, I downloaded 24.3 from the gnu site and tried it with
 >  >  > >> > the latest gnutls-3.0.9  dll's from the suggested
 >  >  > >> > http://sourceforge.net/projects/ezwinports/files/ site.
 >  >  > >> 
 >  >  > >> > And SAME crash of emacs!
 >  >  > >> 
 >  >  > >> Could you please try with the latest 3.x from
 >  >  > >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 >  >  > >> 
 >  >  > > Same crash with 3.2.4 as with 3.0.9
 >  >  > 
 >  >  > Sorry, I don't know how you can be sure of the library loaded in
 >  >  > Windows.  Did you put the DLL in the same directory?
 >  > 
 >  > The dll's are in a directory that lies within my system PATH.
 >  > I know they are loaded because (gnutls-available-p) only proves true
 >  > when I have all the related dll's in the PATH.
 >  > 
 >  >  > 
 >  >  > >> The `gnutls-boot' function is pretty inoffensive so let's make sure the
 >  >  > >> basics are covered before we dig into the C code.
 >  >  > 
 >  >  > Are you able to debug the Emacs executable?  I don't know if the build
 >  >  > you're using includes the necessary symbols, but a backtrace with at
 >  >  > least function names would be extremely helpful to figure this out.
 >  > 
 >  > I just used edebug... for the trace I submitted...
 >  >  > 
 >  >  > Does the problem occur to any server or just the one you listed?  You
 >  >  > can test with
 >  >  > 
 >  >  >  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
 >  >  > 
 >  > 
 >  > This gave the same crash!
 >  > 
 >  >  > It's strange that there's been no other reports of this issue.  Do you
 >  >  > have access to any other systems where you can test?
 >  > 
 >  > Could it have anything to do with 64-bit Win7?
 >  > Could there be any conflicts with cygwin x86_64 that I have installed
 >  > (though I purposely didn't install cygwin gnutls) -- but could other
 >  > dll's be in conflict?
 > 
 > OK - I answered my own question.
 > There seems to be an incompatibility with the cygwin-mount library...

In particular, the problem seems to be with the called cygwin mount
program (/bin/mount.exe). Replacing it with /bin/true.exe prevents the
crash.

I wonder whether this is a dll incompatibility with cygwin.dll or in
particular the x86_64 version...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Tue, 22 Oct 2013 18:43:44 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ>   M-: (get 'gnutls :loaded-from) RET

Oh, thank you, very handy.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Tue, 22 Oct 2013 15:10:29 -0400 <emacs@kosowsky.org> wrote: 

>> OK - I answered my own question.
>> There seems to be an incompatibility with the cygwin-mount library...

> In particular, the problem seems to be with the called cygwin mount
> program (/bin/mount.exe). Replacing it with /bin/true.exe prevents the
> crash.

> I wonder whether this is a dll incompatibility with cygwin.dll or in
> particular the x86_64 version...

Nice detective work.  I don't know why it happens, sorry.  The GnuTLS
project has a mailing list; maybe there someone will know.  There must
be something mixed up between the two DLLs.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 16:06:07 -0400 on Tuesday, October 22, 2013:
 > On Tue, 22 Oct 2013 15:10:29 -0400 <emacs@kosowsky.org> wrote: 
 > 
 > >> OK - I answered my own question.
 > >> There seems to be an incompatibility with the cygwin-mount library...
 > 
 > > In particular, the problem seems to be with the called cygwin mount
 > > program (/bin/mount.exe). Replacing it with /bin/true.exe prevents the
 > > crash.
 > 
 > > I wonder whether this is a dll incompatibility with cygwin.dll or in
 > > particular the x86_64 version...
 > 
 > Nice detective work.  I don't know why it happens, sorry.  The GnuTLS
 > project has a mailing list; maybe there someone will know.  There must
 > be something mixed up between the two DLLs.
 > 
 > Ted

I will follow up...

Still, I'm surprised that Emacs itself would crash so violently. I
have been using Emacs for 30 years and I can't recall ever crashing
Emacs itself (except maybe ~25 years ago when people started first
(unofficially) porting Emacs to X10/X11). Even if there is an external
dll conflict, I would think that at worse it should return an error,
but should not crash emacs -- certainly since gnutls is now included
in Emacs, I would think that it should be crash protected...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Tue, 22 Oct 2013 16:22:58 -0400
> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
> Still, I'm surprised that Emacs itself would crash so violently. I
> have been using Emacs for 30 years and I can't recall ever crashing
> Emacs itself (except maybe ~25 years ago when people started first
> (unofficially) porting Emacs to X10/X11). Even if there is an external
> dll conflict, I would think that at worse it should return an error,
> but should not crash emacs -- certainly since gnutls is now included
> in Emacs, I would think that it should be crash protected...

If you show some data about the crash, from the C level (where the
crash happens), maybe we could figure out how to avoid this in the
future.  So far, you have only shown data from the Lisp level, which
doesn't help, because the crash is not in Lisp.

Without meaningful data about the crash, there's no hope we could do
anything about that.

Thanks in advance.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Andy Moreton]

On Tue 22 Oct 2013, Eli Zaretskii wrote:

>> Date: Tue, 22 Oct 2013 09:27:06 -0400
>> Cc: 15648@debbugs.gnu.org
>> 
>> On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
>> 
>> > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
>> >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
>> >> 
>> >> >   Fault Module Name:			libgnutls-28.dll
>> >> ...
>> >> > That being said, I downloaded 24.3 from the gnu site and tried it with
>> >> > the latest gnutls-3.0.9  dll's from the suggested
>> >> > http://sourceforge.net/projects/ezwinports/files/ site.
>> >> 
>> >> > And SAME crash of emacs!
>> >> 
>> >> Could you please try with the latest 3.x from
>> >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
>> >> 
>> > Same crash with 3.2.4 as with 3.0.9
>> 
>> Sorry, I don't know how you can be sure of the library loaded in
>> Windows.
>
> Like this:
>
>   M-: (get 'gnutls :loaded-from) RET

Thanks for this Eli - it's not obvious that the full path is available
from lisp. It would be useful if it was shown in the output of
(list-dynamic-libraries).

    AndyM

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 22 Oct 2013 21:35:44 +0100
> 
> >   M-: (get 'gnutls :loaded-from) RET
> 
> Thanks for this Eli - it's not obvious that the full path is available
> from lisp. It would be useful if it was shown in the output of
> (list-dynamic-libraries).

Patches are welcome.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

emacs@kosowsky.org wrote at about 15:10:29 -0400 on Tuesday, October 22, 2013:
 > emacs@kosowsky.org wrote at about 11:41:09 -0400 on Tuesday, October 22, 2013:
 >  > emacs@kosowsky.org wrote at about 11:23:59 -0400 on Tuesday, October 22, 2013:
 >  >  > Ted Zlatanov wrote at about 09:27:06 -0400 on Tuesday, October 22, 2013:
 >  >  >  > On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
 >  >  >  > 
 >  >  >  > > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 >  >  >  > >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 >  >  >  > >> 
 >  >  >  > >> >   Fault Module Name:			libgnutls-28.dll
 >  >  >  > >> ...
 >  >  >  > >> > That being said, I downloaded 24.3 from the gnu site and tried it with
 >  >  >  > >> > the latest gnutls-3.0.9  dll's from the suggested
 >  >  >  > >> > http://sourceforge.net/projects/ezwinports/files/ site.
 >  >  >  > >> 
 >  >  >  > >> > And SAME crash of emacs!
 >  >  >  > >> 
 >  >  >  > >> Could you please try with the latest 3.x from
 >  >  >  > >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 >  >  >  > >> 
 >  >  >  > > Same crash with 3.2.4 as with 3.0.9
 >  >  >  > 
 >  >  >  > Sorry, I don't know how you can be sure of the library loaded in
 >  >  >  > Windows.  Did you put the DLL in the same directory?
 >  >  > 
 >  >  > The dll's are in a directory that lies within my system PATH.
 >  >  > I know they are loaded because (gnutls-available-p) only proves true
 >  >  > when I have all the related dll's in the PATH.
 >  >  > 
 >  >  >  > 
 >  >  >  > >> The `gnutls-boot' function is pretty inoffensive so let's make sure the
 >  >  >  > >> basics are covered before we dig into the C code.
 >  >  >  > 
 >  >  >  > Are you able to debug the Emacs executable?  I don't know if the build
 >  >  >  > you're using includes the necessary symbols, but a backtrace with at
 >  >  >  > least function names would be extremely helpful to figure this out.
 >  >  > 
 >  >  > I just used edebug... for the trace I submitted...
 >  >  >  > 
 >  >  >  > Does the problem occur to any server or just the one you listed?  You
 >  >  >  > can test with
 >  >  >  > 
 >  >  >  >  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
 >  >  >  > 
 >  >  > 
 >  >  > This gave the same crash!
 >  >  > 
 >  >  >  > It's strange that there's been no other reports of this issue.  Do you
 >  >  >  > have access to any other systems where you can test?
 >  >  > 
 >  >  > Could it have anything to do with 64-bit Win7?
 >  >  > Could there be any conflicts with cygwin x86_64 that I have installed
 >  >  > (though I purposely didn't install cygwin gnutls) -- but could other
 >  >  > dll's be in conflict?
 >  > 
 >  > OK - I answered my own question.
 >  > There seems to be an incompatibility with the cygwin-mount library...
 > 
 > In particular, the problem seems to be with the called cygwin mount
 > program (/bin/mount.exe). Replacing it with /bin/true.exe prevents the
 > crash.
 > 
 > I wonder whether this is a dll incompatibility with cygwin.dll or in
 > particular the x86_64 version...

I did some more troubleshooting... actually, I don't think it is a
cygwin.dll problem nor is it a problem with the cygwin program
mount.exe.


The problem seems to be with the lisp code in cygwin-mount-activate in
terms of how it changes file-name-handler-alist,
substitute-in-file-name, and expand-file-name.

I say this because gnutls causes the crash after
(cygwin-mount-activate) is run. But if you wait to run
(cygwin-mount-deactivate), then it won't crash. So the problem isn't
with whether mount.exe has been run (I also double checked this by
manually running mount.exe via call-process). Rather, it has something
to do with the file name substitutions set up.

So, while the physical crash may be somewhere in the lisp code. The
issue is with how the cygwin file name handler lisp code interacts with the
gnutls code...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Tue, 22 Oct 2013 18:27:23 -0400
> Cc: Ted Zlatanov <tzz@lifelogs.com>, 15648@debbugs.gnu.org
> 
> I did some more troubleshooting... actually, I don't think it is a
> cygwin.dll problem nor is it a problem with the cygwin program
> mount.exe.
> 
> 
> The problem seems to be with the lisp code in cygwin-mount-activate in
> terms of how it changes file-name-handler-alist,
> substitute-in-file-name, and expand-file-name.
> 
> I say this because gnutls causes the crash after
> (cygwin-mount-activate) is run. But if you wait to run
> (cygwin-mount-deactivate), then it won't crash. So the problem isn't
> with whether mount.exe has been run (I also double checked this by
> manually running mount.exe via call-process). Rather, it has something
> to do with the file name substitutions set up.
> 
> So, while the physical crash may be somewhere in the lisp code. The
> issue is with how the cygwin file name handler lisp code interacts with the
> gnutls code...

You never explained how is cygwin-mount relevant to gnutls in this
case.  Could you please do that?

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

emacs@kosowsky.org wrote at about 18:27:23 -0400 on Tuesday, October 22, 2013:
 > emacs@kosowsky.org wrote at about 15:10:29 -0400 on Tuesday, October 22, 2013:
 >  > emacs@kosowsky.org wrote at about 11:41:09 -0400 on Tuesday, October 22, 2013:
 >  >  > emacs@kosowsky.org wrote at about 11:23:59 -0400 on Tuesday, October 22, 2013:
 >  >  >  > Ted Zlatanov wrote at about 09:27:06 -0400 on Tuesday, October 22, 2013:
 >  >  >  >  > On Mon, 21 Oct 2013 15:30:40 -0400 <emacs@kosowsky.org> wrote: 
 >  >  >  >  > 
 >  >  >  >  > > Ted Zlatanov wrote at about 10:22:03 -0400 on Monday, October 21, 2013:
 >  >  >  >  > >> On Fri, 18 Oct 2013 14:29:04 -0400 "" <emacs@kosowsky.org> wrote: 
 >  >  >  >  > >> 
 >  >  >  >  > >> >   Fault Module Name:			libgnutls-28.dll
 >  >  >  >  > >> ...
 >  >  >  >  > >> > That being said, I downloaded 24.3 from the gnu site and tried it with
 >  >  >  >  > >> > the latest gnutls-3.0.9  dll's from the suggested
 >  >  >  >  > >> > http://sourceforge.net/projects/ezwinports/files/ site.
 >  >  >  >  > >> 
 >  >  >  >  > >> > And SAME crash of emacs!
 >  >  >  >  > >> 
 >  >  >  >  > >> Could you please try with the latest 3.x from
 >  >  >  >  > >> ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ ?
 >  >  >  >  > >> 
 >  >  >  >  > > Same crash with 3.2.4 as with 3.0.9
 >  >  >  >  > 
 >  >  >  >  > Sorry, I don't know how you can be sure of the library loaded in
 >  >  >  >  > Windows.  Did you put the DLL in the same directory?
 >  >  >  > 
 >  >  >  > The dll's are in a directory that lies within my system PATH.
 >  >  >  > I know they are loaded because (gnutls-available-p) only proves true
 >  >  >  > when I have all the related dll's in the PATH.
 >  >  >  > 
 >  >  >  >  > 
 >  >  >  >  > >> The `gnutls-boot' function is pretty inoffensive so let's make sure the
 >  >  >  >  > >> basics are covered before we dig into the C code.
 >  >  >  >  > 
 >  >  >  >  > Are you able to debug the Emacs executable?  I don't know if the build
 >  >  >  >  > you're using includes the necessary symbols, but a backtrace with at
 >  >  >  >  > least function names would be extremely helpful to figure this out.
 >  >  >  > 
 >  >  >  > I just used edebug... for the trace I submitted...
 >  >  >  >  > 
 >  >  >  >  > Does the problem occur to any server or just the one you listed?  You
 >  >  >  >  > can test with
 >  >  >  >  > 
 >  >  >  >  >  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")
 >  >  >  >  > 
 >  >  >  > 
 >  >  >  > This gave the same crash!
 >  >  >  > 
 >  >  >  >  > It's strange that there's been no other reports of this issue.  Do you
 >  >  >  >  > have access to any other systems where you can test?
 >  >  >  > 
 >  >  >  > Could it have anything to do with 64-bit Win7?
 >  >  >  > Could there be any conflicts with cygwin x86_64 that I have installed
 >  >  >  > (though I purposely didn't install cygwin gnutls) -- but could other
 >  >  >  > dll's be in conflict?
 >  >  > 
 >  >  > OK - I answered my own question.
 >  >  > There seems to be an incompatibility with the cygwin-mount library...
 >  > 
 >  > In particular, the problem seems to be with the called cygwin mount
 >  > program (/bin/mount.exe). Replacing it with /bin/true.exe prevents the
 >  > crash.
 >  > 
 >  > I wonder whether this is a dll incompatibility with cygwin.dll or in
 >  > particular the x86_64 version...
 > 
 > I did some more troubleshooting... actually, I don't think it is a
 > cygwin.dll problem nor is it a problem with the cygwin program
 > mount.exe.
 > 
 > 
 > The problem seems to be with the lisp code in cygwin-mount-activate in
 > terms of how it changes file-name-handler-alist,
 > substitute-in-file-name, and expand-file-name.
 > 
 > I say this because gnutls causes the crash after
 > (cygwin-mount-activate) is run. But if you wait to run
 > (cygwin-mount-deactivate), then it won't crash. So the problem isn't
 > with whether mount.exe has been run (I also double checked this by
 > manually running mount.exe via call-process). Rather, it has something
 > to do with the file name substitutions set up.
 > 
 > So, while the physical crash may be somewhere in the lisp code. The
 > issue is with how the cygwin file name handler lisp code interacts with the
 > gnutls code...

Oops I spoke too soon. Cygwin-mount works just fine and properly
parses the file names in the guntls code.

Rather,  the problem is in how the C-code handles the Cygwin/Linux
style paths enumerated in the Lisp variable:
(defcustom gnutls-trustfiles
  '(
    "/etc/ssl/certs/ca-certificates.crt" ; Debian, Ubuntu, Gentoo and Arch Linux
    "/etc/pki/tls/certs/ca-bundle.crt"   ; Fedora and RHEL
    "/etc/ssl/ca-bundle.pem"             ; Suse
    "/usr/ssl/certs/ca-bundle.crt"       ; Cygwin
    )

If I set the final element to the Windows style path equivalent:
   "C:/cygwin/usr/ssl/certs/ca-bundle.crt"
then gnutls works fine without crashing. So, the problem clearly is
with *nix-style paths

Stepping through the *Lisp* code shows that the file paths are all
properly parsed when cygwin-mount is loaded/activated. Indeed,
file-exists-p properly recognizes the cygwin path
"/usr/ssl/certs/ca-bundle.crt" and returns nil on the other paths that
don't exist in a standard Cygwin setup.

Note that if cygwin-mount is not loaded/activated, then
"/usr/ssl/certs/ca-bundle.crt" (along with the other list elements)
fails the file-exists-p test in gnutls-negotiate so that 'trustfiles'
gets set to nil which explains why it doesn't crash in the case when
cygwin-mount is not used since trivially 'trustfiles' has no paths
associated with it.

So, basically, we have the following Catch-22. If cygwin-mount is not
loaded/activated, then the cert location for Cygwin is never found. If
cygwin-mount is activated then it causes a crash. The result being
that in Windows, no certs are ever loaded when using the default
definition of gnutls-trustfiles

Presumably the problem is that the C-code doesn't know how to deal with
a Cygwin (*nix) style path that has been properly recognized by the
Lisp code (via cygwin-mount).

It seems like there are two potential solutions:
1. Use Windows-style paths in the definition of gnutls-trustfiles
   (this should work in Linux too, since
   "C:/usr/ssl/certs/ca-bundle.crt" will generally fail the
   file-exists-p test)

2. Add cygwin-mount functionality to the C-code so that it can parse
   cygwin (Unix) style paths.

In any case, I imagine the C-code crashes because it sees a Unix-style
path while expecting a Windows style path... that being said the
C-code should be better behaved than that... at a minimum the code
should check to make sure the certificate file path is well-formed and
exists.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

Please put this on hold while we discuss it in the GnuTLS mailing list
and let's update it when (if) we find a solution there.  It seems to be
outside the Emacs code right now.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Wed, 23 Oct 2013 00:17:49 -0400
> Cc: Ted Zlatanov <tzz@lifelogs.com>, 15648@debbugs.gnu.org
> 
> Stepping through the *Lisp* code shows that the file paths are all
> properly parsed when cygwin-mount is loaded/activated. Indeed,
> file-exists-p properly recognizes the cygwin path
> "/usr/ssl/certs/ca-bundle.crt" and returns nil on the other paths that
> don't exist in a standard Cygwin setup.
> 
> Note that if cygwin-mount is not loaded/activated, then
> "/usr/ssl/certs/ca-bundle.crt" (along with the other list elements)
> fails the file-exists-p test in gnutls-negotiate so that 'trustfiles'
> gets set to nil which explains why it doesn't crash in the case when
> cygwin-mount is not used since trivially 'trustfiles' has no paths
> associated with it.
> 
> So, basically, we have the following Catch-22. If cygwin-mount is not
> loaded/activated, then the cert location for Cygwin is never found. If
> cygwin-mount is activated then it causes a crash. The result being
> that in Windows, no certs are ever loaded when using the default
> definition of gnutls-trustfiles
> 
> Presumably the problem is that the C-code doesn't know how to deal with
> a Cygwin (*nix) style path that has been properly recognized by the
> Lisp code (via cygwin-mount).

The native Windows build of Emacs certainly doesn't understand the
magic of Cygwin mounts.  How can it?  The cygwin-mount package cannot
possibly work for external DLLs that were developed for native Windows
builds of programs which know nothing about Lisp and Emacs file I/O.

The problem is almost certainly that the GnuTLS code was assured that
a file exists (because Emacs used cygwin-mount), but then the file
could not be reached.  I can understand why GnuTLS becomes confused.

But since you didn't provide any C-level backtraces, we cannot know
where that code is, and thus cannot fix it.

> It seems like there are two potential solutions:
> 1. Use Windows-style paths in the definition of gnutls-trustfiles
>    (this should work in Linux too, since
>    "C:/usr/ssl/certs/ca-bundle.crt" will generally fail the
>    file-exists-p test)
> 
> 2. Add cygwin-mount functionality to the C-code so that it can parse
>    cygwin (Unix) style paths.

 3. Do not use cygwin-mount in conjunction with the native Windows
    build of Emacs.

> In any case, I imagine the C-code crashes because it sees a Unix-style
> path while expecting a Windows style path...

Windows supports Unix-style file names.  The problem is that the file
"/usr/ssl/certs/ca-bundle.crt" cannot be found by starting from the
root directory of the current drive.

> that being said the C-code should be better behaved than that... at
> a minimum the code should check to make sure the certificate file
> path is well-formed and exists.

See above: unless you present the backtrace from the crash, no one can
know where the offending code is, or what it does wrong.  Please
provide that data.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 18:16:33 +0300 on Wednesday, October 23, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Wed, 23 Oct 2013 00:17:49 -0400
 > > Cc: Ted Zlatanov <tzz@lifelogs.com>, 15648@debbugs.gnu.org
 > > 
 > > Stepping through the *Lisp* code shows that the file paths are all
 > > properly parsed when cygwin-mount is loaded/activated. Indeed,
 > > file-exists-p properly recognizes the cygwin path
 > > "/usr/ssl/certs/ca-bundle.crt" and returns nil on the other paths that
 > > don't exist in a standard Cygwin setup.
 > > 
 > > Note that if cygwin-mount is not loaded/activated, then
 > > "/usr/ssl/certs/ca-bundle.crt" (along with the other list elements)
 > > fails the file-exists-p test in gnutls-negotiate so that 'trustfiles'
 > > gets set to nil which explains why it doesn't crash in the case when
 > > cygwin-mount is not used since trivially 'trustfiles' has no paths
 > > associated with it.
 > > 
 > > So, basically, we have the following Catch-22. If cygwin-mount is not
 > > loaded/activated, then the cert location for Cygwin is never found. If
 > > cygwin-mount is activated then it causes a crash. The result being
 > > that in Windows, no certs are ever loaded when using the default
 > > definition of gnutls-trustfiles
 > > 
 > > Presumably the problem is that the C-code doesn't know how to deal with
 > > a Cygwin (*nix) style path that has been properly recognized by the
 > > Lisp code (via cygwin-mount).
 > 
 > The native Windows build of Emacs certainly doesn't understand the
 > magic of Cygwin mounts.  How can it?  The cygwin-mount package cannot
 > possibly work for external DLLs that were developed for native Windows
 > builds of programs which know nothing about Lisp and Emacs file I/O.
 > 
 > The problem is almost certainly that the GnuTLS code was assured that
 > a file exists (because Emacs used cygwin-mount), but then the file
 > could not be reached.  I can understand why GnuTLS becomes confused.
 > 
 > But since you didn't provide any C-level backtraces, we cannot know
 > where that code is, and thus cannot fix it.
 > 
 > > It seems like there are two potential solutions:
 > > 1. Use Windows-style paths in the definition of gnutls-trustfiles
 > >    (this should work in Linux too, since
 > >    "C:/usr/ssl/certs/ca-bundle.crt" will generally fail the
 > >    file-exists-p test)
 > > 
 > > 2. Add cygwin-mount functionality to the C-code so that it can parse
 > >    cygwin (Unix) style paths.
 > 
 >  3. Do not use cygwin-mount in conjunction with the native Windows
 >     build of Emacs.
 > 

4. This small patch to gnutls.el will fix the problem by expanding the
file name to a full, valid path:

--- gnutls.el	2013-03-17 13:52:40.000000000 -0400
+++ gnutls.el.new	2013-10-23 12:47:36.503554500 -0400
@@ -174,7 +174,8 @@
   (let* ((type (or type 'gnutls-x509pki))
          (trustfiles (or trustfiles
                          (delq nil
-                               (mapcar (lambda (f) (and f (file-exists-p f) f))
+                               (mapcar (lambda (f) (and f (file-exists-p f) 
+														(expand-file-name f)))
                                        (if (functionp gnutls-trustfiles)
                                            (funcall gnutls-trustfiles)
                                          gnutls-trustfiles)))))


 > > In any case, I imagine the C-code crashes because it sees a Unix-style
 > > path while expecting a Windows style path...
 > 
 > Windows supports Unix-style file names.  The problem is that the file
 > "/usr/ssl/certs/ca-bundle.crt" cannot be found by starting from the
 > root directory of the current drive.

The above proposed patch would fix that problem.
 > 
 > > that being said the C-code should be better behaved than that... at
 > > a minimum the code should check to make sure the certificate file
 > > path is well-formed and exists.
 > 
 > See above: unless you present the backtrace from the crash, no one can
 > know where the offending code is, or what it does wrong.  Please
 > provide that data.

What do I need to do to get a backtrace?
I don't have any C-debugging software on my Windows laptop... and have
never done C-code debugging in a Windows environment...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 10:52:46 -0400 on Wednesday, October 23, 2013:
 > Please put this on hold while we discuss it in the GnuTLS mailing list
 > and let's update it when (if) we find a solution there.  It seems to be
 > outside the Emacs code right now.
 > 
 > Ted

As per my previous email, the following trivial patch will fix the
Emacs code to make sure that valid paths are always passed...

--- gnutls.el	2013-03-17 13:52:40.000000000 -0400
+++ gnutls.el.new	2013-10-23 12:47:36.503554500 -0400
@@ -174,7 +174,8 @@
   (let* ((type (or type 'gnutls-x509pki))
          (trustfiles (or trustfiles
                          (delq nil
-                               (mapcar (lambda (f) (and f (file-exists-p f) f))
+                               (mapcar (lambda (f) (and f (file-exists-p f) 
+														(expand-file-name f)))
                                        (if (functionp gnutls-trustfiles)
                                            (funcall gnutls-trustfiles)
                                          gnutls-trustfiles)))))

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Wed, 23 Oct 2013 13:12:56 -0400
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
> 
> 4. This small patch to gnutls.el will fix the problem by expanding the
> file name to a full, valid path:

Sorry, but I cannot accept this.  There's absolutely no reason to run
the file through expand-file-name at that place.  More importantly,
there's no reason to believe this is the only place where the same
problem could surface.

>  > Windows supports Unix-style file names.  The problem is that the file
>  > "/usr/ssl/certs/ca-bundle.crt" cannot be found by starting from the
>  > root directory of the current drive.
> 
> The above proposed patch would fix that problem.

Only by sheer luck.

>  > > that being said the C-code should be better behaved than that... at
>  > > a minimum the code should check to make sure the certificate file
>  > > path is well-formed and exists.
>  > 
>  > See above: unless you present the backtrace from the crash, no one can
>  > know where the offending code is, or what it does wrong.  Please
>  > provide that data.
> 
> What do I need to do to get a backtrace?

Run Emacs under GDB, like this:

  gdb ./emacs.exe

When GDB finishes displaying its startup blurbs, just type "run", and
do whatever you do in Emacs to reproduce the problem.  When the
problem does happen, GDB will kick in, and you should type this at the
GDB prompt:

 (gdb) thread apply all bt

> I don't have any C-debugging software on my Windows laptop... and have
> never done C-code debugging in a Windows environment...

A Windows build of GDB suitable for this is available from the MinGW
site:

  http://sourceforge.net/projects/mingw/files/MinGW/Extension/gdb/gdb-7.6.1-1/gdb-7.6.1-1-mingw32-bin.tar.lzma/download

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Wed, 23 Oct 2013 13:25:23 -0400
> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
> Ted Zlatanov wrote at about 10:52:46 -0400 on Wednesday, October 23, 2013:
>  > Please put this on hold while we discuss it in the GnuTLS mailing list
>  > and let's update it when (if) we find a solution there.  It seems to be
>  > outside the Emacs code right now.
>  > 
>  > Ted
> 
> As per my previous email, the following trivial patch will fix the
> Emacs code to make sure that valid paths are always passed...

Thanks, but I'm firmly against such work-arounds.  A file name does
not need to be fully resolved to be valid.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Wed, 23 Oct 2013 21:07:09 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: <emacs@kosowsky.org>
>> Date: Wed, 23 Oct 2013 13:25:23 -0400
>> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
>> 
>> Ted Zlatanov wrote at about 10:52:46 -0400 on Wednesday, October 23, 2013:
>> > Please put this on hold while we discuss it in the GnuTLS mailing list
>> > and let's update it when (if) we find a solution there.  It seems to be
>> > outside the Emacs code right now.
>> > 
>> As per my previous email, the following trivial patch will fix the
>> Emacs code to make sure that valid paths are always passed...

EZ> Thanks, but I'm firmly against such work-arounds.  A file name does
EZ> not need to be fully resolved to be valid.

Right, and in addition any file name at all handed to a C library should
not crash Emacs :)

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 21:00:21 +0300 on Wednesday, October 23, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Wed, 23 Oct 2013 13:12:56 -0400
 > > Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
 > > 
 > > 4. This small patch to gnutls.el will fix the problem by expanding the
 > > file name to a full, valid path:
 > 
 > Sorry, but I cannot accept this.  There's absolutely no reason to run
 > the file through expand-file-name at that place.

There is a *great* reason to add expand-file-name *any* time you pass
a file name or path to external C-code that references the file
system.

This is necessary (and should be best or even required practice) since
Emacs by design may include various filehandlers (like cygwin-mount)
that internally translate path names before accessing the file system
while external C-code has no way of knowing that the path names need
transformation. This is particularly important since the Emacs
filehandlers may be turned on and off or changed at will or
dynamically and the C-code can't possibly know that, so it can't
possibly know how and when to perform the required file handling
operations to access the file system.

Thus, I would argue that it is best practice to use expand-file-name
whenever a file path is passed from Emacs lisp to C-code to ensure
that a system valid (absolute) path is ultimately passed to the file
system independent of how Emacs may be translating paths internally.

 > More importantly, there's no reason to believe this is the only
 > place where the same problem could surface.

There is a wonderful reason to so believe since It is the only place
that gnutls-trustfiles is referenced... hence, it is the natural place to
fix it.

It's pretty simple. You really have only 3 choices:

1. Only allow path names in gnutls-trustfiles that are native and
   understandable as-is to the machine.  In particular, this would
   mean deleting the default gnutls-turstfiles cgywin entry since it
   is written relative to cygwin root which is not known to the C-code
   or to the system. Note that making the path hard-wired absolute
   wouldn't work for cygwin since while cygwin root is typically
   C:\cygwin, it need not be.. Of course, deleting the cygwin entry is
   reasonable, if you don't care about supporting cygwin by default

2. Add my patch or a similar to ensure that a valid path is always
   passed to the C-code -- whether one is using cygwin mount or any
   other code that changes how Emacs translates path names.  This
   approach is simple and non-harmful, should always work, and in
   general protects the C-code from being passed a non-valid path.

3. Modify the C-code to include cygwin-mount type functionality so
   that the cygwin root is prepended to cygwin style path names. This
   is probably not an easy or practical solution since it requires
   changing the compiled C-library with calls to various cygwin
   functions and libraries. Also, it would only work for the file
   handler modifications of cygwin-mount and not to other possible
   file handlers -- thus modifying the C-code would be brittle,
   non-general, and non-portable.

Leaving it as-is shouldn't be an acceptable solution since without
cygwin-mount, the cygwin entry is meaningless and is *never* used --
so it's wasteful at best and confusing or crash-inducing at
worst. Even worse, with cygwin-mount, it causes a crash of the entire
Emacs process.

So, this is a BUG in the Emacs code -- it needs to be fixed!
If you don't like my fix, then feel free to find another...

Of course, there is still arguably a bug in the C-code since passing
an invalid path to the gnutls-boot C-code should not cause the entire
Emacs process to crash.

This bug should be easily reproducible by passing a cygwin path to
gnutls-boot under Windows

 > >  > Windows supports Unix-style file names.  The problem is that the file
 > >  > "/usr/ssl/certs/ca-bundle.crt" cannot be found by starting from the
 > >  > root directory of the current drive.
 > > 
 > > The above proposed patch would fix that problem.
 > 
 > Only by sheer luck.

Really? I can pretty much guarantee it will always fix the problem for
gnutls.el since there is no other reference to gnutls-trustfiles. The
alternative fix is to delete the cygwin entry.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 14:58:21 -0400 on Wednesday, October 23, 2013:
 > On Wed, 23 Oct 2013 21:07:09 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
 > 
 > >> From: <emacs@kosowsky.org>
 > >> Date: Wed, 23 Oct 2013 13:25:23 -0400
 > >> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
 > >> 
 > >> Ted Zlatanov wrote at about 10:52:46 -0400 on Wednesday, October 23, 2013:
 > >> > Please put this on hold while we discuss it in the GnuTLS mailing list
 > >> > and let's update it when (if) we find a solution there.  It seems to be
 > >> > outside the Emacs code right now.
 > >> > 
 > >> As per my previous email, the following trivial patch will fix the
 > >> Emacs code to make sure that valid paths are always passed...
 > 
 > EZ> Thanks, but I'm firmly against such work-arounds.  A file name does
 > EZ> not need to be fully resolved to be valid.

Well, given that Emacs *officially* supports the use of file handlers
(see Elips info 25.11 Making Certain File Names "Magic"), one should
be able to expect that properly written file handlers -- i.e. ones
that address all the documented file access primitives -- should work
with all officially sanctioned Emacs library code.

Indeed the Elisp documentation states:
   	   "Here are the operations that a magic file name handler gets to
	   handle:
	   `access-file', `add-name-to-file', `byte-compiler-base-file-name',
	   `copy-directory', `copy-file', `delete-directory', `delete-file',
	   `diff-latest-backup-file', `directory-file-name', `directory-files',
	   `directory-files-and-attributes', `dired-compress-file',
	   `dired-uncache',
	   `expand-file-name', `file-accessible-directory-p', `file-attributes',
	   `file-directory-p', `file-executable-p', `file-exists-p',
	   `file-local-copy', `file-remote-p', `file-modes',
	   `file-name-all-completions', `file-name-as-directory',
	   `file-name-completion', `file-name-directory',
	   `file-name-nondirectory',
	   `file-name-sans-versions', `file-newer-than-file-p',
	   `file-ownership-preserved-p', `file-readable-p', `file-regular-p',
	   `file-in-directory-p', `file-symlink-p', `file-truename',
	   `file-writable-p', `file-equal-p', `find-backup-file-name',
	   `get-file-buffer', `insert-directory', `insert-file-contents',
	   `load', `make-auto-save-file-name', `make-directory',
	   `make-directory-internal', `make-symbolic-link',
	   `process-file', `rename-file', `set-file-modes', `set-file-times',
	   `set-visited-file-modtime', `shell-command', `start-file-process',
	   `substitute-in-file-name',
	   `unhandled-file-name-directory', `vc-registered',
	   `verify-visited-file-modtime',
	   `write-region'.

 	   ... The handler function must handle all of the above operations, and
	   possibly others to be added in the future."

Clearly, the intent is that by defining the file handler to address
all the above primitive functions, one can be assured that all paths
using the handler will be translated properly.

One should not have to worry that some random Emacs library may choose
to bypass the primitives by passing non-expanded, potentially
non-system recognizable path names to some undocumented C-code that
directly accesses the file system.

Since "expand-file-name" is one of the above documented primitives it
makes sense that the gnutls.el code pass such a primitive to pass
trust file names to the nutls-boot C-code, thus making the handling of
such file names robust and portable.

Otherwise, why support file handlers at all and why be so careful to
document the notion of primitive if there are official Emacs libraries
that knowingly bypass such primitives, causing the file handler to
inexplicably fail without any documentation or rational reason.

This is not just about cygwin-mount. Since file handlers are an
officially supported feature of Emacs, I should be able to for example
write a handler that recognizes '~~' as a synonym for /mnt/media and
expect that "~~/myvideos" will always resolve to "/mnt/media/myvideos"
in any officially sanctioned Emacs function or library.

I should not have to read through the code of every official Emacs
library to determine whether or not they bypass the standard file
access primitives by calling some obscure, undocumented C-code. Isn't
this the entire purpose of having well-defined and documented
primitives in the first place?

If the maintainers of Emacs disagree, then the file handler feature
needs to be deprecated and eliminated... since a feature that cannot
be relied on to work properly within Emacs' own code base is worse
than not having it at all.

The irony is that the patch to fix this is about as simple and natural
as can be... simply use one reference to the primitive
"expand-file-name". Conversely, without such a patch there is no other
way of making file handlers work with gnutls!


 > Right, and in addition any file name at all handed to a C library should
 > not crash Emacs :)

True! but even if the C-code is fixed so as not to crash
Emacs, there is still a bug in the gnutls.el code in that it includes
a Cygwin path as part of the definition of gnutls-trustfiles that
can't possibly EVER EVER work -- whether you are using cygwin-mount or
not -- even if the C-code is patched so as not to crash.

Either the bug is in including a Cygwin path or in not using a simple
path expansion to make the Cygwin path work. You can't include a
Cygwin path but write code in the very same library that by design
won't support it's own feature!

Of course, the more general principle is that all official Emacs code
should play nicely with core Elisp functionality...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

emacs@kosowsky.org wrote at about 19:45:48 -0400 on Wednesday, October 23, 2013:
 > Ted Zlatanov wrote at about 14:58:21 -0400 on Wednesday, October 23, 2013:
 >  > On Wed, 23 Oct 2013 21:07:09 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
 >  > 
 >  > >> From: <emacs@kosowsky.org>
 >  > >> Date: Wed, 23 Oct 2013 13:25:23 -0400
 >  > >> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
 >  > >> 
 >  > >> Ted Zlatanov wrote at about 10:52:46 -0400 on Wednesday, October 23, 2013:
 >  > >> > Please put this on hold while we discuss it in the GnuTLS mailing list
 >  > >> > and let's update it when (if) we find a solution there.  It seems to be
 >  > >> > outside the Emacs code right now.
 >  > >> > 
 >  > >> As per my previous email, the following trivial patch will fix the
 >  > >> Emacs code to make sure that valid paths are always passed...
 >  > 
 >  > EZ> Thanks, but I'm firmly against such work-arounds.  A file name does
 >  > EZ> not need to be fully resolved to be valid.
 > 
 > Well, given that Emacs *officially* supports the use of file handlers
 > (see Elips info 25.11 Making Certain File Names "Magic"), one should
 > be able to expect that properly written file handlers -- i.e. ones
 > that address all the documented file access primitives -- should work
 > with all officially sanctioned Emacs library code.
 > 
 > Indeed the Elisp documentation states:
 >    	   "Here are the operations that a magic file name handler gets to
 > 	   handle:
 > 	   `access-file', `add-name-to-file', `byte-compiler-base-file-name',
 > 	   `copy-directory', `copy-file', `delete-directory', `delete-file',
 > 	   `diff-latest-backup-file', `directory-file-name', `directory-files',
 > 	   `directory-files-and-attributes', `dired-compress-file',
 > 	   `dired-uncache',
 > 	   `expand-file-name', `file-accessible-directory-p', `file-attributes',
 > 	   `file-directory-p', `file-executable-p', `file-exists-p',
 > 	   `file-local-copy', `file-remote-p', `file-modes',
 > 	   `file-name-all-completions', `file-name-as-directory',
 > 	   `file-name-completion', `file-name-directory',
 > 	   `file-name-nondirectory',
 > 	   `file-name-sans-versions', `file-newer-than-file-p',
 > 	   `file-ownership-preserved-p', `file-readable-p', `file-regular-p',
 > 	   `file-in-directory-p', `file-symlink-p', `file-truename',
 > 	   `file-writable-p', `file-equal-p', `find-backup-file-name',
 > 	   `get-file-buffer', `insert-directory', `insert-file-contents',
 > 	   `load', `make-auto-save-file-name', `make-directory',
 > 	   `make-directory-internal', `make-symbolic-link',
 > 	   `process-file', `rename-file', `set-file-modes', `set-file-times',
 > 	   `set-visited-file-modtime', `shell-command', `start-file-process',
 > 	   `substitute-in-file-name',
 > 	   `unhandled-file-name-directory', `vc-registered',
 > 	   `verify-visited-file-modtime',
 > 	   `write-region'.
 > 
 >  	   ... The handler function must handle all of the above operations, and
 > 	   possibly others to be added in the future."
 > 
 > Clearly, the intent is that by defining the file handler to address
 > all the above primitive functions, one can be assured that all paths
 > using the handler will be translated properly.
 > 
 > One should not have to worry that some random Emacs library may choose
 > to bypass the primitives by passing non-expanded, potentially
 > non-system recognizable path names to some undocumented C-code that
 > directly accesses the file system.
 > 
 > Since "expand-file-name" is one of the above documented primitives it
 > makes sense that the gnutls.el code pass such a primitive to pass
 > trust file names to the nutls-boot C-code, thus making the handling of
 > such file names robust and portable.
 > 
 > Otherwise, why support file handlers at all and why be so careful to
 > document the notion of primitive if there are official Emacs libraries
 > that knowingly bypass such primitives, causing the file handler to
 > inexplicably fail without any documentation or rational reason.
 > 
 > This is not just about cygwin-mount. Since file handlers are an
 > officially supported feature of Emacs, I should be able to for example
 > write a handler that recognizes '~~' as a synonym for /mnt/media and
 > expect that "~~/myvideos" will always resolve to "/mnt/media/myvideos"
 > in any officially sanctioned Emacs function or library.
 > 
 > I should not have to read through the code of every official Emacs
 > library to determine whether or not they bypass the standard file
 > access primitives by calling some obscure, undocumented C-code. Isn't
 > this the entire purpose of having well-defined and documented
 > primitives in the first place?
 > 
 > If the maintainers of Emacs disagree, then the file handler feature
 > needs to be deprecated and eliminated... since a feature that cannot
 > be relied on to work properly within Emacs' own code base is worse
 > than not having it at all.
 > 
 > The irony is that the patch to fix this is about as simple and natural
 > as can be... simply use one reference to the primitive
 > "expand-file-name". Conversely, without such a patch there is no other
 > way of making file handlers work with gnutls!
 > 
 > 
 >  > Right, and in addition any file name at all handed to a C library should
 >  > not crash Emacs :)
 > 
 > True! but even if the C-code is fixed so as not to crash
 > Emacs, there is still a bug in the gnutls.el code in that it includes
 > a Cygwin path as part of the definition of gnutls-trustfiles that
 > can't possibly EVER EVER work -- whether you are using cygwin-mount or
 > not -- even if the C-code is patched so as not to crash.
 > 
 > Either the bug is in including a Cygwin path or in not using a simple
 > path expansion to make the Cygwin path work. You can't include a
 > Cygwin path but write code in the very same library that by design
 > won't support it's own feature!
 > 
 > Of course, the more general principle is that all official Emacs code
 > should play nicely with core Elisp functionality...

Just one addendum...

If the maintainers of gnutls.el for some presumably unstated by
compelling reason do not want to support Magic File names (aka file
handlers), then the library code should:

1. Properly disable the use of file handlers so that modified
   primitives (like file-exist-p) fail on cygwin paths.

   One way would be to add the following line to the let* statement in
   gnutls-negotiate:
		(file-name-handler-alist nil)

2. DOCUMENT that gnutls doesn't support Magic File names

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Wed, 23 Oct 2013 15:49:46 -0400
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
> 
>  > > The above proposed patch would fix that problem.
>  > 
>  > Only by sheer luck.
> 
> Really? I can pretty much guarantee it will always fix the problem for
> gnutls.el since there is no other reference to gnutls-trustfiles. The
> alternative fix is to delete the cygwin entry.

Please provide the backtrace from the crashes, so that we could
understand the problem completely.  Then the discussion of the
possible solutions will become rational and free of guesswork.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Wed, 23 Oct 2013 19:45:48 -0400 <emacs@kosowsky.org> wrote: 

> The irony is that the patch to fix this is about as simple and natural
> as can be... simply use one reference to the primitive
> "expand-file-name". Conversely, without such a patch there is no other
> way of making file handlers work with gnutls!

> Either the bug is in including a Cygwin path or in not using a simple
> path expansion to make the Cygwin path work. You can't include a
> Cygwin path but write code in the very same library that by design
> won't support it's own feature!

Provide a backtrace, please.  I promise you we'll consider your proposal
and all the alternatives as soon as we know what's broken.

> Just one addendum...

> If the maintainers of gnutls.el for some presumably unstated by
> compelling reason do not want to support Magic File names (aka file
> handlers),

There's no need to be snarky.  Emacs has many contributors and I
maintain gnutls.el (but welcome contributions).  I will gladly make the
change to avoid a path like "/usr/ssl/whatever" that doesn't work in
Cygwin, but I don't want to make that change in order to hide an
underlying problem.  Eli has the best background to make sense of this
problem and diagnose it.

In other words, if a magic path can break Emacs because of some library
interaction beneath the ELisp code, the fix is *probably* not at the
ELisp level because there are so many potential trigger points for the
bad behavior.

> then the library code should:

> 1. Properly disable the use of file handlers so that modified
>    primitives (like file-exist-p) fail on cygwin paths.

>    One way would be to add the following line to the let* statement in
>    gnutls-negotiate:
> 		(file-name-handler-alist nil)

> 2. DOCUMENT that gnutls doesn't support Magic File names

Thank you, we'll take that under consideration as well.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 06:59:39 -0400 on Thursday, October 24, 2013:
 > On Wed, 23 Oct 2013 19:45:48 -0400 <emacs@kosowsky.org> wrote: 
 > Provide a backtrace, please.  I promise you we'll consider your proposal
 > and all the alternatives as soon as we know what's broken.

OK... but before I invest in downloading and running a C-debugger,
I want to be clear that there are actually 2 bugs here - one at the
elisp level and one at the C-level

1. At the elisp level, gnutls.el fails to consistently and
   appropriately handle (or disable) Magic Files aka file handlers
   (including in particular, cygwin-mount).

   Specifically, gnutls-negotiate uses the Magic File associated
   primitive file-exists-p to test for the existence of a file but
   then passes off the unmodified (and non-existent) Magic File path
   (e.g., path relative to cygwin root) to some C-code. Clearly, short
   of passing nearly the entire Emacs state (including variable &
   function defs) to the C-code, it is impossible for the C-code to
   know what file handler to use to translate the Magic File into a
   system recognizable and valid path.

   I believe the nature of this bug is fully explored and understood.
   
   The solution is to treat Magic files appropriately and consistently
   either (1) by using the Magic File aware primitive expand-file-name
   to expand the file name before passing off to the C-code or (2)
   disabling Magic file handlers (and documenting that fact)

2. At the C-code level, there is a bug in that passing a
   non-existent/invalid file path may cause the entire Emacs code to
   crash.

   It is this bug that requires further C-source debugging to identify
   the source...

 > I will gladly make the change to avoid a path like
 > "/usr/ssl/whatever" that doesn't work in Cygwin, but I don't want
 > to make that change in order to hide an underlying problem.

Again, the issue is broader than just Cygwin paths... the code as-is
fails to handle any type of Magic File and associated file
handlers... and there is no reason or way for a user to suspect or
know that the officially include Emacs library gnutls.el chooses not
to handle Magic files correctly -- short of debugging, tracing, and
reading the detailed gnutls.el elisp code.


Also, as above, there is both a problem in the gnutls.el code AND in
the underlying C-code

 > In other words, if a magic path can break Emacs because of some library
 > interaction beneath the ELisp code, the fix is *probably* not at the
 > ELisp level because there are so many potential trigger points for the
 > bad behavior.

Wrong - the opposite is true. Handling of magic files via their
associated file handlers can *only* be done generally and properly at
the elisp level since short of passing the entire Emacs state to
external C-code, there is no way of the C-code knowing how to handle
potentially arbitrary file handlers.

It's very simple. File access and reference should only be done
via Magic File aware primitives. In particular, when passing a path to
external C-code, that code should first be translated by something like
expand-file-name to system-recognizable and valid paths.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Thu, 24 Oct 2013 10:10:08 -0400 <emacs@kosowsky.org> wrote: 

> Ted Zlatanov wrote at about 06:59:39 -0400 on Thursday, October 24, 2013:
>> On Wed, 23 Oct 2013 19:45:48 -0400 <emacs@kosowsky.org> wrote: 
>> Provide a backtrace, please.  I promise you we'll consider your proposal
>> and all the alternatives as soon as we know what's broken.

> OK... but before I invest in downloading and running a C-debugger,
> I want to be clear that there are actually 2 bugs here - one at the
> elisp level and one at the C-level

There could be 1 or 2 or N bugs, we don't know and would like facts (a backtrace).

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 11:48:55 -0400 on Thursday, October 24, 2013:
 > On Thu, 24 Oct 2013 10:10:08 -0400 <emacs@kosowsky.org> wrote: 
 > 
 > > Ted Zlatanov wrote at about 06:59:39 -0400 on Thursday, October 24, 2013:
 > >> On Wed, 23 Oct 2013 19:45:48 -0400 <emacs@kosowsky.org> wrote: 
 > >> Provide a backtrace, please.  I promise you we'll consider your proposal
 > >> and all the alternatives as soon as we know what's broken.
 > 
 > > OK... but before I invest in downloading and running a C-debugger,
 > > I want to be clear that there are actually 2 bugs here - one at the
 > > elisp level and one at the C-level
 > 
 > There could be 1 or 2 or N bugs, we don't know and would like facts (a backtrace).
 > 
 > Ted

Correction - at *least* one elisp bug and at *least* one C-code bug...
I would bet good money on that :P

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Stefan Monnier]

>    Specifically, gnutls-negotiate uses the Magic File associated
>    primitive file-exists-p to test for the existence of a file but
>    then passes off the unmodified (and non-existent) Magic File path

Good point.  The problem is the use of file-exists-p (which tests if
Elisp code can reach this file) instead of another function along the
lines of file-accessible-directory-p, which checks whether the file name
is reachable to C-level system primitives.

"Unmodified" is not a real problem, although we might want to use some
function to "canonicalize" the file name so as to increase the chances
that the file is reachable to C-level system primitives.

>    either (1) by using the Magic File aware primitive expand-file-name

expand-file-name is not the right function, even if in some cases it may
happen to help.

> 2. At the C-code level, there is a bug in that passing a
>    non-existent/invalid file path may cause the entire Emacs code to
>    crash.

Right, this is the most serious of the two problems.


        Stefan

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Stefan Monnier wrote at about 13:57:13 -0400 on Thursday, October 24, 2013:
 > >    Specifically, gnutls-negotiate uses the Magic File associated
 > >    primitive file-exists-p to test for the existence of a file but
 > >    then passes off the unmodified (and non-existent) Magic File path
 > 
 > Good point.  The problem is the use of file-exists-p (which tests if
 > Elisp code can reach this file) instead of another function along the
 > lines of file-accessible-directory-p, which checks whether the file name
 > is reachable to C-level system primitives.

Note that the predicate "file-accessible-directory-p" is also one of
the primitives that Magic File handlers are "required" to
address... so using "file-accessible-directory-p" would create the
same 'true' response for any valid Magic file path unless one were to
prevent Magic file handling by say wrapping the predicate in a 'let'
where file-name-handler-alist is set to nil).

 > "Unmodified" is not a real problem, although we might want to use some
 > function to "canonicalize" the file name so as to increase the chances
 > that the file is reachable to C-level system primitives.
 > 
 > >    either (1) by using the Magic File aware primitive expand-file-name
 > 
 > expand-file-name is not the right function, even if in some cases it may
 > happen to help.

Why isn't it the right one?
The function name describes it as "Convert filename NAME to absolute,
and *canonicalize* it." (emphasis added)

It seems that any proper Magic File handler should (by definition)
hook into expand-file-name to provide an absolute canonicalized path,
so I would think if the handler is properly written, then it should
work (by definition).


Alternatively, how else would you recommend generally and portably
canonicalizing a Magic file path?

 > > 2. At the C-code level, there is a bug in that passing a
 > >    non-existent/invalid file path may cause the entire Emacs code to
 > >    crash.
 > 
 > Right, this is the most serious of the two problems.
 > 
Agreed.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Stefan Monnier]

>> Good point.  The problem is the use of file-exists-p (which tests if
>> Elisp code can reach this file) instead of another function along the
>> lines of file-accessible-directory-p, which checks whether the file name
>> is reachable to C-level system primitives.
> Note that the predicate "file-accessible-directory-p" is also one of

I didn't mean to say that file-accessible-directory-p is the
right answer.  Only that something *like it* should be used.
I.e. a function specifically meant to give you some idea about whether
that file is known to the OS primitives.

>> expand-file-name is not the right function, even if in some cases it may
>> happen to help.
> Why isn't it the right one?
> The function name describes it as "Convert filename NAME to absolute,
> and *canonicalize* it." (emphasis added)

You give too much meaning to "canonicalize": it does not imply that it's
the name used outside of Emacs.  It just means to try and remove things
like "<something>/../foo/<somethingelse>".

So the output will be a "somewhat canonicalized Elisp file name".
Not necessarily an "OS-level file name".  `cygwin-mount' happens to use
expand-file-name to convert the name from one form to another, basically
for pragmatic reasons: that's what exists and it is called often enough
that it works OK in practice.

> It seems that any proper Magic File handler should (by definition)
> hook into expand-file-name to provide an absolute canonicalized path,

Think of a magic file handler that provides access to members of a zip
or tar archive or think of file name handlers for Tramp.
These "canonical" file names still won't help you.

> Alternatively, how else would you recommend generally and portably
> canonicalizing a Magic file path?

You'd first have to define what kind of "canonicalizing" you want to do.
I think in the present case what you want is to turn an Elisp file name
into a file name understood by OS-level primitives (or nil if that
can't be done).  We don't have such a function right now.  So my
recommendation is to add such a function.


        Stefan

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 21:00:21 +0300 on Wednesday, October 23, 2013:
 > > From: <emacs@kosowsky.org>
 > > What do I need to do to get a backtrace?
 > 
 > Run Emacs under GDB, like this:
 > 
 >   gdb ./emacs.exe
 > 
 > When GDB finishes displaying its startup blurbs, just type "run", and
 > do whatever you do in Emacs to reproduce the problem.  When the
 > problem does happen, GDB will kick in, and you should type this at the
 > GDB prompt:
 > 
 >  (gdb) thread apply all bt
 > 

Here is the backtrace....

Reading symbols from C:\MyPrograms\Emacs\bin\emacs.exe...(no debugging
symbols f
ound)...done.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: C:\MyPrograms\Emacs\bin\emacs.exe
[New Thread 124728.0x31b44]
[New Thread 124728.0x802c]
[New Thread 124728.0x7e7c]
[New Thread 124728.0x7dd8]
[New Thread 124728.0x31e98]
[New Thread 124728.0x318b8]
[New Thread 124728.0x31cbc]
[New Thread 124728.0x31e28]

Program received signal SIGSEGV, Segmentation fault.
0x6d099192 in _gnutls_record_buffer_get_size ()
   from C:\kosowsky\bin\libgnutls-28.dll
(gdb) thread apply all bt

Thread 8 (Thread 124728.0x31e28):
#0  0x77abf8d1 in ntdll!ZwWaitForSingleObject ()
   from C:\Windows\system32\ntdll.dll
#1  0x77abf8d1 in ntdll!ZwWaitForSingleObject ()
   from C:\Windows\system32\ntdll.dll
#2  0x709217cd in ?? () from C:\Windows\SysWOW64\mswsock.dll
#3  0x709276b6 in ?? () from C:\Windows\SysWOW64\mswsock.dll
#4  0x772e6b87 in recv () from C:\Windows\syswow64\ws2_32.dll
#5  0x01021e2d in _sys_read_ahead ()
#6  0x00000000 in ?? ()

Thread 5 (Thread 124728.0x31e98):
#0  0x77abf8d1 in ntdll!ZwWaitForSingleObject ()
   from C:\Windows\system32\ntdll.dll
#1  0x77abf8d1 in ntdll!ZwWaitForSingleObject ()
   from C:\Windows\system32\ntdll.dll
#2  0x76ea149d in WaitForSingleObjectEx ()
   from C:\Windows\syswow64\KernelBase.dll
#3  0x00000278 in ?? ()
#4  0x00000000 in ?? ()

Thread 4 (Thread 124728.0x7dd8):
#0  0x773378d7 in USER32!DispatchMessageW ()
   from C:\Windows\syswow64\user32.dll
#1  0x010c8640 in w32_msg_pump.isra.23 ()
#2  0x80000000 in ?? ()

Thread 3 (Thread 124728.0x7e7c):
#0  0x77abfd91 in ntdll!ZwDelayExecution () from
C:\Windows\system32\ntdll.dll
#1  0x77abfd91 in ntdll!ZwDelayExecution () from
C:\Windows\system32\ntdll.dll
#2  0x76ea3bc8 in SleepEx () from C:\Windows\syswow64\KernelBase.dll
#3  0x00000000 in ?? ()

Thread 2 (Thread 124728.0x802c):
#0  0x77ac015d in ntdll!ZwWaitForMultipleObjects ()
   from C:\Windows\system32\ntdll.dll
#1  0x77ac015d in ntdll!ZwWaitForMultipleObjects ()
   from C:\Windows\system32\ntdll.dll
#2  0x77af2f91 in ntdll!RtlMoveMemory () from
C:\Windows\system32\ntdll.dll
#3  0x00000001 in ?? ()
#4  0x00000001 in ?? ()
#5  0x00000000 in ?? ()

Thread 1 (Thread 124728.0x31b44):
#0  0x6d099192 in _gnutls_record_buffer_get_size ()
   from C:\kosowsky\bin\libgnutls-28.dll
#1  0x6d0995c8 in gnutls_record_check_pending ()
   from C:\kosowsky\bin\libgnutls-28.dll
#2  0x01015e15 in wait_reading_process_output ()
#3  0x0088e7d8 in ?? ()
(gdb)

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Stefan Monnier wrote at about 20:59:06 -0400 on Thursday, October 24, 2013:
 > I didn't mean to say that file-accessible-directory-p is the
 > right answer.  Only that something *like it* should be used.
 > I.e. a function specifically meant to give you some idea about whether
 > that file is known to the OS primitives.

....

 > 
 > You'd first have to define what kind of "canonicalizing" you want to do.
 > I think in the present case what you want is to turn an Elisp file name
 > into a file name understood by OS-level primitives (or nil if that
 > can't be done).  We don't have such a function right now.  So my
 > recommendation is to add such a function.
 > 

I'm surprised that such a function doesn't (yet) exist in Emacs...

But until such a function exists, one might want to consider the
following:

1. Adding expand-file-name will work for any file name that
   cygwin-mount is designed to address (this is by definition)

2. Similarly, adding expand-file-name will likely work for any other
   Magic File handler that works similar to cygwin-mount in terms of
   just converting paths but generally won't work with Magic File
   handlers that work on archives or compressed file formats

3. Adding expand-file-name shouldn't have any ill effect on paths that
   are not Magic files

To take account of the fact that expand-file-name may not give a valid
OS path for all Magic File handlers, I would suggest the following
slight modification to my originally suggested patch.

Basically, this version does 2 things:
1. Pre-expands the file name *if* there is a Magic File handler for that
   file path (this will do nothing if there is no handler for that path)

2. Tests file-exists-p with all Magic File handlers shut off which
   tests to make sure that the OS primitives will work on the actual
   file path that will be passed

--- gnutls.el	2013-03-17 13:52:40.000000000 -0400
+++ gnutls.el.new	2013-10-23 12:47:36.503554500 -0400
@@ -174,7 +174,8 @@
   (let* ((type (or type 'gnutls-x509pki))
          (trustfiles (or trustfiles
                          (delq nil
-                               (mapcar (lambda (f) (and f (file-exists-p f) f))
+                               (mapcar (lambda (f) 
+								   		   (and f 
+										     (if (find-file-name-handler f
												    'expand-file-name)
+								   		      		(setq f (expand-file-name f)))
+											  (let (file-name-handler-alist)
+                                              	   (file-exists-p f)) f))
                                        (if (functionp gnutls-trustfiles)
                                            (funcall gnutls-trustfiles)
                                          gnutls-trustfiles)))))

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely, bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Thu, 24 Oct 2013 23:17:38 -0400
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
> 
> Thread 1 (Thread 124728.0x31b44):
> #0  0x6d099192 in _gnutls_record_buffer_get_size ()
>    from C:\kosowsky\bin\libgnutls-28.dll
> #1  0x6d0995c8 in gnutls_record_check_pending ()
>    from C:\kosowsky\bin\libgnutls-28.dll
> #2  0x01015e15 in wait_reading_process_output ()
> #3  0x0088e7d8 in ?? ()
> (gdb)

Thanks, this tells everything, I think.

Ted, could you please help me out a bit here?  I think I understand
what is going on: we are passing a NULL session to
gnutls_record_check_pending.  What happens next is predictable:
gnutls_record_check_pending calls _gnutls_record_buffer_get_size,
which does this:

  return session->internals.record_buffer.byte_length;

I.e., it dereferences a NULL pointer.

However, to find the best place where to fix this in Emacs, could you
please help me understand in more detail what happens in this case?  I
imagine that gnutls-boot is called with the parameters that specify a
certificate file that GnuTLS cannot access.  But why isn't this caught
inside gnutls-boot, and how come we allow a NULL gnutls_state be
plugged into the process object?  This fragment from gnutls-boot:

  GNUTLS_LOG (1, max_log_level, "gnutls_init");
  ret = fn_gnutls_init (&state, GNUTLS_CLIENT);
  XPROCESS (proc)->gnutls_state = state;
  if (ret < GNUTLS_E_SUCCESS)
    return gnutls_make_error (ret);

ought to fail.  The bug report cites this error message:

  GnuTLS error: #<proces IMAP over SSL>, -64

Is that error the result of the above error checking?  If so, perhaps
the problem is that we leave the process object marked as a GnuTLS
process, but with a NULL state?  Should we remove the mark, or maybe
delete the process object in gnutls-negotiate?

Thanks.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Fri, 25 Oct 2013 17:09:13 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> However, to find the best place where to fix this in Emacs, could you
EZ> please help me understand in more detail what happens in this case?  I
EZ> imagine that gnutls-boot is called with the parameters that specify a
EZ> certificate file that GnuTLS cannot access.  But why isn't this caught
EZ> inside gnutls-boot, and how come we allow a NULL gnutls_state be
EZ> plugged into the process object?  This fragment from gnutls-boot:

EZ>   GNUTLS_LOG (1, max_log_level, "gnutls_init");
EZ>   ret = fn_gnutls_init (&state, GNUTLS_CLIENT);
EZ>   XPROCESS (proc)->gnutls_state = state;
EZ>   if (ret < GNUTLS_E_SUCCESS)
EZ>     return gnutls_make_error (ret);

EZ> ought to fail.  The bug report cites this error message:

EZ>   GnuTLS error: #<proces IMAP over SSL>, -64

EZ> Is that error the result of the above error checking?

(btw, no idea why the subject keeps getting duplicated but it's annoying)

Hard to tell if -64 means anything, but it shouldn't cause a crash.

EZ> If so, perhaps the problem is that we leave the process object
EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
EZ> the mark, or maybe delete the process object in gnutls-negotiate?

I would abort with a message like any other error.  Here's what we do:

	  if (STRINGP (trustfile))
	    {
	      GNUTLS_LOG2 (1, max_log_level, "setting the trustfile: ",
			   SSDATA (trustfile));
	      ret = fn_gnutls_certificate_set_x509_trust_file
		(x509_cred,
		 SSDATA (trustfile),
		 file_format);

	      if (ret < GNUTLS_E_SUCCESS)
		return gnutls_make_error (ret);
	    }
	  else
	    {
	      emacs_gnutls_deinit (proc);
	      error ("Invalid trustfile");
	    }

In other words, we pass the file down and assume the return code from
`fn_gnutls_certificate_set_x509_trust_file' will be accurate.  In this
case I don't know what it returned but would assume GNUTLS_E_SUCCESS
since there was no error.

In general I trust the return codes and don't verify the state
explicitly.  I don't see how the `gnutls_state' could have been set to
NULL by a missing trustfile; the function call that sets the trustfile
only touches the `x509_cred' variable.  Could this NULL be happening
later?

In this specific case I traced the function calls all the way down the
GnuTLS library and see that it checks that file was opened correctly and
more.  It should return an error for that missing file.

http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=blob_plain;f=lib/gnutls_x509.c;hb=HEAD
gnutls.git/lib/gnutls_x509.c:gnutls_certificate_set_x509_trust_file()

...
  cas.data = (void*)read_binary_file (cafile, &size);
  if (cas.data == NULL)
    {
      gnutls_assert ();
      return GNUTLS_E_FILE_ERROR;
    }

and read_binary_file uses this code:

http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=blob_plain;f=gl/read-file.c;hb=HEAD

gnutls.git/gl/read-file.c:fread_file():

...
    if (fstat (fileno (stream), &st) >= 0 && S_ISREG (st.st_mode))
 
I can't test this specific issue so it's hard to know where the error is
happening.  Can anyone duplicate that problem?

I could add extra debugging and checking but I don't understand, if the
file name is invalid, how the fread_file() function is succeeding.
Maybe that's part of the problem.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
> Date: Fri, 25 Oct 2013 11:38:05 -0400
> 
> EZ> If so, perhaps the problem is that we leave the process object
> EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
> EZ> the mark, or maybe delete the process object in gnutls-negotiate?
> 
> I would abort with a message like any other error.

Maybe you should install a change that does that, and see if it solves
the problem.

> Here's what we do:
> 
> 	  if (STRINGP (trustfile))
> 	    {
> 	      GNUTLS_LOG2 (1, max_log_level, "setting the trustfile: ",
> 			   SSDATA (trustfile));
> 	      ret = fn_gnutls_certificate_set_x509_trust_file
> 		(x509_cred,
> 		 SSDATA (trustfile),
> 		 file_format);
> 
> 	      if (ret < GNUTLS_E_SUCCESS)
> 		return gnutls_make_error (ret);
> 	    }
> 	  else
> 	    {
> 	      emacs_gnutls_deinit (proc);
> 	      error ("Invalid trustfile");
> 	    }
> 
> In other words, we pass the file down and assume the return code from
> `fn_gnutls_certificate_set_x509_trust_file' will be accurate.  In this
> case I don't know what it returned but would assume GNUTLS_E_SUCCESS
> since there was no error.
> 
> In general I trust the return codes and don't verify the state
> explicitly.  I don't see how the `gnutls_state' could have been set to
> NULL by a missing trustfile; the function call that sets the trustfile
> only touches the `x509_cred' variable.  Could this NULL be happening
> later?

We _begin_ by setting gnutls_state to NULL.  What could possibly
happen is that we somehow let the process object with a NULL state
escape from the initialization step, and then
wait_reading_process_output stumbles on it and tries to use it,
because the gnutls_p flag is also set right at the beginning.

How about if we set the gnutls_p flag only when the whole
initialization succeeds completely?  It's only then that the process
is ready to be used in conjunction with GnuTLS, isn't it?

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Stefan Monnier]

> I'm surprised that such a function doesn't (yet) exist in Emacs...

It's largely a reflection about the fact that such a need is not very
frequent, and that most people who bump into it prefer hacking their way
around the problem rather than fix it for good.

> 1. Adding expand-file-name will work for any file name that
>    cygwin-mount is designed to address (this is by definition)

"By definition" of cygwin-mount, not of expand-file-name, sadly.

> 2. Similarly, adding expand-file-name will likely work for any other
>    Magic File handler that works similar to cygwin-mount in terms of

I don't know of any other that works this way.

> Basically, this version does 2 things:
> 1. Pre-expands the file name *if* there is a Magic File handler for that
>    file path (this will do nothing if there is no handler for that path)
> 2. Tests file-exists-p with all Magic File handlers shut off which
>    tests to make sure that the OS primitives will work on the actual
>    file path that will be passed

It's a cleaner workaround, but I'd rather we come up with an actual fix.
Do you think you can try to make a patch that adds (in files.el) a new
function meant to turn an Elisp file name into an OS file name or return
nil if the file can't be accessed by OS primitives?  This function would
just delegate its work to the file-name-handler if any, and otherwise
just return its argument unchanged.


        Stefan

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Stefan Monnier wrote at about 21:52:24 -0400 on Friday, October 25, 2013:
 > > Basically, this version does 2 things:
 > > 1. Pre-expands the file name *if* there is a Magic File handler for that
 > >    file path (this will do nothing if there is no handler for that path)
 > > 2. Tests file-exists-p with all Magic File handlers shut off which
 > >    tests to make sure that the OS primitives will work on the actual
 > >    file path that will be passed
 > 
 > It's a cleaner workaround, but I'd rather we come up with an actual fix.
 > Do you think you can try to make a patch that adds (in files.el) a new
 > function meant to turn an Elisp file name into an OS file name or return
 > nil if the file can't be accessed by OS primitives?  This function would
 > just delegate its work to the file-name-handler if any, and otherwise
 > just return its argument unchanged.
 > 
 > 
 >         Stefan

There was a small logic error in my original proposed patch...
Here is a corrected version:

--- gnutls.el      2013-03-17 13:52:40.000000000 -0400
+++ gnutls.el.new  2013-10-29 01:10:49.784647900 -0400
@@ -174,7 +174,13 @@
   (let* ((type (or type 'gnutls-x509pki))
          (trustfiles (or trustfiles
                          (delq nil
-                               (mapcar (lambda (f) (and f (file-exists-p f) f))
+                               (mapcar (lambda (f)
+                                        (when f
+                                          (if (find-file-name-handler
+                                               f 'expand-file-name)
+                                              (setq f (expand-file-name f)))
+                                          (let (file-name-handler-alist)
+                                            (if (file-exists-p f) f))))
                                        (if (functionp gnutls-trustfiles)
                                            (funcall gnutls-trustfiles)
                                          gnutls-trustfiles)))))

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Fri, 25 Oct 2013 21:52:24 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> It's a cleaner workaround, but I'd rather we come up with an actual fix.
SM> Do you think you can try to make a patch that adds (in files.el) a new
SM> function meant to turn an Elisp file name into an OS file name or return
SM> nil if the file can't be accessed by OS primitives?  This function would
SM> just delegate its work to the file-name-handler if any, and otherwise
SM> just return its argument unchanged.

Would any C-facing code, then, need to apply this function before
passing a filename to the OS?  Or do you think this function belongs in
every package, which is a much bigger list of potential fixes?

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 06:42:36 -0500 on Sunday, November 3, 2013:
 > On Fri, 25 Oct 2013 21:52:24 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
 > 
 > SM> It's a cleaner workaround, but I'd rather we come up with an actual fix.
 > SM> Do you think you can try to make a patch that adds (in files.el) a new
 > SM> function meant to turn an Elisp file name into an OS file name or return
 > SM> nil if the file can't be accessed by OS primitives?  This function would
 > SM> just delegate its work to the file-name-handler if any, and otherwise
 > SM> just return its argument unchanged.
 > 
 > Would any C-facing code, then, need to apply this function before
 > passing a filename to the OS?  Or do you think this function belongs in
 > every package, which is a much bigger list of potential fixes?

Looking through files.el, I noticed that there is a function that
seems to be similar in intent.

The function is called convert-standard-filename. All it really does
currently is replace invalid characters/filenames for
cygwin/windows-nt/ms-dos.

So one potential solution would be to add a hook to the file name
handler at the beginning of this function and then add this function
to the (optional) functions that Magic file handlers could
address. Thus, for example, magic file types like 'cygwin-mount' would
use this hook while magic file types like 'ange-ftp' would
intentionally not have a file handler here since they would presumably
never touch actual local file system files.

There are however 3 problems with this approach:

1. This would require adding a new (lower-level) function to the list
   of functions requiring a magic file handler and it might take some
   time for each maintainer to add the required handler to their Magic
   code.

2. Since some functions already handle conversion using higher order
   primitives like expand-file-name or true-filename (see #3), the
   file name conversion would often be duplicated.

3. Indeed (surprise, surprise), most of the core functions in files.el
   actually use 'expand-file-name' to expand the file name sometime
   before referencing a file. In fact, expand-file-name is called 64
   times and file-truename is called 13 times in the files.el elisp
   code.  (Conversely, convert-standard-filename is only called by the
   functions make-auto-save-file-name and make-backup-file-name-1.)

So, it seems that calling 'expand-file-name' or 'file-truename' far
from being a "hack" is actually the standard way that emacs uses to
generate an OS-accessible file name. In particular, the workhorse
primitives find-file-noselect and find-file-noselect1 both use
'expand-file-name'.

Given that adding 'expand-file-name' seems to be standard usage for
creating OS-accessible file names,  I don't see why we wouldn't
want to use the same paradigm for expanding file names before passing
them off to C-code (like gnutls) that bypasses the standard elisp file
access primitives. Indeed, such usage should be required best practice
precisely to avoid the problems uncovered in gnutls.el.

So, I guess short of rewriting many of the functions in files.el to
consistently pass file names through some lower level OS-conversion
primitive rather than using expand-file-name or file-truename and
short of similarly adding such a lower level primitive to all relevant
Magic file handlers (including avoiding duplicate calls when higher
level primitives like expand-file-name have already been used), I
think that my proposed patch is not only simpler but also more
consistent with the current usage in files.el

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

Ping!

Ted, I'd like to solve this one way or the other.  We have the data,
so the onus is now on us to act on it.

Thanks.

> Date: Fri, 25 Oct 2013 21:37:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
> > From: Ted Zlatanov <tzz@lifelogs.com>
> > Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
> > Date: Fri, 25 Oct 2013 11:38:05 -0400
> > 
> > EZ> If so, perhaps the problem is that we leave the process object
> > EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
> > EZ> the mark, or maybe delete the process object in gnutls-negotiate?
> > 
> > I would abort with a message like any other error.
> 
> Maybe you should install a change that does that, and see if it solves
> the problem.
> 
> > Here's what we do:
> > 
> > 	  if (STRINGP (trustfile))
> > 	    {
> > 	      GNUTLS_LOG2 (1, max_log_level, "setting the trustfile: ",
> > 			   SSDATA (trustfile));
> > 	      ret = fn_gnutls_certificate_set_x509_trust_file
> > 		(x509_cred,
> > 		 SSDATA (trustfile),
> > 		 file_format);
> > 
> > 	      if (ret < GNUTLS_E_SUCCESS)
> > 		return gnutls_make_error (ret);
> > 	    }
> > 	  else
> > 	    {
> > 	      emacs_gnutls_deinit (proc);
> > 	      error ("Invalid trustfile");
> > 	    }
> > 
> > In other words, we pass the file down and assume the return code from
> > `fn_gnutls_certificate_set_x509_trust_file' will be accurate.  In this
> > case I don't know what it returned but would assume GNUTLS_E_SUCCESS
> > since there was no error.
> > 
> > In general I trust the return codes and don't verify the state
> > explicitly.  I don't see how the `gnutls_state' could have been set to
> > NULL by a missing trustfile; the function call that sets the trustfile
> > only touches the `x509_cred' variable.  Could this NULL be happening
> > later?
> 
> We _begin_ by setting gnutls_state to NULL.  What could possibly
> happen is that we somehow let the process object with a NULL state
> escape from the initialization step, and then
> wait_reading_process_output stumbles on it and tries to use it,
> because the gnutls_p flag is also set right at the beginning.
> 
> How about if we set the gnutls_p flag only when the whole
> initialization succeeds completely?  It's only then that the process
> is ready to be used in conjunction with GnuTLS, isn't it?
> 
> 
> 
> 

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Sun, 03 Nov 2013 10:12:27 -0500
> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
> Looking through files.el, I noticed that there is a function that
> seems to be similar in intent.
> 
> The function is called convert-standard-filename. All it really does
> currently is replace invalid characters/filenames for
> cygwin/windows-nt/ms-dos.

No, please leave that function alone.  Its purpose is different: to
convert a _standard_ file name used in Emacs, such as ".emacs", to
something the underlying system can use.  IOW, that function is so
that Posix systems could use any file name they want, and non-Posix
systems adapt.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 19:32:04 +0200 on Sunday, November 3, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Sun, 03 Nov 2013 10:12:27 -0500
 > > Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
 > > 
 > > Looking through files.el, I noticed that there is a function that
 > > seems to be similar in intent.
 > > 
 > > The function is called convert-standard-filename. All it really does
 > > currently is replace invalid characters/filenames for
 > > cygwin/windows-nt/ms-dos.
 > 
 > No, please leave that function alone.  Its purpose is different: to
 > convert a _standard_ file name used in Emacs, such as ".emacs", to
 > something the underlying system can use.  IOW, that function is so
 > that Posix systems could use any file name they want, and non-Posix
 > systems adapt.

I agree... as the rest of my note illustrates... it seems clear from
reading the elisp code in file.el that expand-file-name and
file-truename are the appropriate functions to use for translating
file names to ones retrievable by the underlying (local) OS.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Stefan Monnier]

> Would any C-facing code, then, need to apply this function before
> passing a filename to the OS?

I guess so.  I'm much too soaked in the POSIX world where this new
function would default to being a nop, to have a good idea of exactly
where/when this would be needed.

My first idea, w.r.t cygwin-mount was that it should operate "at the
boundary" where we receive filenames from outside, such as in
substitute-in-file-name and when processing command line arguments.

But if we want cygwin names to work on variables set via Elisp
customization, that means it would have to operate at a much
lower level, so maybe "any C-facing code" is the right place, indeed.

But it should probably be "C-facing code except for the code in
file-handlers", since presumably those would already be handled by the
magic-file-handler so by the time we get to the default code there's no
need for any translation.

IOW it should be "any C-facing code which uses files but can't be passed
through the magic file handler".  Of course, it can also be used by
Elisp code when passing file names to external processes.


        Stefan

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Sun, 03 Nov 2013 19:32:04 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: <emacs@kosowsky.org>
>> Date: Sun, 03 Nov 2013 10:12:27 -0500
>> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
>> 
>> Looking through files.el, I noticed that there is a function that
>> seems to be similar in intent.
>> 
>> The function is called convert-standard-filename. All it really does
>> currently is replace invalid characters/filenames for
>> cygwin/windows-nt/ms-dos.

EZ> No, please leave that function alone.  Its purpose is different: to
EZ> convert a _standard_ file name used in Emacs, such as ".emacs", to
EZ> something the underlying system can use.  IOW, that function is so
EZ> that Posix systems could use any file name they want, and non-Posix
EZ> systems adapt.

Isn't that exactly what's breaking here?  We specify a POSIX file name
in `/usr/...' that's not translated automatically to something the
system can handle.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Fri, 25 Oct 2013 21:37:14 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
>> Date: Fri, 25 Oct 2013 11:38:05 -0400
>> 
EZ> If so, perhaps the problem is that we leave the process object
EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
EZ> the mark, or maybe delete the process object in gnutls-negotiate?
>> 
>> I would abort with a message like any other error.

EZ> Maybe you should install a change that does that, and see if it solves
EZ> the problem.

You mean like this?  Should I have a special check for "gnutls_p is set
but gnutls_state is NULL"?

=== modified file 'src/process.c'
--- src/process.c       2013-11-04 06:09:03 +0000
+++ src/process.c       2013-11-04 16:43:26 +0000
@@ -4609,7 +4609,7 @@
                      {
                        struct Lisp_Process *p =
                          XPROCESS (chan_process[channel]);
-                       if (p && p->gnutls_p && p->infd
+                       if (p && p->gnutls_p && p->gnutls_state && p->infd
                            && ((emacs_gnutls_record_check_pending
                                 (p->gnutls_state))
                                > 0))
@@ -4623,6 +4623,7 @@
                {
                  /* Check this specific channel. */
                  if (wait_proc->gnutls_p /* Check for valid process.  */
+                      && p->gnutls_state
                      /* Do we have pending data?  */
                      && ((emacs_gnutls_record_check_pending
                           (wait_proc->gnutls_state))
@@ -5004,7 +5005,7 @@
          proc_buffered_char[channel] = -1;
        }
 #ifdef HAVE_GNUTLS
-      if (p->gnutls_p)
+      if (p->gnutls_p && p->gnutls_state)
        nbytes = emacs_gnutls_read (p, chars + carryover + buffered,
                                    readmax - buffered);
       else
@@ -5498,7 +5499,7 @@
 #endif
            {
 #ifdef HAVE_GNUTLS
-             if (p->gnutls_p)
+             if (p->gnutls_p && p->gnutls_state)
                written = emacs_gnutls_write (p, cur_buf, cur_len);
              else
 #endif

EZ> We _begin_ by setting gnutls_state to NULL.  What could possibly
EZ> happen is that we somehow let the process object with a NULL state
EZ> escape from the initialization step, and then
EZ> wait_reading_process_output stumbles on it and tries to use it,
EZ> because the gnutls_p flag is also set right at the beginning.

EZ> How about if we set the gnutls_p flag only when the whole
EZ> initialization succeeds completely?  It's only then that the process
EZ> is ready to be used in conjunction with GnuTLS, isn't it?

You're absolutely right.  Can you check if this patch is good?  It looks
OK to me.

Thank you very much for the guidance.

Ted

=== modified file 'src/gnutls.c'
--- src/gnutls.c        2013-11-04 06:09:03 +0000
+++ src/gnutls.c        2013-11-04 16:36:18 +0000
@@ -810,7 +822,6 @@
   c_hostname = SSDATA (hostname);
 
   state = XPROCESS (proc)->gnutls_state;
-  XPROCESS (proc)->gnutls_p = 1;
 
   if (TYPE_RANGED_INTEGERP (int, loglevel))
     {
@@ -833,7 +844,6 @@
   emacs_gnutls_deinit (proc);
 
   /* Mark PROC as a GnuTLS process.  */
-  XPROCESS (proc)->gnutls_p = 1;
   XPROCESS (proc)->gnutls_state = NULL;
   XPROCESS (proc)->gnutls_x509_cred = NULL;
   XPROCESS (proc)->gnutls_anon_cred = NULL;
@@ -1093,6 +1103,9 @@
       fn_gnutls_x509_crt_deinit (gnutls_verify_cert);
     }
 
+  // Only set this flag if the whole initialization succeeded.
+  XPROCESS (proc)->gnutls_p = 1;
+
   return gnutls_make_error (ret);
 }

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
> Gmane-Reply-To-List: yes
> Date: Mon, 04 Nov 2013 11:28:36 -0500
> 
> >> The function is called convert-standard-filename. All it really does
> >> currently is replace invalid characters/filenames for
> >> cygwin/windows-nt/ms-dos.
> 
> EZ> No, please leave that function alone.  Its purpose is different: to
> EZ> convert a _standard_ file name used in Emacs, such as ".emacs", to
> EZ> something the underlying system can use.  IOW, that function is so
> EZ> that Posix systems could use any file name they want, and non-Posix
> EZ> systems adapt.
> 
> Isn't that exactly what's breaking here?

No.

> We specify a POSIX file name in `/usr/...' that's not translated
> automatically to something the system can handle.

The Windows filesystems can handle /usr/... very well.  The problem
here is that some external entity is assigning a semantics to that
file name that is different from the meaning it has on the native
filesystem.  By contrast, convert-standard-filename is about _form_,
not about _semantics_.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
> Date: Mon, 04 Nov 2013 11:44:41 -0500
> 
> EZ> If so, perhaps the problem is that we leave the process object
> EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
> EZ> the mark, or maybe delete the process object in gnutls-negotiate?
> >> 
> >> I would abort with a message like any other error.
> 
> EZ> Maybe you should install a change that does that, and see if it solves
> EZ> the problem.
> 
> You mean like this?

Something like that, yes.

> Should I have a special check for "gnutls_p is set but gnutls_state
> is NULL"?

That's what this patch does, doesn't it?  Or did I misunderstand you?

> EZ> We _begin_ by setting gnutls_state to NULL.  What could possibly
> EZ> happen is that we somehow let the process object with a NULL state
> EZ> escape from the initialization step, and then
> EZ> wait_reading_process_output stumbles on it and tries to use it,
> EZ> because the gnutls_p flag is also set right at the beginning.
> 
> EZ> How about if we set the gnutls_p flag only when the whole
> EZ> initialization succeeds completely?  It's only then that the process
> EZ> is ready to be used in conjunction with GnuTLS, isn't it?
> 
> You're absolutely right.  Can you check if this patch is good?  It looks
> OK to me.

It compiles fine.  But since I cannot reproduce the original problem,
I'd ask the OP to please see if this fixes that problem.

Thanks.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Mon, 04 Nov 2013 19:06:48 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
>> Date: Mon, 04 Nov 2013 11:44:41 -0500
>> 
EZ> If so, perhaps the problem is that we leave the process object
EZ> marked as a GnuTLS process, but with a NULL state?  Should we remove
EZ> the mark, or maybe delete the process object in gnutls-negotiate?
>> >> 
>> >> I would abort with a message like any other error.
>> 
EZ> Maybe you should install a change that does that, and see if it solves
EZ> the problem.
>> 
>> You mean like this?

EZ> Something like that, yes.

>> Should I have a special check for "gnutls_p is set but gnutls_state
>> is NULL"?

EZ> That's what this patch does, doesn't it?  Or did I misunderstand you?

The process.c change checks that "gnutls_p is set and gnutls_state
is not NULL" but wouldn't make a note of the case where gnutls_state got
set to NULL somehow.  I meant to have a special check for that case and
maybe issue a message... but it seems a bit like overkill.

>> You're absolutely right.  Can you check if this patch is good?  It looks
>> OK to me.

EZ> It compiles fine.  But since I cannot reproduce the original problem,
EZ> I'd ask the OP to please see if this fixes that problem.

Right, I needed your review to ensure I was not missing something :)

OP, can you please try my patch?

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 13:05:40 -0500 on Monday, November 4, 2013:
 > OP, can you please try my patch?

Sure... would you mind sending me a compiled version since I may be
lacking the build environment...

Thanks,
Jeff

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Mon, 04 Nov 2013 17:14:35 -0500 <emacs@kosowsky.org> wrote: 

> Ted Zlatanov wrote at about 13:05:40 -0500 on Monday, November 4, 2013:
>> OP, can you please try my patch?

> Sure... would you mind sending me a compiled version since I may be
> lacking the build environment...

I also lack such, unfortunately.

I've pushed my changes to trunk just now and would appreciate it if you
picked up a build made afterwards to try.

Thanks
Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 21:30:42 -0500 on Monday, November 4, 2013:
 > On Mon, 04 Nov 2013 17:14:35 -0500 <emacs@kosowsky.org> wrote: 
 > 
 > > Ted Zlatanov wrote at about 13:05:40 -0500 on Monday, November 4, 2013:
 > >> OP, can you please try my patch?
 > 
 > > Sure... would you mind sending me a compiled version since I may be
 > > lacking the build environment...
 > 
 > I also lack such, unfortunately.
 > 
 > I've pushed my changes to trunk just now and would appreciate it if you
 > picked up a build made afterwards to try.
 > 

Where is "trunk" located?
The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
which still crashes as before...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Alp Aker]

[-- Attachment #1: Type: text/plain, Size: 193 bytes --]

> Where is "trunk" located?
> The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
> which still crashes as before...

He meant emacs trunk:

bzr://bzr.sv.gnu.org/emacs/trunk

[-- Attachment #2: Type: text/html, Size: 437 bytes --]

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Alp Aker wrote at about 18:16:14 -0500 on Tuesday, November 5, 2013:
 > > Where is "trunk" located?
 > > The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
 > > which still crashes as before...
 > 
 > He meant emacs trunk:
 > 
 > bzr://bzr.sv.gnu.org/emacs/trunk

I'm confused... the patch Ted wants me to test is part of the gnutls
libraries... which at least until now haven't been part of the windows
emacs package...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Tue, 05 Nov 2013 18:11:12 -0500
> Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
>  > I've pushed my changes to trunk just now and would appreciate it if you
>  > picked up a build made afterwards to try.
>  > 
> 
> Where is "trunk" located?
> The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
> which still crashes as before...

You can find snapshot builds of the trunk here:

  https://sourceforge.net/projects/emacs-bin/files/snapshots/

Wait until there's a build there from a date after the one when Ted
said he pushed the changes, then download and try that.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

OK thanks
Eli Zaretskii wrote at about 05:51:52 +0200 on Wednesday, November 6, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Tue, 05 Nov 2013 18:11:12 -0500
 > > Cc: emacs@kosowsky.org, 15648@debbugs.gnu.org
 > > 
 > >  > I've pushed my changes to trunk just now and would appreciate it if you
 > >  > picked up a build made afterwards to try.
 > >  > 
 > > 
 > > Where is "trunk" located?
 > > The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
 > > which still crashes as before...
 > 
 > You can find snapshot builds of the trunk here:
 > 
 >   https://sourceforge.net/projects/emacs-bin/files/snapshots/
 > 
 > Wait until there's a build there from a date after the one when Ted
 > said he pushed the changes, then download and try that.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Tue, 05 Nov 2013 18:54:49 -0500 <emacs@kosowsky.org> wrote: 

> Alp Aker wrote at about 18:16:14 -0500 on Tuesday, November 5, 2013:
>> > Where is "trunk" located?
>> > The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
>> > which still crashes as before...
>> 
>> He meant emacs trunk:
>> 
>> bzr://bzr.sv.gnu.org/emacs/trunk

> I'm confused... the patch Ted wants me to test is part of the gnutls
> libraries... which at least until now haven't been part of the windows
> emacs package...

The patch is against Emacs trunk.  Downloading an Emacs build made after
my last commit (November 5 2013 or later) will let you confirm if the
fix is good or not.

Thanks
Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 18:58:05 +0200 on Monday, November 4, 2013:
 > > From: Ted Zlatanov <tzz@lifelogs.com>
 > > Cc: emacs@kosowsky.org,  15648@debbugs.gnu.org
 > > Gmane-Reply-To-List: yes
 > > Date: Mon, 04 Nov 2013 11:28:36 -0500
 > > 
 > > >> The function is called convert-standard-filename. All it really does
 > > >> currently is replace invalid characters/filenames for
 > > >> cygwin/windows-nt/ms-dos.
 > > 
 > > EZ> No, please leave that function alone.  Its purpose is different: to
 > > EZ> convert a _standard_ file name used in Emacs, such as ".emacs", to
 > > EZ> something the underlying system can use.  IOW, that function is so
 > > EZ> that Posix systems could use any file name they want, and non-Posix
 > > EZ> systems adapt.
 > > 
 > > Isn't that exactly what's breaking here?
 > 
 > No.
 > 
 > > We specify a POSIX file name in `/usr/...' that's not translated
 > > automatically to something the system can handle.
 > 
 > The Windows filesystems can handle /usr/... very well.  The problem
 > here is that some external entity is assigning a semantics to that
 > file name that is different from the meaning it has on the native
 > filesystem.  By contrast, convert-standard-filename is about _form_,
 > not about _semantics_.

Well, there is also the problem that "/usr" is never present as a root
path on any (standard) Windows machine, so that the path commented as
being valid for cygwin actually never works! 

Indeed, in the cygwin case, "/usr" only makes sense relative to the
cygwin root (which by default is C:\cygwin but could be anywhere). The
purpose of cygwin-mount magic file handling is precisely to insert the
cygwin root into the path so that "/usr" becomes for example,
"/c/cygwin/usr".

The absence of cygwin-mount magic file handling when a file name is
passed directly to the gnutls c-code without going through any of the
standard magic-file-handling file access routines is the crux of the
problem.

So, again I see only 2 solutions:
1. Change (or omit) the "/usr" path and make it relative to cygwin
   root (though this would not work generally since cygwin root is
   changeable)

2. Implement magic handling so that paths are automagically translated
   to be correct at the file system level. In this case, by inserting
   the cygwin root.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 10:53:03 -0500 on Monday, November 11, 2013:
 > On Tue, 05 Nov 2013 18:54:49 -0500 <emacs@kosowsky.org> wrote: 
 > 
 > > Alp Aker wrote at about 18:16:14 -0500 on Tuesday, November 5, 2013:
 > >> > Where is "trunk" located?
 > >> > The latest version in ftp://ftp.gnutls.org/gcrypt/gnutls/w32/ is 3.2.6
 > >> > which still crashes as before...
 > >> 
 > >> He meant emacs trunk:
 > >> 
 > >> bzr://bzr.sv.gnu.org/emacs/trunk
 > 
 > > I'm confused... the patch Ted wants me to test is part of the gnutls
 > > libraries... which at least until now haven't been part of the windows
 > > emacs package...
 > 
 > The patch is against Emacs trunk.  Downloading an Emacs build made after
 > my last commit (November 5 2013 or later) will let you confirm if the
 > fix is good or not.
 > 
 > Thanks
 > Ted

I tested it against the November 6th build.

It seems to work in the sense that now instead of crashing emacs,
gnutls just throws me into the emacs debugger:

Debugger entered--Lisp error: (gnutls-error #<process tls> -64)
  signal(gnutls-error (#<process tls> -64))
  gnutls-negotiate(:process #<process tls> :type gnutls-x509pki
  :hostname "imap.gmail.com")
  open-gnutls-stream("tls" "tls-buffer" "imap.gmail.com" "imaps")
  eval-region(19 83 t #[257 "\300\242b\210\301\207" [(83)
  (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")] 2
  "\n\n(fn IGNORE)"])  ; Reading at buffer position 83
  eval-defun-2()
  eval-defun(nil)
  call-interactively(eval-defun nil nil)
  command-execute(eval-defun)


Presumably, this error is due to the fact that
"/usr/ssl/certs/ca-bundle.crt" is not a valid file on Cygwin (despite
the comment in gnutls.el implying that it is!)

Of course, it all works fine with the elisp patch that I had
previously submitted.

Note it also works fine in a degenerate way without cygwin-mount since
the mapcar file-exists-p statement returns that none of the files
exists.

So, now that the bug that crashes the emacs process has been fixed,
perhaps it's time to make gnutls-trustfiles actually work under
cygwin.

Indeed, it's worse than just not "working". If cygwin-mount is loaded
then open-gnutls-stream will throw an (incomprehensible) error each
and every time due to the fact that file-exists-p shows the trustfile
exists while the C-code is passed a file path that is not recognized by
the OS. So, there remains a fundamental incompatibility between
cygwin-mount and gnutls.el.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Mon, 11 Nov 2013 14:12:34 -0500 <emacs@kosowsky.org> wrote: 

> The absence of cygwin-mount magic file handling when a file name is
> passed directly to the gnutls c-code without going through any of the
> standard magic-file-handling file access routines is the crux of the
> problem.

> So, again I see only 2 solutions:
> 1. Change (or omit) the "/usr" path and make it relative to cygwin
>    root (though this would not work generally since cygwin root is
>    changeable)

> 2. Implement magic handling so that paths are automagically translated
>    to be correct at the file system level. In this case, by inserting
>    the cygwin root.

Could we just call http://cygwin-lite.sourceforge.net/html/cygpath.html
on the path (or the equivalent C function)?  It seems to be guaranteed
to work but may be very slow with the execution overhead.

Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 14:42:46 -0500 on Monday, November 11, 2013:
 > On Mon, 11 Nov 2013 14:12:34 -0500 <emacs@kosowsky.org> wrote: 
 > 
 > > The absence of cygwin-mount magic file handling when a file name is
 > > passed directly to the gnutls c-code without going through any of the
 > > standard magic-file-handling file access routines is the crux of the
 > > problem.
 > 
 > > So, again I see only 2 solutions:
 > > 1. Change (or omit) the "/usr" path and make it relative to cygwin
 > >    root (though this would not work generally since cygwin root is
 > >    changeable)
 > 
 > > 2. Implement magic handling so that paths are automagically translated
 > >    to be correct at the file system level. In this case, by inserting
 > >    the cygwin root.
 > 
 > Could we just call http://cygwin-lite.sourceforge.net/html/cygpath.html
 > on the path (or the equivalent C function)?  It seems to be guaranteed
 > to work but may be very slow with the execution overhead.
 > 
 > Ted

It would work for cygwin... but I see two limitations:

1. The same problem that affects cygwin-mount, would affect any other
   magic file handler that translates file names, since once again
   file-exists-p would return true, while the path passed to the
   c-code would in general not point to an OS-recognizable file.

   I could even right my own file handler called
   'my-2nd-home' whereby any path beginning with ~~ would be
   translated to /home/my-2nd-home. Then, if I put such a file name in
   gnutls-trustfiles, it would cause gnutls to throw an error when the
   path is passed to the c-code.

   Again, my patch would solve the problem

2. The suggestion seems to be far kluggier than my suggested patch
   since it relies on a user elisp routine passing executing a system
   call to return a path every time a gnutls connection is
   requested. At least cygwin-mount does this only once at startup and
   abstracts the translation away from higher level routines -- which
   is the entire purpose of a magic file handler.

   If cygpath belongs anywhere, it would be in the magic file handling
   code. Though cygwin-mount does this in a "smarter" way by calling
   'mount' once to determine the cygwin prefix once-and-for-all, and
   then whenever a path is passed, the magic handler automatically
   prefixes on the cygwin mount prefix.

Once again, the problem is not cygwin-mount, per-se. The problem is
the inconsistency between using the magic file enabled predicate
file-exists-p to test for a file and then passing that same file name
to c-code that knows nothing about magic file handlers.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Achim Gratz]

Ted Zlatanov writes:
> Could we just call http://cygwin-lite.sourceforge.net/html/cygpath.html
> on the path (or the equivalent C function)?  It seems to be guaranteed
> to work but may be very slow with the execution overhead.

Please refer to the official Cygwin documentation, not some unmaintained
project that stopped getting updates over a decade ago:

http://cygwin.com/cygwin-ug-net/using-utils.html

Specifically for Emacs, the "-m" option to cygpath seems the most
suitable.  Please do not link to the Cygwin DLL from Windows Emacs
(i.e. "using the equivalent C function").  The only reason to use a
native Emacs on Windows (for me anyway) is that I can update Cygwin
while I keep Emacs running and that wouldn't be possible if the DLL was
locked because it was still in use.  For anything else (if you don#t
want to start an X server), there's emacs-w32 from Cygwin.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Mon, 11 Nov 2013 14:12:34 -0500
> Cc: Ted Zlatanov <tzz@lifelogs.com>, emacs@kosowsky.org, 15648@debbugs.gnu.org
> 
> Well, there is also the problem that "/usr" is never present as a root
> path on any (standard) Windows machine, so that the path commented as
> being valid for cygwin actually never works! 

Please don't assume that what happens on your machine happens on
everyone else's.  E.g., my systems do have a /usr directory at least
on some of the drives.  No, I don't run Cygwin.

More generally, "/foo/bar" is a valid file name on Windows, whether
foo equals "usr" or not.  It is simply wrong to assume that if you see
/usr on Windows, it _must_ mean a Cygwin mount.  This kind of
reasoning is simply a non-starter.

> The absence of cygwin-mount magic file handling when a file name is
> passed directly to the gnutls c-code without going through any of the
> standard magic-file-handling file access routines is the crux of the
> problem.

No, the crux of the problem is that you are trying to use a natively
built Emacs with file names that make sense only for Cygwin programs.
If you want Cygwin semantics of file names, use a Cygwin build of
Emacs, and this problem will immediately go away.  Alternatively, use
file names for your certificates that native Windows programs can
find, and the problem will also go away immediately.

> So, again I see only 2 solutions:
> 1. Change (or omit) the "/usr" path and make it relative to cygwin
>    root (though this would not work generally since cygwin root is
>    changeable)
> 
> 2. Implement magic handling so that paths are automagically translated
>    to be correct at the file system level. In this case, by inserting
>    the cygwin root.

  3. Don't mix Cygwin file names with native Windows programs.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Mon, 11 Nov 2013 14:40:29 -0500
> Cc: Alp Aker <alptekin.aker@gmail.com>, emacs@kosowsky.org,
> 	15648@debbugs.gnu.org
> 
> I tested it against the November 6th build.
> 
> It seems to work in the sense that now instead of crashing emacs,
> gnutls just throws me into the emacs debugger:
> 
> Debugger entered--Lisp error: (gnutls-error #<process tls> -64)
>   signal(gnutls-error (#<process tls> -64))
>   gnutls-negotiate(:process #<process tls> :type gnutls-x509pki
>   :hostname "imap.gmail.com")
>   open-gnutls-stream("tls" "tls-buffer" "imap.gmail.com" "imaps")
>   eval-region(19 83 t #[257 "\300\242b\210\301\207" [(83)
>   (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")] 2
>   "\n\n(fn IGNORE)"])  ; Reading at buffer position 83
>   eval-defun-2()
>   eval-defun(nil)
>   call-interactively(eval-defun nil nil)
>   command-execute(eval-defun)

Thank you.  So I consider this bug resolved, and I'm closing it.

> Presumably, this error is due to the fact that
> "/usr/ssl/certs/ca-bundle.crt" is not a valid file on Cygwin (despite
> the comment in gnutls.el implying that it is!)

No, this is because "/usr/ssl/certs/ca-bundle.crt" _is_ a valid Cygwin
file name, but you are not running a Cygwin build of Emacs, where it
would be valid.

> Of course, it all works fine with the elisp patch that I had
> previously submitted.

You are free to use that patch, if it suits your needs.  This is Free
Software.

> So, now that the bug that crashes the emacs process has been fixed,
> perhaps it's time to make gnutls-trustfiles actually work under
> cygwin.

They already work under Cygwin.  You probably mean cygwin-mount.el,
which is an incomplete and imperfect way of sneaking Cygwin file names
into a native Windows build of Emacs and pretending they work.

> So, there remains a fundamental incompatibility between cygwin-mount
> and gnutls.el.

Actually, there's a fundamental incompatibility between cygwin-mount
and a native build of Emacs.  Just say no.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 22:06:14 +0200 on Monday, November 11, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Mon, 11 Nov 2013 14:12:34 -0500
 > > Cc: Ted Zlatanov <tzz@lifelogs.com>, emacs@kosowsky.org, 15648@debbugs.gnu.org
 > > 
 > > Well, there is also the problem that "/usr" is never present as a root
 > > path on any (standard) Windows machine, so that the path commented as
 > > being valid for cygwin actually never works! 
 > 
 > Please don't assume that what happens on your machine happens on
 > everyone else's.  E.g., my systems do have a /usr directory at least
 > on some of the drives.  No, I don't run Cygwin.

OK... for the pedantic, I will explain what I mean by "standard
Windows machine"... on probably about 99.999% of Windows machines,
there is no /usr lying at the root of the C-drive. I am probably being
generous given that (1) so few Windows machines have a Unix-like tree
anywhere (2) Of those that do and also have Emacs, most are probably
are using cygwin which doesn't put the root at /

 > More generally, "/foo/bar" is a valid file name on Windows, whether
 > foo equals "usr" or not.  It is simply wrong to assume that if you see
 > /usr on Windows, it _must_ mean a Cygwin mount.  This kind of
 > reasoning is simply a non-starter.

I *never* ever claimed that. Why would I claim that? Creating a
(false) straw man and calling it a "non-starter" is not very
productive.

I merely claim that if one is using a *valid* magic file handler like
cygwin-mount, then gnutls should not crash by virtue of having a magic
file handler installed.

 > > The absence of cygwin-mount magic file handling when a file name is
 > > passed directly to the gnutls c-code without going through any of the
 > > standard magic-file-handling file access routines is the crux of the
 > > problem.

 > No, the crux of the problem is that you are trying to use a natively
 > built Emacs with file names that make sense only for Cygwin programs.

No, the crux of the problem was and is a bug in the gnutls.el code,
due to the inconsistency in which magic files are handled.
1. file-exists-p is enabled with magic file handling turned on
2. The pass to c-code ignores magic file handling
3. This causes gnutls.el to crash with a totally incomprehensible
   error message and an equally incomprehensible return code of '-64'

Gnutls needs to treat magic files consistently... even if it simply
chooses to ignore magic file handling (in a consistent way), then
such behavior should ideally be documented in the code.

 > If you want Cygwin semantics of file names, use a Cygwin build of
 > Emacs, and this problem will immediately go away.  Alternatively, use
 > file names for your certificates that native Windows programs can
 > find, and the problem will also go away immediately.

That is your opinion. Your suggestions that I limit myself to using a
cygwin-compiled version of emacs are just a workaround for poor
coding. I'm sorry.

Others have written cygwin-mount specifically for the use case that I
have. Since when are the two choices you list, the only ones allowed?
This whole idea of doing it just your way (or any way) goes against
the entire emacs philosophy.

Cygwin-mount is a perfectly valid elisp library that does everything a
magic file handler is supposed to do. Rather, the problem lies solely
with the gnutls code. If the authors choose to shut off magic file
handling in gnutls consistently, then I would be disappointed in the
limitations and unnecessary inflexibility of the gnutls code, though I
couldn't argue that it was buggy.

Magic file handling is a core emacs feature. You have to take such
functionality into account whether you choose to leverage it or
not... or else you are contributing to making emacs buggy and
inconsistent (along with limiting its power)

 > 
 > > So, again I see only 2 solutions:
 > > 1. Change (or omit) the "/usr" path and make it relative to cygwin
 > >    root (though this would not work generally since cygwin root is
 > >    changeable)
 > > 
 > > 2. Implement magic handling so that paths are automagically translated
 > >    to be correct at the file system level. In this case, by inserting
 > >    the cygwin root.
 > 
 >   3. Don't mix Cygwin file names with native Windows programs.

#3? Where did you come up with that restriction? Is it documented
somewhere in the Emacs/Elisp manuals and/or in the release notes? Or
did you invent that restriction out of thin air?

I think you are missing the point that gnutls.el has an inherent
inconsistency independent of cygwin or anything else...

Specifically, it leaves magic file handling on when using file-exists-p
to check whether the file exists but then ignores magic file handling
when passing the file to the OS.

The net result is that any path in gnutls-trustfiles that is valid to
a magic file handler but invalid to the OS will cause gnutls to crash
every time without any user-readable explanation. Even worse,
gnutls.el contains a default path labeled "cygwin" that will only work 
with a specially compiled cygwin version of emacs.

This is sloppy coding... no excuse for it... nothing to do with cygwin
or cygwin-mount except that cygwin mount happened to surface the
problem. Even worse, your obstinate refusal to fix the issue is simply
unbelievable.

I have proposed a valid, workable, and very tiny patch. Moreover, my
patch is totally consistent with the usage of expand-file-name in
particular and magic file handlers in particular as used dozens of
times in file.el

You have failed to give a single use case where my patch would cause
any problems. My patch adds immeasurably trivial complexity and
processing time to the routine. In contrast, all you have done is
suggest one-off workarounds which do nothing to solve the problem for
the next unsuspecting user...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 22:11:19 +0200 on Monday, November 11, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Mon, 11 Nov 2013 14:40:29 -0500
 > > Cc: Alp Aker <alptekin.aker@gmail.com>, emacs@kosowsky.org,
 > > 	15648@debbugs.gnu.org
 > > 
 > > I tested it against the November 6th build.
 > > 
 > > It seems to work in the sense that now instead of crashing emacs,
 > > gnutls just throws me into the emacs debugger:
 > > 
 > > Debugger entered--Lisp error: (gnutls-error #<process tls> -64)
 > >   signal(gnutls-error (#<process tls> -64))
 > >   gnutls-negotiate(:process #<process tls> :type gnutls-x509pki
 > >   :hostname "imap.gmail.com")
 > >   open-gnutls-stream("tls" "tls-buffer" "imap.gmail.com" "imaps")
 > >   eval-region(19 83 t #[257 "\300\242b\210\301\207" [(83)
 > >   (open-gnutls-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")] 2
 > >   "\n\n(fn IGNORE)"])  ; Reading at buffer position 83
 > >   eval-defun-2()
 > >   eval-defun(nil)
 > >   call-interactively(eval-defun nil nil)
 > >   command-execute(eval-defun)
 > 
 > Thank you.  So I consider this bug resolved, and I'm closing it.

IT IS NOT RESOLVED - all you did was remove the problem from being one
of emacs crashing to the slightly less severe instance of gnutls
itself crashing!

 > > Presumably, this error is due to the fact that
 > > "/usr/ssl/certs/ca-bundle.crt" is not a valid file on Cygwin (despite
 > > the comment in gnutls.el implying that it is!)
 > 

 > No, this is because "/usr/ssl/certs/ca-bundle.crt" _is_ a valid Cygwin
 > file name, but you are not running a Cygwin build of Emacs, where it
 > would be valid.

Why should I be required to only run a Cygwin build of Emacs? Is that
documented anywhere?

 > 
 > > Of course, it all works fine with the elisp patch that I had
 > > previously submitted.
 > 
 > You are free to use that patch, if it suits your needs.  This is Free
 > Software.

And you are free to be obstinate and not admit your mistakes...

 > 
 > > So, now that the bug that crashes the emacs process has been fixed,
 > > perhaps it's time to make gnutls-trustfiles actually work under
 > > cygwin.
 > 
 > They already work under Cygwin.  You probably mean cygwin-mount.el,
 > which is an incomplete and imperfect way of sneaking Cygwin file names
 > into a native Windows build of Emacs and pretending they work.
 > 
 > > So, there remains a fundamental incompatibility between cygwin-mount
 > > and gnutls.el.
 > 
 > Actually, there's a fundamental incompatibility between cygwin-mount
 > and a native build of Emacs.  Just say no.

Cygwin-mount works perfectly for me... it always has...

Plus, the problem is more general than cygwin-mount. The problem
potentially lies with any magic file handler.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Ted Zlatanov]

On Mon, 11 Nov 2013 21:00:56 +0100 Achim Gratz <Stromeko@nexgo.de> wrote: 

AG> Ted Zlatanov writes:
>> Could we just call http://cygwin-lite.sourceforge.net/html/cygpath.html
>> on the path (or the equivalent C function)?  It seems to be guaranteed
>> to work but may be very slow with the execution overhead.

AG> Please refer to the official Cygwin documentation, not some unmaintained
AG> project that stopped getting updates over a decade ago:

AG> http://cygwin.com/cygwin-ug-net/using-utils.html

Thanks.  I went with Google, incorrectly.

AG> Specifically for Emacs, the "-m" option to cygpath seems the most
AG> suitable.  Please do not link to the Cygwin DLL from Windows Emacs
AG> (i.e. "using the equivalent C function").  The only reason to use a
AG> native Emacs on Windows (for me anyway) is that I can update Cygwin
AG> while I keep Emacs running and that wouldn't be possible if the DLL was
AG> locked because it was still in use.  For anything else (if you don#t
AG> want to start an X server), there's emacs-w32 from Cygwin.

OK, that sounds like a reasonable use case.

On Mon, 11 Nov 2013 15:00:39 -0500 <emacs@kosowsky.org> wrote: 

> Ted Zlatanov wrote at about 14:42:46 -0500 on Monday, November 11, 2013:
>> Could we just call [cygpath]?

> It would work for cygwin... but I see two limitations:

> 1. The same problem that affects cygwin-mount, would affect any other
>    magic file handler that translates file names, since once again
>    file-exists-p would return true, while the path passed to the
>    c-code would in general not point to an OS-recognizable file.

Yes, right.  There would have to be some logic somewhere in Emacs to
recognize magic Cygwin pathnames and treat them specially from a native
non-Cygwin W32 build.  Then packages like gnutls.el that talk to system
libraries can use that logic.  I don't think it can be accomplished
otherwise.

> 2. The suggestion seems to be far kluggier than my suggested patch
>    since it relies on a user elisp routine passing executing a system
>    call to return a path every time a gnutls connection is
>    requested. At least cygwin-mount does this only once at startup and
>    abstracts the translation away from higher level routines -- which
>    is the entire purpose of a magic file handler.

>    If cygpath belongs anywhere, it would be in the magic file handling
>    code. Though cygwin-mount does this in a "smarter" way by calling
>    'mount' once to determine the cygwin prefix once-and-for-all, and
>    then whenever a path is passed, the magic handler automatically
>    prefixes on the cygwin mount prefix.

> Once again, the problem is not cygwin-mount, per-se. The problem is
> the inconsistency between using the magic file enabled predicate
> file-exists-p to test for a file and then passing that same file name
> to c-code that knows nothing about magic file handlers.

I honestly don't have a preference for cygpath vs. cygwin-mount but
would like to continue the discussion in a new ticket please.  Could we
start a new ticket specifically for the generic problem we've found with
native Emacs W32 builds vs. Cygwin?

I should add here that I'm grateful for all the help and suggestions I
got from you, Eli, and the other participants.  The bug in the GnuTLS
interface code affected potentially all users, not just this one case.

Thanks
Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Ted Zlatanov wrote at about 18:58:12 -0500 on Monday, November 11, 2013:
 > On Mon, 11 Nov 2013 21:00:56 +0100 Achim Gratz <Stromeko@nexgo.de> wrote: 
 > AG> Specifically for Emacs, the "-m" option to cygpath seems the most
 > AG> suitable.  Please do not link to the Cygwin DLL from Windows Emacs
 > AG> (i.e. "using the equivalent C function").  The only reason to use a
 > AG> native Emacs on Windows (for me anyway) is that I can update Cygwin
 > AG> while I keep Emacs running and that wouldn't be possible if the DLL was
 > AG> locked because it was still in use.  For anything else (if you don#t
 > AG> want to start an X server), there's emacs-w32 from Cygwin.
 > 
 > OK, that sounds like a reasonable use case.

That is very similar to my use case...
Similarly, I tend to keep my Emacs windows open until I reboot. On the
other hand, I not infrequently need to close all cygwin processes
(including servers) either to upgrade (as AG notes) or to troubleshoot
some cygwin issue since so long as the cygwin.dll is bound to one
process, I can't completely restart the state of cygwin.

PLUS on some computers, I might have emacs only, on others cygwin
only, on others multiple version of cygwin (I used to have both Cygwin
1.5 and 1.7), and on others all combinations.

Since both cygwin and emacs are workhorses for me, I actually prefer
to not have them bound by dependencies whereby one requires the
installation of the other to work... Nor do I want my emacs updates
tied to the cygwin project (for example, I am currently using cygwin
X_64 but many packages have not yet been ported there. I would hate to
have to wait for an important package like Emacs to be ported every
time cygwin issues a new branch)

Indeed in my emacs-loving-centric world, emacs is more fundamental in
some ways than cygwin. If cygwin ceased to exist for windows, I would
still use Emacs on Windows. So, I like the fact that I can make Emacs adapt
to cygwin via a library (cygwin-mount) rather than requiring a
separate cygwin compile. Introducing a forced dependency to make
things work just seems so unemacs-like.

 > On Mon, 11 Nov 2013 15:00:39 -0500 <emacs@kosowsky.org> wrote: 
 > > Ted Zlatanov wrote at about 14:42:46 -0500 on Monday, November 11, 2013:
 > >> Could we just call [cygpath]?
 > 
 > > It would work for cygwin... but I see two limitations:
 > 
 > > 1. The same problem that affects cygwin-mount, would affect any other
 > >    magic file handler that translates file names, since once again
 > >    file-exists-p would return true, while the path passed to the
 > >    c-code would in general not point to an OS-recognizable file.
 > 
 > Yes, right.  There would have to be some logic somewhere in Emacs to
 > recognize magic Cygwin pathnames and treat them specially from a native
 > non-Cygwin W32 build.  Then packages like gnutls.el that talk to system
 > libraries can use that logic.  I don't think it can be accomplished
 > otherwise.

One of the reasons I prefer cygwin-mount to a cygwin-specific build is
that I can see, understand, and potentially modify the treatment of
such pathnames straight in the elisp code rather than having it hidden
in OS-specific C-code. And if I want to change the behavior I can
patch it using standard elisp functions without having to patch the
c-code and recompile.

In my understanding of emacs, as much as possible should be done in
elisp. C-code is best reserved only for direct system interface calls
and/or for calls where interpreted (byte-compiled) elisp just runs too
slow.

 > I honestly don't have a preference for cygpath vs. cygwin-mount but
 > would like to continue the discussion in a new ticket please.  Could we
 > start a new ticket specifically for the generic problem we've found with
 > native Emacs W32 builds vs. Cygwin?

Sure... though how broad vs. narrow would you like that thread to be?

Again, in my world, emacs is fundamental -- I use it across
platforms by choice. Cygwin to me is just a tool I use to make Windows
more like Linux... but it's not something I would use by choice nor is
it something I use across platforms.

So, while I am totally supportive of cygwin compiling their own
version of Emacs, I think that Emacs should also maintain a library
for supporting cygwin in its standalone W32 version -- just like Emacs
has modes for supporting just about anything -- just like if I were on
an old mac, I would want Emacs to have modes for supporting the
original mac os file system & hierarchy.




 > I should add here that I'm grateful for all the help and suggestions I
 > got from you, Eli, and the other participants.  The bug in the GnuTLS
 > interface code affected potentially all users, not just this one case.
 > 
 > Thanks
 > Ted

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Mon, 11 Nov 2013 16:53:31 -0500
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
> 
>  > > The absence of cygwin-mount magic file handling when a file name is
>  > > passed directly to the gnutls c-code without going through any of the
>  > > standard magic-file-handling file access routines is the crux of the
>  > > problem.
> 
>  > No, the crux of the problem is that you are trying to use a natively
>  > built Emacs with file names that make sense only for Cygwin programs.
> 
> No, the crux of the problem was and is a bug in the gnutls.el code,
> due to the inconsistency in which magic files are handled.
> 1. file-exists-p is enabled with magic file handling turned on

Only because you use cygwin-mount.

> 2. The pass to c-code ignores magic file handling

Only because you use cygwin-mount.

> 3. This causes gnutls.el to crash with a totally incomprehensible
>    error message and an equally incomprehensible return code of '-64'

It no longer crashes.

> Gnutls needs to treat magic files consistently... even if it simply
> chooses to ignore magic file handling (in a consistent way), then
> such behavior should ideally be documented in the code.

Remove cygwin-mount, and that's what will happen.

>  > If you want Cygwin semantics of file names, use a Cygwin build of
>  > Emacs, and this problem will immediately go away.  Alternatively, use
>  > file names for your certificates that native Windows programs can
>  > find, and the problem will also go away immediately.
> 
> That is your opinion. Your suggestions that I limit myself to using a
> cygwin-compiled version of emacs are just a workaround for poor
> coding. I'm sorry.

It's not an issue with code quality.  It's simply a consequence of the
fact that Cygwin file names _cannot_ be consistently supported in
native Windows programs.  You can solve perhaps 95% or 99% of that,
but not 100%.  If you are willing to live with those limitations,
please be free.  But it will never be supported all the way, it simply
cannot be.

> Others have written cygwin-mount specifically for the use case that I
> have. Since when are the two choices you list, the only ones allowed?
> This whole idea of doing it just your way (or any way) goes against
> the entire emacs philosophy.

You are entitled to your opinions.  But I respectfully disagree, and
as long as I'm hold responsible for the Windows port of Emacs, I'm
sorry, but my opinions weigh slightly more.

> Magic file handling is a core emacs feature. You have to take such
> functionality into account whether you choose to leverage it or
> not... or else you are contributing to making emacs buggy and
> inconsistent (along with limiting its power)

Except that Cygwin "magic" cannot be reliably handled that way.  The
difference is that for other magic file names, we have a handling
agent that is outside of Emacs.  By contrast, cygwin-mount pretends to
do that entirely in Emacs Lisp, which cannot work reliably enough.  As
this example shows.

>  >   3. Don't mix Cygwin file names with native Windows programs.
> 
> #3? Where did you come up with that restriction? Is it documented
> somewhere in the Emacs/Elisp manuals and/or in the release notes? Or
> did you invent that restriction out of thin air?

It's not an invention.  It's the result of many years using Emacs on
Windows and hacking it.  You are free to ignore that experience, of
course, but I assure you it is valid.

> I think you are missing the point that gnutls.el has an inherent
> inconsistency independent of cygwin or anything else...

It's not just gnutls.el, it's all of Emacs.  And for a good reason:
these file names cannot be supported by cygwin-mount or any similar
solutions.  You need an external agent that resolves all the Cygwin
file names, any time Emacs needs to access a file, be it in Lisp or in
C.

> I have proposed a valid, workable, and very tiny patch. Moreover, my
> patch is totally consistent with the usage of expand-file-name in
> particular and magic file handlers in particular as used dozens of
> times in file.el

As long as the solution is plugging cygwin-mount into more places in
Emacs, such a solution will not be acceptable, sorry.

> You have failed to give a single use case where my patch would cause
> any problems. My patch adds immeasurably trivial complexity and
> processing time to the routine. In contrast, all you have done is
> suggest one-off workarounds which do nothing to solve the problem for
> the next unsuspecting user...

I have enough use cases to fill a lecture, I just have no time to
write them all.  Sorry, please accept my judgment on this, even if you
disagree.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Mon, 11 Nov 2013 16:56:41 -0500
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, alptekin.aker@gmail.com,
>  15648-done@debbugs.gnu.org
> 
>  > Thank you.  So I consider this bug resolved, and I'm closing it.
> 
> IT IS NOT RESOLVED - all you did was remove the problem from being one
> of emacs crashing to the slightly less severe instance of gnutls
> itself crashing!

The crash is the only problem I considered in this bug report that we
must resolve.

>  > No, this is because "/usr/ssl/certs/ca-bundle.crt" _is_ a valid Cygwin
>  > file name, but you are not running a Cygwin build of Emacs, where it
>  > would be valid.
> 
> Why should I be required to only run a Cygwin build of Emacs? Is that
> documented anywhere?

If you want to use Cygwin features, you should use a Cygwin build of
Emacs, yes.

>  > Actually, there's a fundamental incompatibility between cygwin-mount
>  > and a native build of Emacs.  Just say no.
> 
> Cygwin-mount works perfectly for me... it always has...
> 
> Plus, the problem is more general than cygwin-mount. The problem
> potentially lies with any magic file handler.

No, the problem is that cygwin-mount is an incomplete solution for
handling such file names.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Stefan -- please pay attention to the hostile, unfriendly attitude and
ultimate arrogance of the Windows Emacs maintainer...

Eli Zaretskii wrote at about 05:56:06 +0200 on Tuesday, November 12, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Mon, 11 Nov 2013 16:53:31 -0500
 > > Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org
 > > 
 > >  > > The absence of cygwin-mount magic file handling when a file name is
 > >  > > passed directly to the gnutls c-code without going through any of the
 > >  > > standard magic-file-handling file access routines is the crux of the
 > >  > > problem.
 > > 
 > >  > No, the crux of the problem is that you are trying to use a natively
 > >  > built Emacs with file names that make sense only for Cygwin programs.
 > > 
 > > No, the crux of the problem was and is a bug in the gnutls.el code,
 > > due to the inconsistency in which magic files are handled.
 > > 1. file-exists-p is enabled with magic file handling turned on
 > 
 > Only because you use cygwin-mount.

Or any other magic file handler...
where file-exists-p changes the file path from something not OS
recognizable to something OS recognizable.

The same error would occur with ange-ftp and tramp
should you use that style filename in gnutls-trustfiles.

Unless you want to document gnutls as disallowing magic file handler
(and write the code to disable file handlers), then you have a
problem. Otherwise, you are knowingly and intentionally writing code
that is incompatible with a core emacs feature set.

 > 
 > > 2. The pass to c-code ignores magic file handling
 > 
 > Only because you use cygwin-mount.

Or any other magic file handler...

 > 
 > > 3. This causes gnutls.el to crash with a totally incomprehensible
 > >    error message and an equally incomprehensible return code of '-64'
 > 
 > It no longer crashes.

It doesn't crash emacs... but the routine exist on error without
human-readable explanation (other than an obscure -64 return value).
Is this code? Is this user friendly?

 > > chooses to ignore magic file handling (in a consistent way), then
 > > such behavior should ideally be documented in the code.
 > 
 > Remove cygwin-mount, and that's what will happen.

Remove (almost) all file handlers actually... so let's just make magic
file handlers illegal or remove them from Emacs....

 > 
 > >  > If you want Cygwin semantics of file names, use a Cygwin build of
 > >  > Emacs, and this problem will immediately go away.  Alternatively, use
 > >  > file names for your certificates that native Windows programs can
 > >  > find, and the problem will also go away immediately.
 > > 
 > > That is your opinion. Your suggestions that I limit myself to using a
 > > cygwin-compiled version of emacs are just a workaround for poor
 > > coding. I'm sorry.
 > 
 > It's not an issue with code quality.  It's simply a consequence of the
 > fact that Cygwin file names _cannot_ be consistently supported in
 > native Windows programs.  You can solve perhaps 95% or 99% of that,
 > but not 100%.  If you are willing to live with those limitations,
 > please be free.  But it will never be supported all the way, it simply
 > cannot be.

The perfect is the enemy of the good enough.
I have offered a patch that substantially improves compatibility
without any downside other than your own unsubstantiated biases.

 > > Others have written cygwin-mount specifically for the use case that I
 > > have. Since when are the two choices you list, the only ones allowed?
 > > This whole idea of doing it just your way (or any way) goes against
 > > the entire emacs philosophy.
 > 
 > You are entitled to your opinions.  But I respectfully disagree, and
 > as long as I'm hold responsible for the Windows port of Emacs, I'm
 > sorry, but my opinions weigh slightly more.

You are being arrogant. Emacs is about more people than just yourself
and about more use cases than just you. Such an attitude is
incongruous with an open source project and discourages both users and
participants.

If such an attitude persists, I will regret helping with debugging
this and taking my precious time to test your c-code fix. I had
already fixed the problem myself, so with your attitude why would I
share my insights with you.

I am copying Stefan Monnier here to see if he agrees with your
arrogant attitude.

 > > Magic file handling is a core emacs feature. You have to take such
 > > functionality into account whether you choose to leverage it or
 > > not... or else you are contributing to making emacs buggy and
 > > inconsistent (along with limiting its power)
 > 
 > Except that Cygwin "magic" cannot be reliably handled that way.  The
 > difference is that for other magic file names, we have a handling
 > agent that is outside of Emacs.  By contrast, cygwin-mount pretends to
 > do that entirely in Emacs Lisp, which cannot work reliably enough.  As
 > this example shows.

It will happen with almost any file handler that uses file-exists-p.

 > >  >   3. Don't mix Cygwin file names with native Windows programs.
 > > 
 > > #3? Where did you come up with that restriction? Is it documented
 > > somewhere in the Emacs/Elisp manuals and/or in the release notes? Or
 > > did you invent that restriction out of thin air?
 > 
 > It's not an invention.  It's the result of many years using Emacs on
 > Windows and hacking it.  You are free to ignore that experience, of
 > course, but I assure you it is valid.

But why would you preclude this from working.
Please give me ONE use case where my patch either causes a failure
elsewhere or where it bogs down the code.
 > 
 > > I think you are missing the point that gnutls.el has an inherent
 > > inconsistency independent of cygwin or anything else...
 > 
 > It's not just gnutls.el, it's all of Emacs.  And for a good reason:
 > these file names cannot be supported by cygwin-mount or any similar
 > solutions.  You need an external agent that resolves all the Cygwin
 > file names, any time Emacs needs to access a file, be it in Lisp or in
 > C.

That is your opinion. I have been using cygwin-mount for about a
decade without any problems... until now.

 > 
 > > I have proposed a valid, workable, and very tiny patch. Moreover, my
 > > patch is totally consistent with the usage of expand-file-name in
 > > particular and magic file handlers in particular as used dozens of
 > > times in file.el
 > 
 > As long as the solution is plugging cygwin-mount into more places in
 > Emacs, such a solution will not be acceptable, sorry.

You are biased against cygwin-mount. A maintainer needs to check his
biases at the door.

 > 
 > > You have failed to give a single use case where my patch would cause
 > > any problems. My patch adds immeasurably trivial complexity and
 > > processing time to the routine. In contrast, all you have done is
 > > suggest one-off workarounds which do nothing to solve the problem for
 > > the next unsuspecting user...
 > 
 > I have enough use cases to fill a lecture, I just have no time to
 > write them all.  Sorry, please accept my judgment on this, even if you
 > disagree.

I am just asking for one (reasonable) failure case.

I call BS. You are clearly just posturing -- given that you have taken
hours to refute everything but the patch itself.

If you had a use case at your fingertips, surely you would happily
have suggested it long ago...

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[emacs]

Eli Zaretskii wrote at about 05:58:21 +0200 on Tuesday, November 12, 2013:
 > > From: <emacs@kosowsky.org>
 > > Date: Mon, 11 Nov 2013 16:56:41 -0500
 > > Cc: emacs@kosowsky.org, tzz@lifelogs.com, alptekin.aker@gmail.com,
 > >  15648-done@debbugs.gnu.org
 > > 
 > >  > Thank you.  So I consider this bug resolved, and I'm closing it.
 > > 
 > > IT IS NOT RESOLVED - all you did was remove the problem from being one
 > > of emacs crashing to the slightly less severe instance of gnutls
 > > itself crashing!
 > 
 > The crash is the only problem I considered in this bug report that we
 > must resolve.

Well, I regret helping then. I was specifically led to believe that if
I helped resolve the emacs crash, then you would consider addressing
the original bug report. 

If you are going to be selfish and only address your personal issues,
then I will *never* again contribute to this effort. I am more than
competent to write all the elisp code I need for my own support.

 > >  > No, this is because "/usr/ssl/certs/ca-bundle.crt" _is_ a valid Cygwin
 > >  > file name, but you are not running a Cygwin build of Emacs, where it
 > >  > would be valid.
 > > 
 > > Why should I be required to only run a Cygwin build of Emacs? Is that
 > > documented anywhere?
 > 
 > If you want to use Cygwin features, you should use a Cygwin build of
 > Emacs, yes.

That is your choice. Others have other needs and other preferences.
You need to stop being so arrogant and narrow-minded.

 > 
 > >  > Actually, there's a fundamental incompatibility between cygwin-mount
 > >  > and a native build of Emacs.  Just say no.
 > > 
 > > Cygwin-mount works perfectly for me... it always has...
 > > 
 > > Plus, the problem is more general than cygwin-mount. The problem
 > > potentially lies with any magic file handler.
 > 
 > No, the problem is that cygwin-mount is an incomplete solution for
 > handling such file names.

The problem lies with (almost) any magic file handler. Gnutls.el code
is broken. You are either intentionally or unintentionally being
pig-headed and missing the broader point that the code breaks with
almost any valid magic file handler.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Eli Zaretskii]

> From: <emacs@kosowsky.org>
> Date: Tue, 12 Nov 2013 10:19:42 -0500
> Cc: emacs@kosowsky.org, tzz@lifelogs.com, 15648@debbugs.gnu.org,
>  Stefan Monnier <monnier@IRO.UMontreal.CA>
> 
> Stefan -- please pay attention to the hostile, unfriendly attitude and
> ultimate arrogance of the Windows Emacs maintainer...

There's no hostility here.  Disagreement, yes; but that's something
different.  As for the other vices, I'll leave it to others to judge
that.

>  > > No, the crux of the problem was and is a bug in the gnutls.el code,
>  > > due to the inconsistency in which magic files are handled.
>  > > 1. file-exists-p is enabled with magic file handling turned on
>  > 
>  > Only because you use cygwin-mount.
> 
> Or any other magic file handler...
> where file-exists-p changes the file path from something not OS
> recognizable to something OS recognizable.

No, cygwin-mount is fundamentally different from most (all?) other
file-handlers.  IMO, it is also a far cry from what the current design
of file-handlers in Emacs assumes.

Let me explain.

First, "magic file names" are IMO really a euphemism.  A much more
accurate term is "remote" or "foreign" file names.  At least this is
the only interpretation I can think of under which the current design
of file-handlers makes sense.  Specifically, Emacs does not want to
know anything about handling such files -- it just calls the handler
and returns the result, without any additional processing.  Emacs does
that consistently in many primitives, and therefore it assumes that a
handler for a class of file names will implement _all_ of the handlers
for those operations that make sense with that class of files.

By contrast, cygwin-mount implements handlers for a very small set (I
think 3 or 4) of file-related operations, and for the rest it turns
around and calls Emacs's original code to do the job.  But Emacs's
internal processing of files assumes without checking that _any_ file
not handled by some handler is a local file that fully abides by the
conventions of the local filesystem.  In particular, the
Windows-specific code in Emacs is _riddled_ with such assumptions.  It
will prepend a drive letter to file names that lack one, convert
forward slashes to backslashes and back at will, downcase file names
without thinking twice, even convert them to 8+3 alias form in some
cases.  Put a "magic" or "semi-magic" file name through that code, and
you cannot trust the result.

Here's a simple example: half way through expand-file-name, Emacs
accesses the home directory of a user, to support "~" in file names.
This is done entirely in C, so if the home directory is in Cygwin
/foo/bar format, the result will be a failure to resolve "~".

Another example is with file-attributes: if someone expects to see
Cygwin-style emulation of Posix mode bits that resembles what Cygwin's
'ls -l' will show, they will be deeply disappointed at best.

Etc., etc. -- there are a lot of use cases where cygwin-mount in its
present shape simply isn't complete enough to pretend to be a
full-fledged file-handler agent that plays by the rules that Emacs
expects.  We can add a little Lisp here and a little more there to fix
some use case or another, but that just obscures and obfuscates the
code, and in a few months or years no one knows why we have this or
that piece of code with some strange effect.

Now, it might be possible to develop cygwin-mount to the degree that
it does work reliably (which will most probably mean either
introduction of much more hooks into Emacs to allow that, or
reimplementation of many of the C portions in Lisp).  If someone
submits patches to do that, it would probably be a welcome addition to
Emacs.  But we are not there yet.

> The same error would occur with ange-ftp and tramp
> should you use that style filename in gnutls-trustfiles.

gnutls.el doesn't make sense with remote file names, I agree.  Your
suggestion to bind the handlers alist to nil is IMO TRT for gnutls (as
well as for many other Emacs packages).

>  > > 3. This causes gnutls.el to crash with a totally incomprehensible
>  > >    error message and an equally incomprehensible return code of '-64'
>  > 
>  > It no longer crashes.
> 
> It doesn't crash emacs... but the routine exist on error without
> human-readable explanation (other than an obscure -64 return value).
> Is this code? Is this user friendly?

Probably not; patches to make that error message more legible are
welcome.

But please don't underestimate the value of the solution that
eliminated the crash: not only Emacs will not crash anymore in that
case, something that Emacs should never do due to bugs on the Lisp
level, but it also corrected a fundamental flaw in the code, whereby a
Lisp object was marked as being a GuTLS connection too early, before
it was completely initialized.  So I think this bug report served
Emacs very well indeed, and thank you for helping us resolve it.

>  > It's not an issue with code quality.  It's simply a consequence of the
>  > fact that Cygwin file names _cannot_ be consistently supported in
>  > native Windows programs.  You can solve perhaps 95% or 99% of that,
>  > but not 100%.  If you are willing to live with those limitations,
>  > please be free.  But it will never be supported all the way, it simply
>  > cannot be.
> 
> The perfect is the enemy of the good enough.

This goes both ways, you know.

> I have offered a patch that substantially improves compatibility
> without any downside other than your own unsubstantiated biases.

The downside you don't see is the additional code it adds to Emacs
that has no other reason but to support cygwin-mount.  There's already
more than enough code in Emacs which no one fully understands, because
it was introduced to "fix" this or that corner case on Windows.  With
almost all the active maintainers coming from the Posix world, I fear
that leaving all this stuff, let alone adding to it, is a slippery
slope to an abyss.

>  > You are entitled to your opinions.  But I respectfully disagree, and
>  > as long as I'm hold responsible for the Windows port of Emacs, I'm
>  > sorry, but my opinions weigh slightly more.
> 
> You are being arrogant. Emacs is about more people than just yourself
> and about more use cases than just you. Such an attitude is
> incongruous with an open source project and discourages both users and
> participants.
> 
> If such an attitude persists, I will regret helping with debugging
> this and taking my precious time to test your c-code fix. I had
> already fixed the problem myself, so with your attitude why would I
> share my insights with you.

I'm sorry that you feel that way.  I tried to explain above why your
help was very important and appreciated beyond the narrow context of
the crash.

>  > Except that Cygwin "magic" cannot be reliably handled that way.  The
>  > difference is that for other magic file names, we have a handling
>  > agent that is outside of Emacs.  By contrast, cygwin-mount pretends to
>  > do that entirely in Emacs Lisp, which cannot work reliably enough.  As
>  > this example shows.
> 
> It will happen with almost any file handler that uses file-exists-p.

I agree that gnutls.el should ignore the file handlers when it tests
for existence of certificate files.  Clearly, having those files on
remote systems makes no sense.

That is not where I was disagreeing with you.  If that is what I
somehow made you understand, I'm sorry for my imperfect wording.

>  > >  >   3. Don't mix Cygwin file names with native Windows programs.
>  > > 
>  > > #3? Where did you come up with that restriction? Is it documented
>  > > somewhere in the Emacs/Elisp manuals and/or in the release notes? Or
>  > > did you invent that restriction out of thin air?
>  > 
>  > It's not an invention.  It's the result of many years using Emacs on
>  > Windows and hacking it.  You are free to ignore that experience, of
>  > course, but I assure you it is valid.
> 
> But why would you preclude this from working.
> Please give me ONE use case where my patch either causes a failure
> elsewhere or where it bogs down the code.

I'm not sure we are talking about the same thing.  Again, I don't
disagree with the change in gnutls.el, to ignore file handlers.

What I was talking about was adding code that is only needed to
support cygwin-mount, like calling expand-file-name, or changing
convert-standard-filename for the same reason.

>  > As long as the solution is plugging cygwin-mount into more places in
>  > Emacs, such a solution will not be acceptable, sorry.
> 
> You are biased against cygwin-mount.

Yes, I am.  I tried to explain above why.  I hope now you understand
my reasons better.

bug#15648: 24.2.50; gnutls SSL connection to IMAP server causes emacs to crash completely

[Drew Adams]

> Now, it might be possible to develop cygwin-mount to the degree that
> it does work reliably (which will most probably mean either
> introduction of much more hooks into Emacs to allow that, or
> reimplementation of many of the C portions in Lisp).  If someone
> submits patches to do that, it would probably be a welcome addition
> to Emacs.  But we are not there yet.


--> enhancement request #15878