bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Carsten Bormann]

(ns-convert-utf8-nfd-to-nfc "\377")

crashes in strlen (called from build_string), 
as it is run on a null-pointer (apparently returned from [utfStr UTF8String]).

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
0x00007fff94ac3670 in strlen ()
(gdb) where
#0  0x00007fff94ac3670 in strlen ()
#1  0x000000010009fbb4 in build_string ()
#2  0x000000010019300d in Fns_convert_utf8_nfd_to_nfc ()
...
0x00007fff94ac3670 <strlen+16>:	pcmpeqb (%rdi),%xmm0
...
(gdb) info reg
rdi            0x0	0

Further, there are no tests for this function.

(Bug manifests itself with tramp no longer working on hosts stuck in Latin-1-land, which might be due to an unrelated change.)

Grüße, Carsten


In GNU Emacs 24.3.50.1 (x86_64-apple-darwin, NS apple-appkit-1038.36)
 of 2013-10-06 on bob.porkrind.org
Bzr revision: 114537 dgutov@yandex.ru-20131006012151-wprswhei2ipqvr0c
Windowing system distributor `Apple', version 10.3.1187
Configured using:
 `configure --host=x86_64-apple-darwin --build=i686-apple-darwin
 --with-ns'

Important settings:
  value of $LC_CTYPE: UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<menu-bar> <help-menu> <send-emacs-bug-report>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
current-kill: Kill ring is empty

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel ns-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process ns multi-tty
emacs)

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Jan Djärv]

Hello.


9 okt 2013 kl. 13:37 skrev Carsten Bormann <cabo@tzi.org>:

> (ns-convert-utf8-nfd-to-nfc "\377")
> 
> crashes in strlen (called from build_string), 
> as it is run on a null-pointer (apparently returned from [utfStr UTF8String]).
> 
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000

Eh, don't do that?

> 0x00007fff94ac3670 in strlen ()
> (gdb) where
> #0  0x00007fff94ac3670 in strlen ()
> #1  0x000000010009fbb4 in build_string ()
> #2  0x000000010019300d in Fns_convert_utf8_nfd_to_nfc ()
> ...
> 0x00007fff94ac3670 <strlen+16>:	pcmpeqb (%rdi),%xmm0
> ...
> (gdb) info reg
> rdi            0x0	0
> 
> Further, there are no tests for this function.
> 
> (Bug manifests itself with tramp no longer working on hosts stuck in Latin-1-land, which might be due to an unrelated change.)
> 

The function clearly expects valid UTF-8 as input.  Why is tramp feeding it invalid UTF-8?    What is tramp trying to accomplish?  What would be the expected return value on invalid UTF-8?

	Jan D.

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Andreas Schwab]

Jan Djärv <jan.h.d@swipnet.se> writes:

> 9 okt 2013 kl. 13:37 skrev Carsten Bormann <cabo@tzi.org>:
>
>> (ns-convert-utf8-nfd-to-nfc "\377")
>> 
>> crashes in strlen (called from build_string), 
>> as it is run on a null-pointer (apparently returned from [utfStr UTF8String]).
>> 
>> Program received signal EXC_BAD_ACCESS, Could not access memory.
>> Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
>
> Eh, don't do that?

Lisp-level functions should always validate their arguments.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Jan Djärv]

Hello.

9 okt 2013 kl. 18:57 skrev Andreas Schwab <schwab@linux-m68k.org>:

> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> 9 okt 2013 kl. 13:37 skrev Carsten Bormann <cabo@tzi.org>:
>> 
>>> (ns-convert-utf8-nfd-to-nfc "\377")
>>> 
>>> crashes in strlen (called from build_string), 
>>> as it is run on a null-pointer (apparently returned from [utfStr UTF8String]).
>>> 
>>> Program received signal EXC_BAD_ACCESS, Could not access memory.
>>> Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
>> 
>> Eh, don't do that?
> 
> Lisp-level functions should always validate their arguments.

So it should throw an error?  I'm ok with that, but I suspect it doesn't help tramp.

	Jan D.

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Stefan Monnier]

> So it should throw an error?  I'm ok with that, but I suspect it
> doesn't help tramp.

If you signal an error, Tramp can catch it and do something with it, so
yes, it will definitely help.


        Stefan

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Carsten Bormann]

On Oct 9, 2013, at 18:31, Jan Djärv <jan.h.d@swipnet.se> wrote:

> The function clearly expects valid UTF-8 as input.  Why is tramp feeding it invalid UTF-8?    What is tramp trying to accomplish?  What would be the expected return value on invalid UTF-8?

I haven't looked at the details yet (that will be easier once the null pointer reference is fixed).

That needn't stop me from hypothesizing...

ns-convert-utf8-nfd-to-nfc is used in places where system output might contain Apple's slightly crazy not-quite-NFD file names, so that you can usefully cut and paste them etc. to places that expect the usual not-quite-NFC.  So one should expect a lot of not-really-UTF-8-after-all input to be fed into this thing.
I'm presuming tramp just feeds whatever it got from the remote system through this to get more useful output e.g. for a directory listing.

It probably would be useful to have a robust version of this that just chokes on nothing.
Raising an error on non-UTF-8 input may be a desirable behavior in other places.
(Crashing Emacs never is.)

I'm a bit surprised that this bug apparently was around for a number of years already...

Grüße, Carsten

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Jan Djärv]

Hello.

We now throw an error.

	Jan D.

9 okt 2013 kl. 20:33 skrev Carsten Bormann <cabo@tzi.org>:

> On Oct 9, 2013, at 18:31, Jan Djärv <jan.h.d@swipnet.se> wrote:
> 
>> The function clearly expects valid UTF-8 as input.  Why is tramp feeding it invalid UTF-8?    What is tramp trying to accomplish?  What would be the expected return value on invalid UTF-8?
> 
> I haven't looked at the details yet (that will be easier once the null pointer reference is fixed).
> 
> That needn't stop me from hypothesizing...
> 
> ns-convert-utf8-nfd-to-nfc is used in places where system output might contain Apple's slightly crazy not-quite-NFD file names, so that you can usefully cut and paste them etc. to places that expect the usual not-quite-NFC.  So one should expect a lot of not-really-UTF-8-after-all input to be fed into this thing.
> I'm presuming tramp just feeds whatever it got from the remote system through this to get more useful output e.g. for a directory listing.
> 
> It probably would be useful to have a robust version of this that just chokes on nothing.
> Raising an error on non-UTF-8 input may be a desirable behavior in other places.
> (Crashing Emacs never is.)
> 
> I'm a bit surprised that this bug apparently was around for a number of years already...
> 
> Grüße, Carsten

bug#15570: 24.3.50; Null pointer crash in (ns-convert-utf8-nfd-to-nfc "\377")

[Carsten Bormann]

On Oct 9, 2013, at 20:50, Jan Djärv <jan.h.d@swipnet.se> wrote:

> We now throw an error.

Thanks.

With that fix, the following workaround makes tramp serviceable again (ns-win.el).

Grüße, Carsten

  (defun ns-utf8-nfd-post-read-conversion (length)
    "Calls `ns-convert-utf8-nfd-to-nfc' to compose char sequences."
    (save-excursion
      (save-restriction
        (narrow-to-region (point) (+ (point) length))
        (let ((str (buffer-string)))
          (delete-region (point-min) (point-max))
          (insert
           (condition-case nil
               (ns-convert-utf8-nfd-to-nfc str)
             (error str)))
          (- (point-max) (point-min))))))