Typing and evaluating the following code in the scratch
buffer results in an (appropriate) error, but with an incorrect
message:

(setq new-kmap (make-sparse-keymap))
(define-key new-kmap [?a 27] 'command)
(define-key new-kmap [?a ?\M-x] 'command)
;; Debugger entered--Lisp error:
;;   (error "Key sequence a M-x starts with non-prefix key a")
;; (should be:
;;          "Key sequence a M-x starts with non-prefix key a ESC")

The code in keymap.c loops over the characters in the key sequence and
says to report everything before the current character as the
non-prefix key.  But that misses the case when M-x is converted into
[27 ?x] and it's the `27' part that causes the error.

Please see the attached patch.

A few other cases that show the problem:

(setq a (make-sparse-keymap))
(define-key a [27] 'command)
(define-key a [?\M-x] 'command)
;; Debugger entered--Lisp error:
;;   (error "Key sequence M-x starts with non-prefix key ")
;; (should be:
;;          "Key sequence M-x starts with non-prefix key ESC")

(setq new-kmap (make-sparse-keymap))
(define-key new-kmap [?a 27] 'command)
(define-key new-kmap [?a 27 ?x] 'command)
;; Debugger entered--Lisp error:
;;   (error "Key sequence a M-x starts with non-prefix key a ESC")
;; (error text is correct)

(setq new-kmap (make-sparse-keymap))
(define-key new-kmap [?a] 'command)
(define-key new-kmap [?a ?\M-x] 'command)
;; Debugger entered--Lisp error:
;;    (error "Key sequence a M-x starts with non-prefix key a")
;; (error text is correct)

In GNU Emacs 24.0.50.1 (i686-pc-linux-gnu, GTK+ Version 2.22.0)
 of 2010-12-03 on lappy
Windowing system distributor `The X.Org Foundation', version 11.0.10900000