Why is `C-x 8' limited to Latin-1 for search?

Why is `C-x 8' limited to Latin-1 for search?

[Drew Adams]

I was thinking that one could now use `C-x 8 RET' (which is now `insert-char')
to insert any Unicode char into the Isearch search string.  Apparently not.

At least it doesn't seem to work (`C-x 8 RET' is bound to
`isearch-other-control-char'), and I couldn't find it mentioned in the doc.
Looking for it in the doc, I found only a mention that you can use `C-x 8' to
insert Latin-1 chars (and so search for them).

Is it intentional that you cannot search for other Unicode chars using `C-x 8
RET'?  Is that not possible only because it is not yet implemented, or is there
some reason why it should not be available (implemented)?

Yes, I know that `C-x 8 RET' is not necessarily an ideal way to input chars.
But it seems like you should be able to use it here anyway.

If there is a good reason why one should not be able to use `C-x 8 RET' to
directly input chars during Isearch, what about letting users at least use `C-x
8 RET' after `M-e', to input chars that way?

---

To try the latter quickly (no, this is not the way to code this; it's just a
quick way to let you try it):

(unless (fboundp 'ORIG-isearch-edit-string)
  (defalias 'ORIG-isearch-edit-string
            (symbol-function 'isearch-edit-string)))

(defun isearch-edit-string ()
  (interactive)
  (let ((enable-recursive-minibuffers  t))
    (ORIG-isearch-edit-string)))

In a buffer with some Unicode chars, e.g., `λ':

C-s M-e C-x 8 RET GREEK SMALL LETTER LAMBDA C-s

(You have completion for the character name, at least.)

RE: Why is `C-x 8' limited to Latin-1 for search?

[Drew Adams]

> To try the latter quickly (no, this is not the way to code 
> this; it's just a quick way to let you try it):
> 
> (unless (fboundp 'ORIG-isearch-edit-string)
>   (defalias 'ORIG-isearch-edit-string
>             (symbol-function 'isearch-edit-string)))
> 
> (defun isearch-edit-string ()
>   (interactive)
>   (let ((enable-recursive-minibuffers  t))
>     (ORIG-isearch-edit-string)))
> 
> In a buffer with some Unicode chars, e.g., `λ':
> 
> C-s M-e C-x 8 RET GREEK SMALL LETTER LAMBDA C-s
> 
> (You have completion for the character name, at least.)

And you can try the former quickly too (i.e., use `C-x 8 RET' directly in
Isearch without going through `M-e') with the code below. Again, I'm not
proposing using this code as is; this is just to let you see how the feature
might act.

E.g., after evaling the code below, to search for `abcλdef', do this (with
completion for the char name):

  C-s a b c C-x 8 RET GREEK SMALL LETTER LAMBDA RET d e f

(define-key isearch-mode-map "\C-x8\r" 'foo)

;; 99% the same as `isearch-edit-string' (`M-e'). This just calls
`read-char-by-name'.
;; The common code could be factored out, if this were how we wanted to
implement the feature.
;; (I didn't try to see if everything else here is needed for this feature, as
it is needed for `M-e'.)
;;
(defun foo ()
  (interactive)
  (condition-case nil
      (progn
	(let ((enable-recursive-minibuffer  t) ; <====== ADDED ====
              (isearch-nonincremental isearch-nonincremental)
	      ;; Locally bind all isearch global variables to protect them
	      ;; from recursive isearching.
	      ;; isearch-string -message and -forward are not bound
	      ;; so they may be changed.  Instead, save the values.
	      (isearch-new-string isearch-string)
	      (isearch-new-message isearch-message)
	      (isearch-new-forward isearch-forward)
	      (isearch-new-word isearch-word)
	      (isearch-new-case-fold isearch-case-fold-search)
	      (isearch-regexp isearch-regexp)
	      (isearch-op-fun isearch-op-fun)
	      (isearch-cmds isearch-cmds)
	      (isearch-success isearch-success)
	      (isearch-wrapped isearch-wrapped)
	      (isearch-barrier isearch-barrier)
	      (isearch-adjusted isearch-adjusted)
	      (isearch-yank-flag isearch-yank-flag)
	      (isearch-error isearch-error)
	      (isearch-opoint isearch-opoint)
	      (isearch-slow-terminal-mode isearch-slow-terminal-mode)
	      (isearch-small-window isearch-small-window)
	      (isearch-recursive-edit isearch-recursive-edit)
	      ;; Save current configuration so we can restore it here.
	      (isearch-window-configuration (current-window-configuration))
	      ;; This could protect the index of the search rings,
	      ;; but we can't reliably count the number of typed M-p
	      ;; in `read-from-minibuffer' to adjust the index accordingly.
	      ;; So when the following is commented out, `isearch-mode'
	      ;; below resets the index to the predictable value nil.
	      ;; (search-ring-yank-pointer search-ring-yank-pointer)
	      ;; (regexp-search-ring-yank-pointer
regexp-search-ring-yank-pointer)
	      ;; Temporarily restore `minibuffer-message-timeout'.
	      (minibuffer-message-timeout
	       isearch-original-minibuffer-message-timeout)
	      (isearch-original-minibuffer-message-timeout
	       isearch-original-minibuffer-message-timeout)
	      old-point old-other-end)
	  ;; Actually terminate isearching until editing is done.
	  ;; This is so that the user can do anything without failure,
	  ;; like switch buffers and start another isearch, and return.
	  (condition-case nil
	      (isearch-done t t)
	    (exit nil))			; was recursive editing
	  ;; Save old point and isearch-other-end before reading from minibuffer
	  ;; that can change their values.
	  (setq old-point (point) old-other-end isearch-other-end)
	  (unwind-protect
               (let* ((message-log-max nil)
                      ;; Don't add a new search string to the search ring here
                      ;; in `read-from-minibuffer'. It should be added only
                      ;; by `isearch-update-ring' called from `isearch-done'.
                      (history-add-new-input nil)
                      ;; Binding minibuffer-history-symbol to nil is a
work-around
                      ;; for some incompatibility with gmhist.
                      (minibuffer-history-symbol))
                 (setq isearch-new-string
                       (concat
                        isearch-string

                       ;; ========= THIS IS ALL THAT'S NEW =========
                        (string (read-char-by-name "Unicode (name or hex): ")))

                       isearch-new-message
                       (mapconcat 'isearch-text-char-description
                                  isearch-new-string "")))
	    ;; Set point at the start (end) of old match if forward (backward),
	    ;; so after exiting minibuffer isearch resumes at the start (end)
	    ;; of this match and can find it again.
	    (if (and old-other-end (eq old-point (point))
		     (eq isearch-forward isearch-new-forward))
		(goto-char old-other-end))
	    ;; Always resume isearching by restarting it.
	    (isearch-mode isearch-forward
			  isearch-regexp
			  isearch-op-fun
			  nil
			  isearch-word)
	    ;; Copy new local values to isearch globals
	    (setq isearch-string isearch-new-string
		  isearch-message isearch-new-message
		  isearch-forward isearch-new-forward
		  isearch-word isearch-new-word
		  isearch-case-fold-search isearch-new-case-fold))
	  ;; Empty isearch-string means use default.
	  (when (= 0 (length isearch-string))
	    (setq isearch-string (or (car (if isearch-regexp
					      regexp-search-ring
					    search-ring))
				     "")
		  isearch-message
		  (mapconcat 'isearch-text-char-description
			     isearch-string ""))
	    ;; After taking the last element, adjust ring to previous one.
	    (isearch-ring-adjust1 nil)))
	;; This used to push the state as of before this C-s, but it adds
	;; an inconsistent state where part of variables are from the
	;; previous search (e.g. `isearch-success'), and part of variables
	;; are just entered from the minibuffer (e.g. `isearch-string').
	;; (isearch-push-state)
	;; Reinvoke the pending search.
	(isearch-search)
	(isearch-push-state)           ; this pushes the correct state
	(isearch-update)
	(if isearch-nonincremental
	    (progn
	      ;; (sit-for 1) ;; needed if isearch-done does: (message "")
	      (isearch-done)
	      ;; The search done message is confusing when the string
	      ;; is empty, so erase it.
	      (if (equal isearch-string "")
		  (message "")))))
    (quit            ; handle abort-recursive-edit
     (isearch-abort) ;; outside of let to restore outside global values
     )))

Re: Why is `C-x 8' limited to Latin-1 for search?

[Juri Linkov]

> ;; 99% the same as `isearch-edit-string' (`M-e'). This just calls `read-char-by-name'.
> ;; The common code could be factored out, if this were how we wanted to implement the feature.
> ;; (I didn't try to see if everything else here is needed for this feature, as it is needed for `M-e'.)

To avoid code duplication, a new macro could be created
with a name like `with-isearch-paused' and the body from `isearch-edit-string'
except the part that changes `isearch-new-string' and `isearch-new-message'.
It could be used in a new command with a name like `isearch-insert-char-by-name'.

RE: Why is `C-x 8' limited to Latin-1 for search?

[Drew Adams]

> To avoid code duplication, a new macro could be created
> with a name like `with-isearch-paused' and the body from 
> `isearch-edit-string' except the part that changes
> `isearch-new-string' and `isearch-new-message'.
> It could be used in a new command with a name like 
> `isearch-insert-char-by-name'.

FWIW, I did that yesterday in my own code (isearch+.el).
If Emacs adds this or similar then I won't need to do that. 

I named the macro `isearchp-use-new-search-string' and the new command
`isearchp-read-unicode-char'.  I bind `isearchp-read-unicode-char' to `C-x 8
RET' in `isearch-mode-map' and `insert-char' to `C-x 8 RET' in
`minibuffer-local-isearch-map'.

The macro is passed a sexp that is evaluated to produce the new search string.
So for `isearchp-read-unicode-char', this is the call:

(defun isearchp-read-unicode-char ()
    (interactive)
    (isearchp-use-new-search-string
     (concat isearch-string
             (string
              (read-char-by-name
                "Unicode (name or hex): ")))))

And for `isearch-edit-string' the sexp passed is just the current
`read-from-minibuffer' sexp.

But I don't know what you mean by "except the part that changes
`isearch-new-string' and `isearch-new-message'.  I do change both, to reflect
the added char.  No doubt you had a slightly different macro in mind.  Anyway,
FWIW, the code I used is here:
http://www.emacswiki.org/emacs-en/download/isearch%2b.el

Something else to consider perhaps is whether another (perhaps additional),
shorter key sequence should be bound for this.  My thinking was that `C-x 8 RET'
is enough, since (a) it is what users will remember for this, and (b) reading
the char name is anyway an interruption/slowdown, so it would not help much to
introduce the reading with a quick key sequence.

Another possibility, which I used at first but abandoned, would be to not bind
`enable-recursive-minibuffers' to non-nil systematically in the macro, but to
bind it to a value passed as an additional (optional) parameter.  For my use, at
least, I decided this was not necessary/useful - I just bind it to t.

HTH.

Re: Why is `C-x 8' limited to Latin-1 for search?

[Juri Linkov]

> But I don't know what you mean by "except the part that changes
> `isearch-new-string' and `isearch-new-message'.  I do change both, to reflect
> the added char.  No doubt you had a slightly different macro in mind.

I meant that `isearch-edit-string' does more than needed for the command
that will read and insert an Unicode char.  For example, currently
it binds `history-add-new-input' to nil.  This means that your code
doesn't add the char name to the minibuffer's history.

Executing any code in the macro's body will also allow doing more
amazing things like temporarily going into a recursive edit with:

(define-key isearch-mode-map "\M-r"
  (lambda ()
    (interactive)
    (with-isearch-suspend (recursive-edit))))

where `C-M-c' will continue the suspended isearch.

This will be possible to do with this patch:

=== modified file 'lisp/isearch.el'
--- lisp/isearch.el	2012-11-21 10:32:38 +0000
+++ lisp/isearch.el	2012-12-10 21:57:00 +0000
@@ -534,7 +534,7 @@ (defvar isearch-mode-map
     (define-key map "\C-x" nil)
     (define-key map [?\C-x t] 'isearch-other-control-char)
     (define-key map "\C-x8" nil)
-    (define-key map "\C-x8\r" 'isearch-other-control-char)
+    (define-key map "\C-x8\r" 'isearch-insert-char-by-name)
 
     map)
   "Keymap for `isearch-mode'.")
@@ -1140,23 +1140,17 @@ (defun isearch-fail-pos (&optional msg)
 	  (length succ-msg)
 	0))))
 
-(defun isearch-edit-string ()
-  "Edit the search string in the minibuffer.
-The following additional command keys are active while editing.
-\\<minibuffer-local-isearch-map>
-\\[exit-minibuffer] to resume incremental searching with the edited string.
-\\[isearch-nonincremental-exit-minibuffer] to do one nonincremental search.
-\\[isearch-forward-exit-minibuffer] to resume isearching forward.
-\\[isearch-reverse-exit-minibuffer] to resume isearching backward.
-\\[isearch-complete-edit] to complete the search string using the search ring."
-
+(defmacro with-isearch-suspend (&rest body)
+  "Exit Isearch mode, run BODY, and reinvoke the pending search.
+You can update the global isearch variables by setting new values to
+`isearch-new-string', `isearch-new-message', `isearch-new-forward',
+`isearch-new-word', `isearch-new-case-fold'."
   ;; This code is very hairy for several reasons, explained in the code.
   ;; Mainly, isearch-mode must be terminated while editing and then restarted.
   ;; If there were a way to catch any change of buffer from the minibuffer,
   ;; this could be simplified greatly.
   ;; Editing doesn't back up the search point.  Should it?
-  (interactive)
-  (condition-case nil
+  `(condition-case nil
       (progn
 	(let ((isearch-nonincremental isearch-nonincremental)
 
@@ -1219,29 +1213,7 @@ (defun isearch-edit-string ()
 	  (setq old-point (point) old-other-end isearch-other-end)
 
 	  (unwind-protect
-	      (let* ((message-log-max nil)
-		     ;; Don't add a new search string to the search ring here
-		     ;; in `read-from-minibuffer'. It should be added only
-		     ;; by `isearch-update-ring' called from `isearch-done'.
-		     (history-add-new-input nil)
-		     ;; Binding minibuffer-history-symbol to nil is a work-around
-		     ;; for some incompatibility with gmhist.
-		     (minibuffer-history-symbol))
-		(setq isearch-new-string
-                      (read-from-minibuffer
-                       (isearch-message-prefix nil isearch-nonincremental)
-		       (cons isearch-string (1+ (or (isearch-fail-pos)
-						    (length isearch-string))))
-                       minibuffer-local-isearch-map nil
-                       (if isearch-regexp
-			   (cons 'regexp-search-ring
-				 (1+ (or regexp-search-ring-yank-pointer -1)))
-			 (cons 'search-ring
-			       (1+ (or search-ring-yank-pointer -1))))
-                       nil t)
-		      isearch-new-message
-		      (mapconcat 'isearch-text-char-description
-				 isearch-new-string "")))
+	      (progn ,@body)
 
 	    ;; Set point at the start (end) of old match if forward (backward),
 	    ;; so after exiting minibuffer isearch resumes at the start (end)
@@ -1300,6 +1272,41 @@ (defun isearch-edit-string ()
      (isearch-abort)  ;; outside of let to restore outside global values
      )))
 
+(defun isearch-edit-string ()
+  "Edit the search string in the minibuffer.
+The following additional command keys are active while editing.
+\\<minibuffer-local-isearch-map>
+\\[exit-minibuffer] to resume incremental searching with the edited string.
+\\[isearch-nonincremental-exit-minibuffer] to do one nonincremental search.
+\\[isearch-forward-exit-minibuffer] to resume isearching forward.
+\\[isearch-reverse-exit-minibuffer] to resume isearching backward.
+\\[isearch-complete-edit] to complete the search string using the search ring."
+  (interactive)
+  (with-isearch-suspend
+   (let* ((message-log-max nil)
+	  ;; Don't add a new search string to the search ring here
+	  ;; in `read-from-minibuffer'. It should be added only
+	  ;; by `isearch-update-ring' called from `isearch-done'.
+	  (history-add-new-input nil)
+	  ;; Binding minibuffer-history-symbol to nil is a work-around
+	  ;; for some incompatibility with gmhist.
+	  (minibuffer-history-symbol))
+     (setq isearch-new-string
+	   (read-from-minibuffer
+	    (isearch-message-prefix nil isearch-nonincremental)
+	    (cons isearch-string (1+ (or (isearch-fail-pos)
+					 (length isearch-string))))
+	    minibuffer-local-isearch-map nil
+	    (if isearch-regexp
+		(cons 'regexp-search-ring
+		      (1+ (or regexp-search-ring-yank-pointer -1)))
+	      (cons 'search-ring
+		    (1+ (or search-ring-yank-pointer -1))))
+	    nil t)
+	   isearch-new-message
+	   (mapconcat 'isearch-text-char-description
+		      isearch-new-string "")))))
+
 (defun isearch-nonincremental-exit-minibuffer ()
   (interactive)
   (setq isearch-nonincremental t)
@@ -1870,6 +1877,17 @@ (defun isearch-yank-line ()
    (lambda () (let ((inhibit-field-text-motion t))
 		(line-end-position (if (eolp) 2 1))))))
 
+(defun isearch-insert-char-by-name ()
+  "Read a character by its Unicode name and insert it into search string."
+  (interactive)
+  (with-isearch-suspend
+   (let ((char (read-char-by-name "Insert character (Unicode name or hex): ")))
+     (when char
+       (setq isearch-new-string (concat isearch-string (string char))
+	     isearch-new-message (concat isearch-message
+					 (mapconcat 'isearch-text-char-description
+						    (string char) "")))))))
+
 (defun isearch-search-and-update ()
   ;; Do the search and update the display.
   (when (or isearch-success

RE: Why is `C-x 8' limited to Latin-1 for search?

[Drew Adams]

> > But I don't know what you mean by "except the part that changes
> > `isearch-new-string' and `isearch-new-message'.  I do 
> > change both, to reflect the added char.  No doubt you had a
> > slightly different macro in mind.
> 
> I meant that `isearch-edit-string' does more than needed for 
> the command that will read and insert an Unicode char.  For example,
> currently it binds `history-add-new-input' to nil.  This means that
> your code doesn't add the char name to the minibuffer's history.

Yes, your macro is the same as mine, except that you do not include these 3
bindings (you use them only in `isearch-edit-string'):

 (message-log-max           nil)
 (history-add-new-input     nil)
 (minibuffer-history-symbol nil)

And you do not update `isearch-new-string' and `isearch-new-message' in the
macro itself, but outside it (in the macro callers).

> (define-key isearch-mode-map "\M-r"
>   (lambda ()
>     (interactive)
>     (with-isearch-suspend (recursive-edit))))
>
> `C-M-c' will continue the suspended isearch.

Your tweak improves the macro - my vote says go for it.

I suggest changing the name slightly: `with-isearch-suspended' (or perhaps just
`isearch-suspend').

Re: Why is `C-x 8' limited to Latin-1 for search?

[Juri Linkov]

> To try the latter quickly (no, this is not the way to code this; it's just a
> quick way to let you try it):
>
> (unless (fboundp 'ORIG-isearch-edit-string)
>   (defalias 'ORIG-isearch-edit-string
>             (symbol-function 'isearch-edit-string)))
>
> (defun isearch-edit-string ()
>   (interactive)
>   (let ((enable-recursive-minibuffers  t))
>     (ORIG-isearch-edit-string)))
>
> In a buffer with some Unicode chars, e.g., `λ':
>
> C-s M-e C-x 8 RET GREEK SMALL LETTER LAMBDA C-s
>
> (You have completion for the character name, at least.)

Shouldn't users be able to insert Unicode characters in any minibuffer,
not only in `isearch-edit-string'?  If these is no reason to disallow,
this patch should allow this for `completing-read' in `read-char-by-name':

=== modified file 'lisp/international/mule-cmds.el'
--- lisp/international/mule-cmds.el	2012-10-16 23:09:00 +0000
+++ lisp/international/mule-cmds.el	2012-12-10 07:43:46 +0000
@@ -2945,7 +2945,8 @@ (defun read-char-by-name (prompt)
 This function also accepts a hexadecimal number of Unicode code
 point or a number in hash notation, e.g. #o21430 for octal,
 #x2318 for hex, or #10r8984 for decimal."
-  (let ((input
+  (let* ((enable-recursive-minibuffers t)
+	 (input
          (completing-read
           prompt
           (lambda (string pred action)

RE: Why is `C-x 8' limited to Latin-1 for search?

[Drew Adams]

> Shouldn't users be able to insert Unicode characters in any 
> minibuffer, not only in `isearch-edit-string'?  If these is no
> reason to disallow, this patch should allow this for
> `completing-read' in `read-char-by-name':
> 
> +  (let* ((enable-recursive-minibuffers t)

FWIW, I typically add such a binding not to a function like this but to
functions that call it.  `read-char-by-name' does not know whether it is being
called from a minibuffer, i.e., whether it needs such a binding, but a function
that calls it generally does know that.

I don't think there is necessarily something wrong with doing what you suggest,
so I have no objection.  I just generally prefer to do only what the current
code knows it needs.

One could make an even more radical suggestion than yours: that
`enable-recursive-minibuffers' simply be non-nil by default.  (I am not
suggesting that.)  But the design intention in the past has been to not bind it
to non-nil any more than necessary because it has been thought that recursive
minibuffers entered accidentally can confuse users.

I use mb-depth.el (which I tweak to use a simpler depth indicator and to let
users customize it).  That helps a lot wrt recursive minibuffers.  The main
confusion about recursive minibuffers is not knowing that you are in one or how
deep it is.

With mb-depth.el (which should be turned on by default, IMHO), I'm not sure how
much the original design decision of avoiding non-nil
`enable-recursive-minibuffers' is still appropriate/necessary.