bug#21780: 25.0.50; Saving *Help* results in bad encoding because of curly quotes

[Drew Adams <drew.adams@oracle.com>]

[-- Attachment #1: Type: text/plain, Size: 4724 bytes --]

> It might work as you expect already.  You can try this:
> 
>   . After "C-h f some-function RET", switch to the *Help* buffer and
>     type "C-x RET f utf-8 RET", then save the buffer as in your
>     recipe.
> 
>   . Now visit the file where you saved the *Help* buffer: if the
>     curved quotes display correctly, then "it works for you".

That's exactly what I reported I did.  The result was seeing octal
escapes when I opened the file in a new Emacs session.

Or so I thought.  I just repeated it now with `emacs -Q' and it does
not seem to be a problem if utf-8 is used.

I think now that what I did earlier, when specified utf-8, I was
using my setup.  When I try that again (with my setup), I see the
problem I reported.

Also, with my setup the *Warning* text is different.  Instead of
providing lots of possible encoding choices and using
chinese-iso-8bit as the default (which is what I get with emacs
-Q - why is that, BTW?), it says:

Select one of the safe coding systems listed below,
or cancel the writing with C-g and edit the buffer
   to remove or modify the problematic characters,
or specify any other coding system (and risk losing
   the problematic characters).

  raw-text no-conversion

It was in that context that I anyway tried utf-8 and got the
awful result.  I also tried raw-text, since utf-8 was not in
the prompt (but was apparently accepted, and apparently did
not help).

The full warning is this, when I use my setup (which uses font
"-outline-Lucida Console-normal-normal-normal-mono-14-*-*-*-c-*-iso8859-1"):

These default coding systems were tried to encode text
in the buffer 'throw-isearch-help5.txt':
  (undecided-unix (489 . 8216) (491 . 8217) (499 . 8216) (503 . 8217)
  (577 . 8216) (583 . 8217) (875 . 8216) (892 . 8217) (912 . 8216)
  (931 . 8217) (963 . 8216)) (iso-latin-1-unix (489 . 8216) (491
  . 8217) (499 . 8216) (503 . 8217) (577 . 8216) (583 . 8217) (875
  . 8216) (892 . 8217) (912 . 8216) (931 . 8217) (963 . 8216))
However, each of them encountered characters it couldn't encode:
  undecided-unix cannot encode these: ' ' ' ' ' ' ' ' ' ' ...
  iso-latin-1-unix cannot encode these: ' ' ' ' ' ' ' ' ' ' ...

Click on a character (or switch to this window by 'C-x o'
and select the characters by RET) to jump to the place it appears,
where 'C-u C-x =' will give information about it.

Select one of the safe coding systems listed below,
or cancel the writing with C-g and edit the buffer
   to remove or modify the problematic characters,
or specify any other coding system (and risk losing
   the problematic characters).

  raw-text no-conversion

I got only a #...# file written with utf-8, as the Emacs 25 build
I have crashes all the time, and trying to select the minibuffer
frame after the popped-up *Warning* frame grabs the selection just
crashes Emacs.  And that happens now when I try again, with my setup.

I've attached those files from the first try I made for this,
with my setup.  I see now that the `#...#' one has UTF-8 encoding
and the other, `...' without the #s, has encoding `t' in the mode
line, which I guess means raw text.

To report the problem I then used `emacs -Q', and this time I
tried raw text, and I saw the octal escapes.  So I mistakenly
reported that I saw them after specifying both utf-8 and raw text.

>   . If the curved quotes look like raw bytes or, worse, pairs of
>     non-ASCII characters, you need to visit such file like this:
> 
>       C-x RET c utf-8 RET C-x C-f FILE-NAME RET

Users should not have to do that.  I thought they would, like me
have that annoyance.

> > In Emacs, before saving, the buffer looks fine.
> 
> It looks fine, but the encoding mnemonic on the mode line is not "U"
> (which stands for UTF-8), right?  That is why Emacs asks you for
> encoding: it cannot save these characters using your locale's default
> encoding (which is what the *Help* buffer uses by default).

> Yes, because you probably told Emacs to use raw-text or somesuch, when
> it asked.

See above.

> > Do I need to save the buffer using some other encoding?  If so, which?
> 
> Yes, you could tell it to use UTF-8 when it asked.  (After my change,
> Emacs will do that automatically, no questions asked, when saving
> *Help* buffers with curved quotes.)

Sounds good.

> > Emacs proposed two encodings (one of which was raw text, which I tried;
> > and I tried also utf-8, which I would have thought would show curly
> > quotes OK.
> 
> UTF-8 should have worked.  I wouldn't expect you to see octal escapes
> after saving in UTF-8.

See above for, I think, the explanation of what I did and saw.

Thx.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: throw-isearch-help.txt --]
[-- Type: text/plain; charset=Windows-1252; name="throw-isearch-help.txt", Size: 11918 bytes --]

isearch-forward is an interactive compiled Lisp function in
`isearch+.el'.

It is bound to C-s, menu-bar search i-search isearch-forward.

(isearch-forward &optional ARG NO-RECURSIVE-EDIT)

For more information check the manuals.

Search forward incrementally - Isearch+ version.

With a non-negative prefix arg, do an incremental regular expression
search instead.

With a negative prefix arg, do a (plain, not regexp) incremental
search across multiple buffers:
 * If the prefix arg is ‘-’ (from ‘M--’) then you are prompted for the
   list of buffers. 
 * Otherwise, (e.g. ‘M-- 2’), you are prompted for a regexp that
   matches the names of the buffers to be searched.

If you try to exit with the search string empty then nonincremental
search is used.


As you type characters, they add to the search string and are found.
The following non-printing keys are bound in ‘isearch-mode-map’.

Options
-------
‘isearchp-case-fold’			- search is case sensitive?
‘isearchp-dim-outside-search-area-flag’ [*] - dim non-search zones?
‘isearchp-dimming-color’ [*]		- color for non-search zones
‘isearchp-set-region-flag’		- select last search target?
‘isearchp-restrict-to-region-flag’	- restrict search to region?
‘isearchp-deactivate-region-flag’	- search deactivates region?
‘isearchp-ignore-comments-flag’ [*]	- ignore THINGs in comments?
‘isearchp-hide-whitespace-before-comment-flag’ [*] - precomment space?
‘isearchp-mouse-2-flag’		- ‘mouse-2’ anywhere yanks selection?
‘isearchp-regexp-quote-yank-flag’	- regexp-quote yanked text?
‘isearchp-toggle-option-flag’		- toggle options too?
‘isearchp-drop-mismatch’	- handling input after search mismatch
‘isearchp-drop-mismatch-regexp-flag’	- regexp search drop mismatch?
‘isearchp-initiate-edit-commands’	- keys that edit, not exit

 [*] Requires library ‘isearch-prop.el’.

Commands
--------
DEL	- cancel last input item from end of search string
RET	- exit, leaving point at location found
C-s	- search again forward, C-r backward
C-y C-w	- yank a word or char from buffer onto search string
C-z	- yank a char from buffer onto search string
C-M-w	- delete char from end of search string
C-y C-e	- yank text up to end of line onto search string
C-y C-y	- yank the last string of killed or copied text
M-y	- replace string just yanked with string killed/copied before it
M-w	- copy current search string to kill ring
C-_	- yank a symbol or char from buffer onto search string
C-(	- yank sexp, symbol, or char from buffer onto search string
C-q	- quote a control character, to search for it
C-x 8 RET	- add a Unicode char to search string by Unicode name
C-M-l	- remove failed part of search string, if any
C-g	- remove failed part of search string, or cancel if none
C-x o	- invoke Emacs command loop recursively, during Isearch
M-g	- insert successful search string from when you hit ‘C-g’
M-s e	- edit the search string in the minibuffer
M-n, M-p	- search for next/previous item in search ring
M-x isearchp-complete	- complete the search string using the search ring
M-%	- run ‘query-replace’ to replace search-string matches
C-M-%	- run ‘query-replace-regexp’
M-s o	- run ‘occur’ for search-string matches
M-s h r	- run ‘highlight-regexp’ to highlight search-string matches
M-x isearchp-fontify-buffer-now	- fontify whole buffer
M-x isearchp-set-region-around-search-target	- select last search

<f1> b	- list all Isearch key bindings
<f1> k	- show documentation of an Isearch key
<f1> m	- show documentation for Isearch mode

M-k	- cycle option ‘isearchp-drop-mismatch’
M-s c	- toggle case-sensitivity (for current search or more: ‘C-u’)
M-s h l	- option ‘lazy-highlight-cleanup’ (removal of highlighting)
C-+	- toggle searching invisible text
M-s i	- toggle searching invisible text, for current search or more
M-s v	- toggle option ‘isearchp-toggle-option-flag’
C-x n	- toggle restricting search to active region
C-SPC	- toggle setting region around search target
C-`	- toggle quoting (escaping) of regexp special characters
M-s w	- toggle word-searching
M-s _	- toggle symbol-searching
M-s SPC	- toggle whitespace matching

A ‘SPC’ char normally matches all whitespace defined by variable
‘search-whitespace-regexp’.  See also variables
‘isearch-lax-whitespace’ and ‘isearch-regexp-lax-whitespace’.

Commands that Require Library ‘isearch-prop.el’
----------------------------------------------- 

C-t	- search for a character (overlay or text) property
C-M-t	- regexp-search for a character (overlay or text) property
C-M-~	- toggle searching complements of normal search contexts
C-M-S-d	- toggle dimming non-search zones
C-M-;	- toggle ignoring comments for ‘isearchp-thing’
M-;	- hide or (‘C-u’) show comments

M-x isearchp-put-prop-on-region	- add a text property to region
M-x isearchp-add-regexp-as-property	- add prop to regexp matches
M-x isearchp-regexp-context-search	- search regexp contexts
M-x isearchp-regexp-define-contexts	- define regexp contexts

M-x isearchp-imenu 	- search Emacs-Lisp definitions
M-x isearchp-imenu-command 	- search Emacs command definitions
M-x isearchp-imenu-non-interactive-function 	- search non-commands
M-x isearchp-imenu-macro 	- search Emacs-Lisp macro definitions

M-x isearchp-thing	- search THING search contexts
M-x isearchp-thing-define-contexts	- define THING contexts
M-x isearchp-previous-visible-thing	- go to previous visible THING
M-x isearchp-next-visible-thing	- go to next visible THING

Input Methods
-------------

If an input method is turned on in the current buffer, that input
method is also active while you are typing characters to search.
To toggle the input method, type C-\.  It also toggles the input
method in the current buffer.

To use a different input method for searching, type C-^,
and specify an input method you want to use.

---

The above keys, bound in ‘isearch-mode-map’, are often controlled by
user options - do <f1> C-a on search-.* to find them.

If either option ‘isearch-allow-prefix’ or option
‘isearch-allow-scroll’ is non-nil then you can use a prefix arg with
an Isearch key.  If option ‘isearch-allow-scroll’ is non-nil then you
can use scrolling keys without exiting Isearch.

If these options are both nil then other control and meta chars
terminate the search and are then used normally (depending on
‘search-exit-option’).  Likewise for function keys and mouse button
events.

If this function is called non-interactively with nil argument
NO-RECURSIVE-EDIT then it does not return to the calling function
until the search is done.  See function ‘isearch-mode’.


Bindings in Isearch minor mode:
------------------------------

key             binding
---             -------

TAB .. C-j      isearch-printing-char
SPC .. ~        isearch-printing-char
€ .. ø¿½¿       isearch-printing-char
€ .. ÿ    isearch-printing-char

C-g             isearch-abort
C-h             isearch-mode-help
RET             isearch-exit
C-o             isearch-moccur
C-q             isearch-quote-char
C-r             isearch-repeat-backward
C-s             isearch-repeat-forward
C-t             isearchp-property-forward
C-w             isearch-yank-word-or-char
C-x             Prefix Command
C-y             Prefix Command
C-z             isearchp-yank-char
ESC             Prefix Command
C-\             isearch-toggle-input-method
C-^             isearch-toggle-specified-input-method
C-_             isearchp-yank-symbol-or-char
DEL             isearch-delete-char
S-SPC           isearch-printing-char
C-SPC           isearchp-toggle-set-region
C-(             isearchp-yank-sexp-symbol-or-char
C-+             isearchp-toggle-search-invisible
C-;             iedit-mode
C-`             isearchp-toggle-regexp-quote-yank
C-S-SPC         isearchp-narrow-to-lazy-highlights
<C-M-return>    isearchp-act-on-demand
<C-M-tab>       icicle-isearch-complete
<C-end>         goto-longest-line
<M-S-delete>    isearchp-cleanup
<M-tab>         icicle-isearch-complete
<backtab>       icicle-search-w-isearch-string
<down-mouse-2>  ignore
<escape>        Prefix Command
<f1>            Prefix Command
<help>          Prefix Command
<mouse-2>       isearch-mouse-2
<next>          isearch-repeat-forward
<prior>         isearch-repeat-backward
<remap>         Prefix Command
<return>        isearch-exit
<switch-frame>  ignore

C-x 8           Prefix Command
C-x n           isearchp-toggle-region-restriction
C-x o           isearchp-open-recursive-edit
C-x r           Prefix Command

C-y C-c         isearchp-yank-char
C-y C-e         isearchp-yank-line
C-y C-w         isearchp-yank-word-or-char
C-y C-y         isearch-yank-kill
C-y ESC         Prefix Command
C-y C-_         isearchp-yank-symbol-or-char
C-y C-(         isearchp-yank-sexp-symbol-or-char
C-y C-2         isearch-yank-secondary

C-M-i           icicle-isearch-complete
C-M-l           isearchp-remove-failed-part
C-M-r           isearch-repeat-backward
C-M-s           isearch-repeat-forward
C-M-t           isearchp-property-forward-regexp
C-M-w           isearch-del-char
C-M-y           isearch-yank-secondary
ESC ESC         Prefix Command
M-%             isearch-query-replace
M-:             isearchp-eval-sexp-and-insert
M-;             isearchp-toggle-hiding-comments
M-O             isearch-moccur-all
M-c             isearch-toggle-case-fold
M-e             isearch-edit-string
M-g             isearchp-retrieve-last-quit-search
M-k             isearchp-cycle-mismatch-removal
M-n             isearch-ring-advance
M-o             icicle-isearch-history-insert
M-p             isearch-ring-retreat
M-r             isearch-toggle-regexp
M-s             Prefix Command
M-w             isearchp-kill-ring-save
M-y             isearch-yank-pop
C-M-S-d         isearchp-toggle-dimming-outside-search-area
C-M-%           isearch-query-replace-regexp
C-M-;           isearchp-toggle-ignoring-comments
C-M-`           isearchp-toggle-literal-replacement
C-M-~           isearchp-toggle-complementing-domain

M-ESC ESC       isearch-cancel

M-s C-e         isearch-yank-line
M-s SPC         isearch-toggle-lax-whitespace
M-s '           isearch-toggle-character-fold
M-s _           isearch-toggle-symbol
M-s c           isearch-toggle-case-fold
M-s e           isearch-edit-string
M-s h           Prefix Command
M-s i           isearch-toggle-invisible
M-s o           isearch-occur
M-s r           isearch-toggle-regexp
M-s v           isearchp-toggle-option-toggle
M-s w           isearch-toggle-word

<escape> <tab>  icicle-isearch-complete

<remap> <isearch-complete>      isearchp-complete

<f1> C-h        isearch-help-for-help
<f1> ?          isearch-help-for-help
<f1> b          isearch-describe-bindings
<f1> k          isearch-describe-key
<f1> m          isearch-describe-mode
<f1> q          help-quit
<f1> <f1>       isearch-help-for-help
<f1> <help>     isearch-help-for-help

<help> C-h      isearch-help-for-help
<help> ?        isearch-help-for-help
<help> b        isearch-describe-bindings
<help> k        isearch-describe-key
<help> m        isearch-describe-mode
<help> q        help-quit
<help> <f1>     isearch-help-for-help
<help> <help>   isearch-help-for-help

C-x r g         isearchp-append-register

C-x 8 RET       isearch-char-by-name

C-y M-g         isearchp-retrieve-last-quit-search
C-y M-y         isearch-yank-pop

M-s h h         hlt-highlight-isearch-matches
M-s h l         isearchp-toggle-lazy-highlight-cleanup
M-s h r         isearch-highlight-regexp
M-s h u         hlt-unhighlight-isearch-matches


[-- Attachment #3: #throw-isearch-help.txt# --]
[-- Type: application/octet-stream, Size: 11620 bytes --]