Unable to match octal character

Unable to match octal character

[Boylan, Ross]

I have a file with some characters that display in the emacs buffer as \203.  Yet when I search for that, using
C-s C-q 203 <ret>
I can't match it.  Likewise if I use search and replace.

I have verified that \203 is the display of a single character both by using the arrow key (it's a single step to move over it) and by searching on the literal string \203 or 203.

What's going on?

When I put the cursor on one of these characters and do describe-char I get
            position: 474 of 48736 (1%), column: 25
            character: \203 (displayed as \203) (codepoint 4194179, #o17777603, #x3fff83)
    preferred charset: tis620-2533 (TIS620.2533)
code point in charset: 0x83
               syntax: w        which means: word
             category: L:Left-to-right (strong)
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #x83
            file code: #x83 (encoded by coding system raw-text-unix)
              display: not encodable for terminal

Character code properties: customize what to show
  general-category: Cn (Other, Not Assigned)
  decomposition: (4194179) ('?')

Is the fact that the codepoint is not 203 significant?

History of the file:
SAS running on MS Windows produced rtf output.  SAS has its own font it likes to use,
and the horizontal bar used in tables does not travel well (even on Windows).  That's the character that's causing trouble.
Opened in Wordpad on Windows and exported text.
Move the file to Debian GNU/Linux (UTF-8 environment) and opened the text file in emacs 24.4.1.

Using a slightly different procedure I was able to replace octal characters:
Started with same rtf output.
On linux ran unrtf -- text to convert to text.
Open text file in emacs.  In this case the character was \220.
I tried wordpad because unrtf did not preserve the column alignment

Thanks.
Ross Boylan.

RE: Unable to match octal character

[Drew Adams]

> I have a file with some characters that display in the emacs buffer as
> \203.  Yet when I search for that, using C-s C-q 203 <ret> I can't match it.
> Likewise if I use search and replace.

Works for me.  Do you see the same thing if you start Emacs using
`emacs -Q' (no init file)?

What happens if you do this, starting from emacs -Q:

1. In *scratch*, move point into the text somewhere and use
   `C-q 203 RET' to insert the \203 character.

2. `M-<'

3. `C-s C-q 2 0 3 RET'

Re: Unable to match octal character

[Ben Bacarisse]

"Boylan, Ross" <Ross.Boylan@ucsf.edu> writes:

> I have a file with some characters that display in the emacs buffer as
> \203.  Yet when I search for that, using C-s C-q 203 <ret> I can't
> match it.  Likewise if I use search and replace.

What is the value of the read-quoted-char-radix variable?

<snip>
-- 
Ben.

RE: Unable to match octal character

[Boylan, Ross]

________________________________________
From: Drew Adams [drew.adams@oracle.com]
Sent: Wednesday, April 13, 2016 1:34 PM
To: Boylan, Ross; help-gnu-emacs@gnu.org
Subject: RE: Unable to match octal character

> I have a file with some characters that display in the emacs buffer as
> \203.  Yet when I search for that, using C-s C-q 203 <ret> I can't match it.
> Likewise if I use search and replace.

Works for me.  Do you see the same thing if you start Emacs using
`emacs -Q' (no init file)?

YES

What happens if you do this, starting from emacs -Q:

1. In *scratch*, move point into the text somewhere and use
   `C-q 203 RET' to insert the \203 character.

2. `M-<'

3. `C-s C-q 2 0 3 RET'

THE SEARCH SUCCEEDS.

Ross

RE: Unable to match octal character

[Drew Adams]

> > I have a file with some characters that display in the emacs buffer as
> > \203.  Yet when I search for that, using C-s C-q 203 <ret> I can't match
> > it.  Likewise if I use search and replace.
> 
> Works for me.  Do you see the same thing if you start Emacs using
> `emacs -Q' (no init file)?
> 
> YES

So you see the same problem even without your init file.
But when you do it in *scratch* (below) you don't see the problem.

> What happens if you do this, starting from emacs -Q:
> 1. In *scratch*, move point into the text somewhere and use
>    `C-q 203 RET' to insert the \203 character.
> 2. `M-<'
> 3. `C-s C-q 2 0 3 RET'
> 
> THE SEARCH SUCCEEDS.

Then I guess you will need to explore the difference between
the context where you see the problem, even starting from `emacs -Q',
and the above context (recipe using *scratch*).

Try to provide a step-by-step recipe, which starts from `emacs -Q',
to reproduce the problem.

RE: Unable to match octal character

[Boylan, Ross]

The \203 I insert manually has a much different describe-char than the one from the file.  Here's the manually inserted one:
            position: 212 of 48737 (0%), column: 0
            character: \203 (displayed as \203) (codepoint 131, #o203, #x83)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x83
               syntax: w        which means: word
             category: l:Latin
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #xC2 #x83
            file code: #xC2 #x83 (encoded by coding system raw-text-unix)
              display: terminal code #xC2 #x83

Character code properties: customize what to show
  old-name: NO BREAK HERE
  general-category: Cc (Other, Control)
  decomposition: (131) ('')

So, different character set and codepoint.

I examined the file in binary and there are a bunch of hex 83 = octal 203.  I thought maybe the fact that there is a string of such characters was making a difference, since x83 is not a single byte in UTF-8.  But inserting a space after one of the characters doesn't help.  Perhaps this is because emacs has already encoded it.
Ross
________________________________________
From: Drew Adams [drew.adams@oracle.com]
Sent: Wednesday, April 13, 2016 2:07 PM
To: Boylan, Ross; help-gnu-emacs@gnu.org
Subject: RE: Unable to match octal character

> > I have a file with some characters that display in the emacs buffer as
> > \203.  Yet when I search for that, using C-s C-q 203 <ret> I can't match
> > it.  Likewise if I use search and replace.
>
> Works for me.  Do you see the same thing if you start Emacs using
> `emacs -Q' (no init file)?
>
> YES

So you see the same problem even without your init file.
But when you do it in *scratch* (below) you don't see the problem.

> What happens if you do this, starting from emacs -Q:
> 1. In *scratch*, move point into the text somewhere and use
>    `C-q 203 RET' to insert the \203 character.
> 2. `M-<'
> 3. `C-s C-q 2 0 3 RET'
>
> THE SEARCH SUCCEEDS.

Then I guess you will need to explore the difference between
the context where you see the problem, even starting from `emacs -Q',
and the above context (recipe using *scratch*).

Try to provide a step-by-step recipe, which starts from `emacs -Q',
to reproduce the problem.