Any way to remove ^L or ^M from some buffer outputs?

Any way to remove ^L or ^M from some buffer outputs?

[Jean Louis]

\f
Compiling file /home/data1/protected/Programming/emacs-lisp/rcd-cf.el at Sun Oct  4 17:48:33 2020

In cf-sms-exists:
rcd-cf.el:437:18: Warning: reference to free variable ‘n’ Disable showing
    Disable logging

I am not sure if you can see ^L above, let us say I am compiling or
having some shell output, then I see often ^L, this may be line feed,
I do not know, but it looks green here on my side and is often not
interpreted correctly, line after line, but just as special character
after which other output lines are shown.

In this case I see Emacs compiling buffer. It seems not necessary to
be in the buffer like that.

Then in shell buffers, I see something like this below:

Output file: /home/data1/protected/Media/Videos/Recordings/2020/10/2020-10-04/2020-10-04-17:51:02.ogv
^M[0%] ^M[1%] ^M[2%] ^M[3%] ^M[4%] ^M[5%] ^M[6%] ^M[7%] ^M[8%] ^M[9%]
^M[10%] ^M[10%] ^M[11%] ^M[12%] ^M[13%] ^M[14%] ^M[15%] ^M[16%]
^M[17%] ^M[18%] ^M[19%] ^M[20%] ^M[20%] ^M[21%] ^M[22%] ^M[23%]
^M[24%] ^M[25%] ^M[26%] ^M[27%] ^M[28%] ^M[29%] ^M[30%]

Is that alright to be so? Maybe there is way to remove those by
setting something?

RE: Any way to remove ^L or ^M from some buffer outputs?

[Drew Adams]

Use `query-replace' (`M-%') or `M-x replace-string'.

When prompted for what to replace, use `C-q' followed by the Control key (char) you want to replace. When prompted for what to replace it with, just hit `RET' (replace it with the empty string).

E.g., to remove Control-M chars:

M-x replace-string RET C-q C-m RET RET

(Using `C-m' is the same as using `RET' - same char.  But I write it that way to make it clear that `C-q C-m', which inserts a `C-m' char, is the string of chars to replace.)

If you want to be sure/careful of what you're doing, use `M-%' instead.  Once you see that it's doing what you want (by using `y' to replace some occurrences), just hit `!' to replace the rest of the chars.

In both cases, start with the cursor at bob (beginning of buffer), or `point-min' if you want to do it on the narrowed portion the buffer.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Drew Adams wrote:

> Use `query-replace' (`M-%') or `M-x replace-string'

But this assumes the buffer is editable ... besides
there should be a command (onee) that toggles the
visibility of that specific kind of representation,
whatever it is called, for _all_ such chars...

And, even if one wishes to do it manually one would
use a regexp, not not query about it, right?

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Any way to remove ^L or ^M from some buffer outputs?

[Jean Louis]

* Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> [2020-10-04 21:42]:
> Drew Adams wrote:
> 
> > Use `query-replace' (`M-%') or `M-x replace-string'
> 
> But this assumes the buffer is editable ... besides
> there should be a command (onee) that toggles the
> visibility of that specific kind of representation,
> whatever it is called, for _all_ such chars...

It is in read-only buffers, I was just thinking there is option to
remove that as it looks as garbiage.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Eli Zaretskii]

> Date: Mon, 5 Oct 2020 12:06:44 +0300
> From: Jean Louis <bugs@gnu.support>
> 
> > But this assumes the buffer is editable ... besides
> > there should be a command (onee) that toggles the
> > visibility of that specific kind of representation,
> > whatever it is called, for _all_ such chars...
> 
> It is in read-only buffers

You can make it writable with "C-x C-q".

Re: Any way to remove ^L or ^M from some buffer outputs?

[Joost Kremers]

On Mon, Oct 05 2020, Jean Louis wrote:
> It is in read-only buffers, I was just thinking there is option 
> to
> remove that as it looks as garbiage.

There is a package that should take care of ^L at least:

https://github.com/purcell/page-break-lines

-- 
Joost Kremers
Life has its moments

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Joost Kremers wrote:

> There is a package that should take care of ^L at least:

What is the deal with them chars, what are they
called, and why do they appear to begin with,
and where?

I know them only from IRC where you can do, with ERC
first

  (setq erc-interpret-mirc-color t)

then

  $ 08,04 * red army *  ( <-- here I use real chars, i.e
                                  M-x insert-char RET 3 RET )

which looks like this

  $ ^C08,04 * red army * ^C

only the ^C are in cyan (wierd, `what-face' [1] don't
report any face for them?)

BTW, actually, you don't need the closing char if
there is just one thing to color and that's to the
EOL. but consider:

  $ 08,03it\x0302,00al\x0308,04ia

which looks like

  $ ^C08,03it^C^C02,00al^C^C08,04ia^C

See this dump for how that looks, the red army one :) [2]

How to use the colors: [3]

Bonus photo: another red machine [4]


[1] https://dataswamp.org/~incal/emacs-init/my-faces.el

[2] https://dataswamp.org/~incal/figures/emacs/red-army.png

[3] https://help.codeux.com/textual/Text-Formatting.kb

[4] https://dataswamp.org/~incal/pimgs/red-machine.png

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

RE: Any way to remove ^L or ^M from some buffer outputs?

[Drew Adams]

> > > Use `query-replace' (`M-%') or `M-x replace-string'
> >
> > But this assumes the buffer is editable ... besides
> > there should be a command (onee) that toggles the
> > visibility of that specific kind of representation,
> > whatever it is called, for _all_ such chars...
> 
> It is in read-only buffers, 

If you want to edit a read-only buffer, toggle it
to writable with `C-x C-q'.

If it's a file buffer and you don't have write
permission for the file, then you likely won't be
able to write the updated buffer back to the file.
But that's a different problem from being able to
edit the buffer.

> I was just thinking there is option to
> remove that as it looks as garbiage.

Garbage/noise is in the eye of the beholder.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Jean Louis]

* Drew Adams <drew.adams@oracle.com> [2020-10-05 20:06]:
> > > > Use `query-replace' (`M-%') or `M-x replace-string'
> > >
> > > But this assumes the buffer is editable ... besides
> > > there should be a command (onee) that toggles the
> > > visibility of that specific kind of representation,
> > > whatever it is called, for _all_ such chars...
> > 
> > It is in read-only buffers, 
> 
> If you want to edit a read-only buffer, toggle it
> to writable with `C-x C-q'.

Those are shell output buffers and similar, and it looks ugly, I was
thinking there is some option to set, to avoid that. Of course I will
not edit it with hand, it is about watching output nicely formatted.

For example instead of ^L to get new line, I do not see new line, so
it just continues.

> If it's a file buffer and you don't have write
> permission for the file, then you likely won't be
> able to write the updated buffer back to the file.
> But that's a different problem from being able to
> edit the buffer.
> 
> > I was just thinking there is option to
> > remove that as it looks as garbiage.
> 
> Garbage/noise is in the eye of the beholder.

Exactly. So it is not expected output. If I do shell command, there is
expected output that should be similar or same as in the shell, in
Emacs the new lines are not always interpreted correctly.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Eli Zaretskii]

> Date: Mon, 5 Oct 2020 21:14:25 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: help-gnu-emacs@gnu.org
> 
> Those are shell output buffers and similar, and it looks ugly, I was
> thinking there is some option to set, to avoid that. Of course I will
> not edit it with hand, it is about watching output nicely formatted.

If this text comes from the shell, you can make Emacs eliminate the ^M
characters by setting the read part of process-coding-system to *-dos.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Eli Zaretskii]

> Date: Mon, 05 Oct 2020 21:21:57 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> If this text comes from the shell, you can make Emacs eliminate the ^M
> characters by setting the read part of process-coding-system to *-dos.

Or maybe *-mac, if the ^M characters appear alone, not before a
newline.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Yuri Khan]

On Tue, 6 Oct 2020 at 01:22, Eli Zaretskii <eliz@gnu.org> wrote:

> > Those are shell output buffers and similar, and it looks ugly, I was
> > thinking there is some option to set, to avoid that. Of course I will
> > not edit it with hand, it is about watching output nicely formatted.
>
> If this text comes from the shell, you can make Emacs eliminate the ^M
> characters by setting the read part of process-coding-system to *-dos.

I believe the issue is not about eliminating the ^M characters as
such. The program is clearly using the CR character to overwrite its
earlier output. Jean probably wants to obtain that effect in an Emacs
shell buffer — seeing only the last string output to a line.

To achieve that, I’d probably try adding a font-lock pattern of .*\r
and fontify it with an invisible face.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Stefan Monnier]

>> Those are shell output buffers and similar, and it looks ugly, I was
>> thinking there is some option to set, to avoid that. Of course I will
>> not edit it with hand, it is about watching output nicely formatted.
>
> If this text comes from the shell, you can make Emacs eliminate the ^M
> characters by setting the read part of process-coding-system to *-dos.

I think in the OP's case what the users really want is not that but to
treat it like a terminal would (i.e. erasing to BOL).


        Stefan

Re: Any way to remove ^L or ^M from some buffer outputs?

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-10-05 21:23]:
> > Date: Mon, 5 Oct 2020 21:14:25 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: help-gnu-emacs@gnu.org
> > 
> > Those are shell output buffers and similar, and it looks ugly, I was
> > thinking there is some option to set, to avoid that. Of course I will
> > not edit it with hand, it is about watching output nicely formatted.
> 
> If this text comes from the shell, you can make Emacs eliminate the ^M
> characters by setting the read part of process-coding-system to *-dos.

Do you mean the variable default-process-coding-system ?

Re: Any way to remove ^L or ^M from some buffer outputs?

[Eli Zaretskii]

> Date: Sat, 10 Oct 2020 03:23:02 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: help-gnu-emacs@gnu.org
> 
> * Eli Zaretskii <eliz@gnu.org> [2020-10-05 21:23]:
> > > Date: Mon, 5 Oct 2020 21:14:25 +0300
> > > From: Jean Louis <bugs@gnu.support>
> > > Cc: help-gnu-emacs@gnu.org
> > > 
> > > Those are shell output buffers and similar, and it looks ugly, I was
> > > thinking there is some option to set, to avoid that. Of course I will
> > > not edit it with hand, it is about watching output nicely formatted.
> > 
> > If this text comes from the shell, you can make Emacs eliminate the ^M
> > characters by setting the read part of process-coding-system to *-dos.
> 
> Do you mean the variable default-process-coding-system ?

I'm not sure, because you didn't tell enough details regarding how
this text was received from the shell.  In the simplest case, using
"C-x RET c" prefix before invoking the shell command could be a better
solution; no need to change the defaults.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Patrick Mahan]

I have a user defined lisp function -

;; If you encounter a file with ^M or ... at the end of every line
;; this gets rid of them by pressing F5.
(defun cut-ctrlM ()
  "Cut all visible ^M."
  (interactive)
  (beginning-of-buffer)
  (while (search-forward "\r" nil t)
    (replace-match "" nil t))
  (not-modified)
  (beginning-of-buffer))

Patrick
Emacs addict since 1994


On Sun, Oct 4, 2020 at 7:53 AM Jean Louis <bugs@gnu.support> wrote:

>
> Compiling file /home/data1/protected/Programming/emacs-lisp/rcd-cf.el at
> Sun Oct  4 17:48:33 2020
>
> In cf-sms-exists:
> rcd-cf.el:437:18: Warning: reference to free variable ‘n’ Disable showing
>     Disable logging
>
> I am not sure if you can see ^L above, let us say I am compiling or
> having some shell output, then I see often ^L, this may be line feed,
> I do not know, but it looks green here on my side and is often not
> interpreted correctly, line after line, but just as special character
> after which other output lines are shown.
>
> In this case I see Emacs compiling buffer. It seems not necessary to
> be in the buffer like that.
>
> Then in shell buffers, I see something like this below:
>
> Output file:
> /home/data1/protected/Media/Videos/Recordings/2020/10/2020-10-04/2020-10-04-17:51:02.ogv
> ^M[0%] ^M[1%] ^M[2%] ^M[3%] ^M[4%] ^M[5%] ^M[6%] ^M[7%] ^M[8%] ^M[9%]
> ^M[10%] ^M[10%] ^M[11%] ^M[12%] ^M[13%] ^M[14%] ^M[15%] ^M[16%]
> ^M[17%] ^M[18%] ^M[19%] ^M[20%] ^M[20%] ^M[21%] ^M[22%] ^M[23%]
> ^M[24%] ^M[25%] ^M[26%] ^M[27%] ^M[28%] ^M[29%] ^M[30%]
>
> Is that alright to be so? Maybe there is way to remove those by
> setting something?
>
>

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Patrick Mahan wrote:

> (defun cut-ctrlM ()
>   "Cut all visible ^M."
>   (interactive)
>   (beginning-of-buffer)
>   (while (search-forward "\r" nil t)
>     (replace-match "" nil t))
>   (not-modified)
>   (beginning-of-buffer))

Well, yeah, if we want to replace (not hide), only,
as always, the byte compiler is your friend!

  In cut-ctrlM:
  [...]Warning: ‘beginning-of-buffer’ is for interactive use only; use
      ‘(goto-char (point-min))’ instead.
  [...]Warning: ‘not-modified’ is for interactive use only; use
      ‘set-buffer-modified-p’ instead.
  [...]Warning: ‘beginning-of-buffer’ is for interactive use only; use
      ‘(goto-char (point-min))’ instead.

And... maybe wrap it in `save-excursion' as well?

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Any way to remove ^L or ^M from some buffer outputs?

[Patrick Mahan]

Yes, I have never byte-compiled it as I have only used it interactively to
clean-up text files coming from a DOS world to linux/unix/macos.  It is
defined in my .emacs init file.

On Sun, Oct 4, 2020 at 12:50 PM Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> Patrick Mahan wrote:
>
> > (defun cut-ctrlM ()
> >   "Cut all visible ^M."
> >   (interactive)
> >   (beginning-of-buffer)
> >   (while (search-forward "\r" nil t)
> >     (replace-match "" nil t))
> >   (not-modified)
> >   (beginning-of-buffer))
>
> Well, yeah, if we want to replace (not hide), only,
> as always, the byte compiler is your friend!
>
>   In cut-ctrlM:
>   [...]Warning: ‘beginning-of-buffer’ is for interactive use only; use
>       ‘(goto-char (point-min))’ instead.
>   [...]Warning: ‘not-modified’ is for interactive use only; use
>       ‘set-buffer-modified-p’ instead.
>   [...]Warning: ‘beginning-of-buffer’ is for interactive use only; use
>       ‘(goto-char (point-min))’ instead.
>
> And... maybe wrap it in `save-excursion' as well?
>
> --
> underground experts united
> http://user.it.uu.se/~embe8573
> https://dataswamp.org/~incal
>
>
>

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Patrick Mahan wrote:

> Yes, I have never byte-compiled it as I have only
> used it interactively to clean-up text files coming
> from a DOS world to linux/unix/macos. It is defined
> in my .emacs init file.

Well, you can byte-compile your .emacs as well. It is
just another Elisp file...

Anyway, here (last) is a refined version, if you
don't mind...

I wonder tho... why does (insert-char 13) insert ^M
but M-x insert-char 13 RET inserts ^S?

(defun delete-c-m (&optional beg end)
  (interactive (if (use-region-p)
                   (list (region-beginning) (region-end))
                 (list (point-min) (point-max)) ))
  (save-excursion
    (goto-char beg)
    (while (search-forward "\r" end t)
      (replace-match "" nil t) )
    (set-buffer-modified-p nil) ; can be user or omitted...
    ))
;; (insert-char 13)

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Any way to remove ^L or ^M from some buffer outputs?

[2QdxY4RzWzUUiLuE]

On 2020-10-05 at 00:56:05 +0200,
Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> I wonder tho... why does (insert-char 13) insert ^M
> but M-x insert-char 13 RET inserts ^S?

Interactively, insert-char takes a *hex* value.

13 (hex) is 19 (decimal), which is Control-S.

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

2QdxY4RzWzUUiLuE wrote:

>> I wonder tho... why does (insert-char 13) insert
>> ^M but M-x insert-char 13 RET inserts ^S?
>
> Interactively, insert-char takes a *hex* value.
>
> 13 (hex) is 19 (decimal), which is Control-S.

Ha! Clever...

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Any way to remove ^L or ^M from some buffer outputs?

[Emanuel Berg via Users list for the GNU Emacs text editor]

> (defun delete-c-m (&optional beg end)
>   (interactive (if (use-region-p)
>                    (list (region-beginning) (region-end))
>                  (list (point-min) (point-max)) ))
>   (save-excursion
>     (goto-char beg)
>     (while (search-forward "\r" end t)
>       (replace-match "" nil t) )
>     (set-buffer-modified-p nil) ; can be used or omitted...
>     ))

BTW `search-forward' has a formally incorrect
docstring in

  GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu) of
  2019-09-23, modified by Debian

that way "bounds" won't get font locked with
`help-argument-name' (it should be BOUNDs, or
reformulated: "An optional second argument bounds the
search" - ... and optimally the word appears as the
argument is and nothing else, right? but regardless
of whatever, please just upcase it so one can see it
instantly, and I'm happy :))

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: Any way to remove ^L or ^M from some buffer outputs?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On Mon, Oct 05, 2020 at 12:56:05AM +0200, Emanuel Berg via Users list for the GNU Emacs text editor wrote:
> Patrick Mahan wrote:
> 
> > Yes, I have never byte-compiled it as I have only
> > used it interactively to clean-up text files coming
> > from a DOS world to linux/unix/macos. It is defined
> > in my .emacs init file.
> 
> Well, you can byte-compile your .emacs as well. It is
> just another Elisp file...
> 
> Anyway, here (last) is a refined version, if you
> don't mind...
> 
> I wonder tho... why does (insert-char 13) insert ^M
> but M-x insert-char 13 RET inserts ^S?

Because the interactive version (your second above) tries to be
helpful and interprets the 13 as hex, whereas the non-interactive
is plain lisp, and there 13 is decimal 13, as it always has been.

To be fair, the interactive version has told you, in the prompt.

But who reads prompts these days? ;-D

(Not intended by any means as a slur at anyone: I've been caught
doing that more than once!)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Any way to remove ^L or ^M from some buffer outputs?

[Nick Dokos]

When you do M-x insert-char RET, you get a prompt that says:

Insert character (Unicode name or hex):

So you need to enter D to get decimal 13.

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler

Re: Any way to remove ^L or ^M from some buffer outputs?

[Nick Dokos]

Jean Louis <bugs@gnu.support> writes:

> \f
> Compiling file /home/data1/protected/Programming/emacs-lisp/rcd-cf.el at Sun Oct  4 17:48:33 2020
>
> In cf-sms-exists:
> rcd-cf.el:437:18: Warning: reference to free variable ‘n’ Disable showing
>     Disable logging
>
> I am not sure if you can see ^L above, let us say I am compiling or
> having some shell output, then I see often ^L, this may be line feed,
> I do not know, but it looks green here on my side and is often not
> interpreted correctly, line after line, but just as special character
> after which other output lines are shown.

#+BEGIN_ANCIENT_HISTORY

It is a "form-feed" - in the olden days, the line printer program would output
one of these at the end of a job (although programs could do that as well) in
order to line up the output of the next program with the perforations in the
continuous-feed paper.

Terminals sometimes used it to clear the screen. And it is still used e.g. when
you define file-local variables in emacs:

#+begin_quote
   If some unrelated text might look to Emacs as a local variables list,
you can countermand that by inserting a form-feed character (a page
delimiter, *note Pages::) after that text.  Emacs only looks for
file-local variables in the last page of a file, after the last page
delimiter.
#+end_quote

>
> In this case I see Emacs compiling buffer. It seems not necessary to
> be in the buffer like that.
>
> Then in shell buffers, I see something like this below:
>
> Output file: /home/data1/protected/Media/Videos/Recordings/2020/10/2020-10-04/2020-10-04-17:51:02.ogv
> ^M[0%] ^M[1%] ^M[2%] ^M[3%] ^M[4%] ^M[5%] ^M[6%] ^M[7%] ^M[8%] ^M[9%]
> ^M[10%] ^M[10%] ^M[11%] ^M[12%] ^M[13%] ^M[14%] ^M[15%] ^M[16%]
> ^M[17%] ^M[18%] ^M[19%] ^M[20%] ^M[20%] ^M[21%] ^M[22%] ^M[23%]
> ^M[24%] ^M[25%] ^M[26%] ^M[27%] ^M[28%] ^M[29%] ^M[30%]
>
> Is that alright to be so? Maybe there is way to remove those by
> setting something?
>
>
This is the output of a program that implements a poor man's version
of a progress meter: ^M is a carriage return (CR) which moves the
typewriter head to the beginning of the current line (it is then
usually - but not in this case - followed by a line feed in order to
allow you to type on the next line - I believe Windows text files
still have a CR/LF combination marking the end of each line in the
file).

By prepending a CR on each output, the next output overwrites the
previous value, showing you the progress of the task.

But it only works on typewrites, teletypes, terminals and terminal
emulators that implement these operations exactly. A shell in emacs
does not do that and prefers to output everything more-or-less verbatim.

#END_ANCIENT_HISTORY

We now return you to the 21st century...

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler