BOM (byte order mark) in process stdout and stderr

BOM (byte order mark) in process stdout and stderr

[Jürgen Hötzel]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Hi,

I compiled  this C#/Mono program on an UTF-8 GNU/Linux System


using System;

public class Hello
{
    static void Main()
    {
Console.WriteLine("STDOUT");
Console.Error.WriteLine("STDERR");
    }
}

and used this elisp code to get the process output:


(let ((default-process-coding-system '(utf-8-with-signature .
utf-8-with-signature)))
  (start-file-process
   "BOM"
   (generate-new-buffer "*bom-test*")
   "~/tmp/Hello.exe"))

This results in the following process-buffer (hexl-mode):

00000000: efbb bf53 5444 4f55 540a 5354 4445 5252  ...STDOUT.STDERR
00000010: 0a0a 5072 6f63 6573 7320 424f 4d20 6669  ..Process BOM fi
00000020: 6e69 7368 6564 0a                        nished.

The stdout BOM was correctly removed but the stderr BOM is still present.

According to
http://www.gnu.org/software/emacs/manual/html_node/elisp/Output-from-Processes.html

there is now way to handle stderr separately. I made this workaround for
the Emacs fsharp-mode:

https://github.com/juergenhoetzel/fsharpbinding/commit/d0d3600c38d0e9f47a621583d0fc82685feda196

Are there any better solutions?

Jürgen

[-- Attachment #2: Type: text/html, Size: 3574 bytes --]

Re: BOM (byte order mark) in process stdout and stderr

[Eli Zaretskii]

> From: Jürgen Hötzel <juergen@hoetzel.info>
> Date: Thu, 25 Jun 2015 18:41:50 +0200
> 
> (let ((default-process-coding-system '(utf-8-with-signature .
> utf-8-with-signature)))
> (start-file-process
> "BOM"
> (generate-new-buffer "*bom-test*")
> "~/tmp/Hello.exe"))
> 
> This results in the following process-buffer (hexl-mode):
> 
> 00000000: efbb bf53 5444 4f55 540a 5354 4445 5252 ...STDOUT.STDERR
> 00000010: 0a0a 5072 6f63 6573 7320 424f 4d20 6669 ..Process BOM fi
> 00000020: 6e69 7368 6564 0a nished.
> 
> The stdout BOM was correctly removed but the stderr BOM is still present.
> 
> According to
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Output-from-Processes.html
> 
> there is now way to handle stderr separately. I made this workaround for the
> Emacs fsharp-mode:
> 
> https://github.com/juergenhoetzel/fsharpbinding/commit/d0d3600c38d0e9f47a621583d0fc82685feda196
> 
> Are there any better solutions?

You didn't tell what version of Emacs do you want this to be solved.

Re: BOM (byte order mark) in process stdout and stderr

[Jürgen Hötzel]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

On Thu, Jun 25, 2015 at 7:28 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jürgen Hötzel <juergen@hoetzel.info>
> > Date: Thu, 25 Jun 2015 18:41:50 +0200
> >
> > (let ((default-process-coding-system '(utf-8-with-signature .
> > utf-8-with-signature)))
> > (start-file-process
> > "BOM"
> > (generate-new-buffer "*bom-test*")
> > "~/tmp/Hello.exe"))
> >
> > This results in the following process-buffer (hexl-mode):
> >
> > 00000000: efbb bf53 5444 4f55 540a 5354 4445 5252 ...STDOUT.STDERR
> > 00000010: 0a0a 5072 6f63 6573 7320 424f 4d20 6669 ..Process BOM fi
> > 00000020: 6e69 7368 6564 0a nished.
> >
> > The stdout BOM was correctly removed but the stderr BOM is still present.
> >
> > According to
> >
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Output-from-Processes.html
> >
> > there is now way to handle stderr separately. I made this workaround for
> the
> > Emacs fsharp-mode:
> >
> >
> https://github.com/juergenhoetzel/fsharpbinding/commit/d0d3600c38d0e9f47a621583d0fc82685feda196
> >
> > Are there any better solutions?
>
> You didn't tell what version of Emacs do you want this to be solved.
>

GNU Emacs 24.5.1

[-- Attachment #2: Type: text/html, Size: 2114 bytes --]

Re: BOM (byte order mark) in process stdout and stderr

[Eli Zaretskii]

> From: Jürgen Hötzel <juergen@archlinux.org>
> Date: Thu, 25 Jun 2015 20:37:41 +0200
> Cc: emacs-devel@gnu.org
> 
>     > (let ((default-process-coding-system '(utf-8-with-signature .
>     > utf-8-with-signature)))
>     > (start-file-process
>     > "BOM"
>     > (generate-new-buffer "*bom-test*")
>     > "~/tmp/Hello.exe"))
>     >
>     > This results in the following process-buffer (hexl-mode):
>     >
>     > 00000000: efbb bf53 5444 4f55 540a 5354 4445 5252 ...STDOUT.STDERR
>     > 00000010: 0a0a 5072 6f63 6573 7320 424f 4d20 6669 ..Process BOM fi
>     > 00000020: 6e69 7368 6564 0a nished.
>     >
>     > The stdout BOM was correctly removed but the stderr BOM is still present.
>     >
>     > According to
>     >
>     http://www.gnu.org/software/emacs/manual/html_node/elisp/Output-from-Processes.html
>    
>     >
>     > there is now way to handle stderr separately. I made this workaround for
>     the
>     > Emacs fsharp-mode:
>     >
>     >
>     https://github.com/juergenhoetzel/fsharpbinding/commit/d0d3600c38d0e9f47a621583d0fc82685feda196
>    
>     >
>     > Are there any better solutions?
>     
>     You didn't tell what version of Emacs do you want this to be solved.
>     
> 
> GNU Emacs 24.5.1

Then I think you will have to remove the second BOM in your
application's Lisp code.  start-file-process redirects both standard
output and standard error to the same file descriptor, and Emacs reads
their mix from that file descriptor.  It will remove 1 BOM, but the
other will be left alone.

Alternatively, separate the standard output and standard error via
shell redirection, and then read one of them separately from a file.