unofficial mirror of guile-user@gnu.org 
 help / color / mirror / Atom feed
* UTF16 encoding adds BOM everywhere?
@ 2022-07-14 19:16 Jean Abou Samra
  2022-07-20 20:42 ` Mark H Weaver
  0 siblings, 1 reply; 3+ messages in thread
From: Jean Abou Samra @ 2022-07-14 19:16 UTC (permalink / raw)
  To: Guile User

Hi,

With this code:

(let ((p (open-output-file "x.txt")))
   (set-port-encoding! p "UTF16")
   (display "ABC" p)
   (close-port p))

the sequence of bytes in the output file x.txt is

['FF', 'FE', '41', '0', 'FF', 'FE', '42', '0', 'FF', 'FE', '43', '0']

FFE is a little-endian Byte Order Mark (BOM), fine.
But why is Guile adding it before every character
instead of just at the beginning of the string?
Is that expected?

This is a curiosity question; since what I want is
big endian, I just used "UTF16BE", which outputs
big endian, and doesn't add the BOM -- I can then just
add a BOM manually. Still, I'm puzzled by this
behavior.

Thanks,
Jean




^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: UTF16 encoding adds BOM everywhere?
  2022-07-14 19:16 UTF16 encoding adds BOM everywhere? Jean Abou Samra
@ 2022-07-20 20:42 ` Mark H Weaver
  2022-07-20 21:45   ` Jean Abou Samra
  0 siblings, 1 reply; 3+ messages in thread
From: Mark H Weaver @ 2022-07-20 20:42 UTC (permalink / raw)
  To: Jean Abou Samra; +Cc: guile-user

Hi,

Jean Abou Samra <jean@abou-samra.fr> wrote:

> With this code:
> 
> (let ((p (open-output-file "x.txt")))
>    (set-port-encoding! p "UTF16")
>    (display "ABC" p)
>    (close-port p))
> 
> the sequence of bytes in the output file x.txt is
> 
> ['FF', 'FE', '41', '0', 'FF', 'FE', '42', '0', 'FF', 'FE', '43', '0']
> 
> FFE is a little-endian Byte Order Mark (BOM), fine.
> But why is Guile adding it before every character
> instead of just at the beginning of the string?
> Is that expected?

No, this is certainly a bug.  It sounds like the
'at_stream_start_for_bom_write' port flag is not being cleared, as it
should be, after the first character is written.  I suspect that it
worked correctly when I first implemented proper BOM handling in 2013
(commit cdd3d6c9f423d5b95f05193fe3c27d50b56957e9), but the ports code
has seen some major reworking since then.  I guess that BOM handling was
broken somewhere along the way.

I would suggest filing a bug report.  I don't have time to look into it,
sorry.  I don't work on Guile anymore.  I only happened to see your
message by chance.

     Regards,
       Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>.



^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: UTF16 encoding adds BOM everywhere?
  2022-07-20 20:42 ` Mark H Weaver
@ 2022-07-20 21:45   ` Jean Abou Samra
  0 siblings, 0 replies; 3+ messages in thread
From: Jean Abou Samra @ 2022-07-20 21:45 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guile-user

Hi Mark,

Thanks for your reply!


Le 20/07/2022 à 22:42, Mark H Weaver a écrit :
> Hi,
>
> No, this is certainly a bug.  It sounds like the
> 'at_stream_start_for_bom_write' port flag is not being cleared, as it
> should be, after the first character is written.  I suspect that it
> worked correctly when I first implemented proper BOM handling in 2013
> (commit cdd3d6c9f423d5b95f05193fe3c27d50b56957e9), but the ports code
> has seen some major reworking since then.  I guess that BOM handling was
> broken somewhere along the way.
>
> I would suggest filing a bug report.


OK, opened

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56675



> I don't have time to look into it, sorry.  I don't work on Guile anymore.

A pity :-)

Cheers,
Jean




^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-07-20 21:45 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-14 19:16 UTF16 encoding adds BOM everywhere? Jean Abou Samra
2022-07-20 20:42 ` Mark H Weaver
2022-07-20 21:45   ` Jean Abou Samra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).