string-bytes and coding systems

string-bytes and coding systems

[Eric Abrahamsen]

I'm writing a function that's supposed to wrap too-long text lines; the
RFC says anything over 75 octets (excluding eol) needs to be wrapped,
but multibyte characters must not be split.

Everything seems to be working fine, but I want to make sure I'm not
making any dangerous assumptions about `string-bytes' and encoding.

I'm essentially taking the `string-bytes' of each line, and if it's too
long, popping characters off the end until it's fewer than 75 bytes.

My understanding/assumption is that `string-bytes' returns the number of
bytes according to Emacs' internal coding system, which is close enough
to utf-8 to make no difference. When this text gets written to file it
will also be encoded as utf-8, ergo testing string lengths with
`string-bytes' is going to always produce the right results in the final
file.

Have I understood things correctly?

Thanks!
Eric

Re: string-bytes and coding systems

[hector]

On Wed, Mar 08, 2017 at 03:17:07PM -0800, Eric Abrahamsen wrote:
> I'm writing a function that's supposed to wrap too-long text lines; the
> RFC says anything over 75 octets (excluding eol) needs to be wrapped,
> but multibyte characters must not be split.

Why don't you just use fill-paragraph?

Re: string-bytes and coding systems

[Yuri Khan]

On Thu, Mar 9, 2017 at 2:46 PM, hector <hectorlahoz@gmail.com> wrote:
> On Wed, Mar 08, 2017 at 03:17:07PM -0800, Eric Abrahamsen wrote:
>> I'm writing a function that's supposed to wrap too-long text lines; the
>> RFC says anything over 75 octets (excluding eol) needs to be wrapped,
>> but multibyte characters must not be split.
>
> Why don't you just use fill-paragraph?

Because that works in terms of characters, not octets?

Re: string-bytes and coding systems

[hector]

On Thu, Mar 09, 2017 at 02:54:16PM +0700, Yuri Khan wrote:
> On Thu, Mar 9, 2017 at 2:46 PM, hector <hectorlahoz@gmail.com> wrote:
> > On Wed, Mar 08, 2017 at 03:17:07PM -0800, Eric Abrahamsen wrote:
> >> I'm writing a function that's supposed to wrap too-long text lines; the
> >> RFC says anything over 75 octets (excluding eol) needs to be wrapped,
> >> but multibyte characters must not be split.
> >
> > Why don't you just use fill-paragraph?
> 
> Because that works in terms of characters, not octets?

Right.
It would be weird for a user command to work in terms of octets, wouldn't it?

Re: string-bytes and coding systems

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Wed, 08 Mar 2017 15:17:07 -0800
> 
> I'm essentially taking the `string-bytes' of each line, and if it's too
> long, popping characters off the end until it's fewer than 75 bytes.
> 
> My understanding/assumption is that `string-bytes' returns the number of
> bytes according to Emacs' internal coding system

Yes.

> which is close enough to utf-8 to make no difference.

No.  The deviations from UTF-8 could be significant in some cases,
with some exotic characters and with raw bytes.

> When this text gets written to file it will also be encoded as
> utf-8, ergo testing string lengths with `string-bytes' is going to
> always produce the right results in the final file.

I suggest to use filepos-to-bufferpos to find where to break text into
lines.

Re: string-bytes and coding systems

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Wed, 08 Mar 2017 15:17:07 -0800
>> 
>> I'm essentially taking the `string-bytes' of each line, and if it's too
>> long, popping characters off the end until it's fewer than 75 bytes.
>> 
>> My understanding/assumption is that `string-bytes' returns the number of
>> bytes according to Emacs' internal coding system
>
> Yes.
>
>> which is close enough to utf-8 to make no difference.
>
> No.  The deviations from UTF-8 could be significant in some cases,
> with some exotic characters and with raw bytes.

Good to know.

>> When this text gets written to file it will also be encoded as
>> utf-8, ergo testing string lengths with `string-bytes' is going to
>> always produce the right results in the final file.
>
> I suggest to use filepos-to-bufferpos to find where to break text into
> lines.

I'll look into that. Thank you!

Eric

Re: string-bytes and coding systems

[Eric Abrahamsen]

hector <hectorlahoz@gmail.com> writes:

> On Thu, Mar 09, 2017 at 02:54:16PM +0700, Yuri Khan wrote:
>> On Thu, Mar 9, 2017 at 2:46 PM, hector <hectorlahoz@gmail.com> wrote:
>> > On Wed, Mar 08, 2017 at 03:17:07PM -0800, Eric Abrahamsen wrote:
>> >> I'm writing a function that's supposed to wrap too-long text lines; the
>> >> RFC says anything over 75 octets (excluding eol) needs to be wrapped,
>> >> but multibyte characters must not be split.
>> >
>> > Why don't you just use fill-paragraph?
>> 
>> Because that works in terms of characters, not octets?
>
> Right.
> It would be weird for a user command to work in terms of octets, wouldn't it?

It's not really a user command, I'm exporting vCard objects to a file.

Re: string-bytes and coding systems

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]


() Eric Abrahamsen <eric@ericabrahamsen.net>
() Thu, 09 Mar 2017 09:36:03 -0800

   It's not really a user command, I'm exporting vCard objects
   to a file.

I'm interested in having bindat.el exercised more, hence the
tangential question: Do you think this could be implemented
using bindat.el?

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: string-bytes and coding systems

[Alexis]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> It's not really a user command, I'm exporting vCard objects to a file.

Might my `org-vcard` package be of any help to you?

https://github.com/flexibeast/org-vcard

It very much needs refactoring to separate out the vCard
input-and-output stuff from everything else - i've been struggling to
find the time and energy to do so - but maybe you'll find some of the
machinery useful?


Alexis.

Re: string-bytes and coding systems

[Eric Abrahamsen]

Alexis <flexibeast@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> It's not really a user command, I'm exporting vCard objects to a file.
>
> Might my `org-vcard` package be of any help to you?
>
> https://github.com/flexibeast/org-vcard
>
> It very much needs refactoring to separate out the vCard
> input-and-output stuff from everything else - i've been struggling to
> find the time and energy to do so - but maybe you'll find some of the
> machinery useful?

Yes! I didn't know this was out there -- it's always good to see other
people's approach to the same problems. Thanks!

It's true your package is closely tied to Org mode. I'm sure I'll still
be able to take some pointers from it, though. Thank god I'm not trying
to support vCard 2.1.

Here's where I'm at now. It does the escaping, but the line folding
(while implemented) isn't actually called; I'm still chewing on Eli's
pointers.

https://github.com/girzel/ebdb/blob/master/ebdb-vcard.el

Eric

Re: string-bytes and coding systems

[Eric Abrahamsen]

Thien-Thi Nguyen <ttn@gnu.org> writes:

> () Eric Abrahamsen <eric@ericabrahamsen.net>
> () Thu, 09 Mar 2017 09:36:03 -0800
>
>    It's not really a user command, I'm exporting vCard objects
>    to a file.
>
> I'm interested in having bindat.el exercised more, hence the
> tangential question: Do you think this could be implemented
> using bindat.el?

I'm ashamed to say that I understand binary only enough to see why I
have problems with encoding. I aspire to someday use `logior' in anger.
I don't understand this library!

Re: string-bytes and coding systems

[Stefan Monnier]

>> When this text gets written to file it will also be encoded as
>> utf-8, ergo testing string lengths with `string-bytes' is going to
>> always produce the right results in the final file.
> I suggest to use filepos-to-bufferpos to find where to break text into
> lines.

BTW, filepos-to-bufferpos uses string-bytes (or equivalent data, with
the same caveat for raw bytes) for utf-8 buffers ;-)


        Stefan

Re: string-bytes and coding systems

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> When this text gets written to file it will also be encoded as
>>> utf-8, ergo testing string lengths with `string-bytes' is going to
>>> always produce the right results in the final file.
>> I suggest to use filepos-to-bufferpos to find where to break text into
>> lines.
>
> BTW, filepos-to-bufferpos uses string-bytes (or equivalent data, with
> the same caveat for raw bytes) for utf-8 buffers ;-)

Hmm, maybe it will be bindat.el after all! Or I'll just keep the current
in-buffer string-bytes approach and wait for something to explode.

Re: string-bytes and coding systems

[Stefan Monnier]

> Hmm, maybe it will be bindat.el after all! Or I'll just keep the current
> in-buffer string-bytes approach and wait for something to explode.

Basically, as long as the data is truly utf-8, string-bytes should give
you the correct answer.  The differences should only occur on characters
which are outside of utf-8 proper (used to represent invalid utf-8 data).


        Stefan

Re: string-bytes and coding systems

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Hmm, maybe it will be bindat.el after all! Or I'll just keep the current
>> in-buffer string-bytes approach and wait for something to explode.
>
> Basically, as long as the data is truly utf-8, string-bytes should give
> you the correct answer.  The differences should only occur on characters
> which are outside of utf-8 proper (used to represent invalid utf-8 data).

Sounds close enough for me.

Re: string-bytes and coding systems

[Stefan Monnier]

>>> Hmm, maybe it will be bindat.el after all! Or I'll just keep the current
>>> in-buffer string-bytes approach and wait for something to explode.
>> Basically, as long as the data is truly utf-8, string-bytes should give
>> you the correct answer.  The differences should only occur on characters
>> which are outside of utf-8 proper (used to represent invalid utf-8 data).
> Sounds close enough for me.

That's also my opinion.


        Stefan