Question: estimating size of a buffer when written to file?

Question: estimating size of a buffer when written to file?

[Qiantan Hong]

Is there any reliable way to get (or conservatively estimate) 
the size of a buffer when written to file?

I’m thinking about using it for an optimization for persistent kv store.
If we know a log entry, when written out, is shorter than PIPE_BUF,
then we can skip the file locking/unlocking.
Emacs lock file doesn’t seem to be particularly fast.

Best,
Qiantan

Re: Question: estimating size of a buffer when written to file?

[Stephen Berman]

On Wed, 8 Dec 2021 15:33:06 +0000 Qiantan Hong <qhong@mit.edu> wrote:

> Is there any reliable way to get (or conservatively estimate)
> the size of a buffer when written to file?

Is (string-bytes (buffer-string)) not reliable?

Steve Berman

Re: Question: estimating size of a buffer when written to file?

[Qiantan Hong]

> Is (string-bytes (buffer-string)) not reliable?
> 
> Steve Berman
I’m not very familiar with different kinds of encoding,
but would some coding system result in final
writes longer than internal string-bytes?

Best,
Qiantan

Re: Question: estimating size of a buffer when written to file?

[Stephen Berman]

On Wed, 8 Dec 2021 15:55:42 +0000 Qiantan Hong <qhong@mit.edu> wrote:

>> Is (string-bytes (buffer-string)) not reliable?
>> 
>> Steve Berman
> I’m not very familiar with different kinds of encoding,
> but would some coding system result in final
> writes longer than internal string-bytes?

I'm afraid I don't know either.

Steve Berman

> Best,
> Qiantan

Re: Question: estimating size of a buffer when written to file?

[Daniel Martín]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Wed, 8 Dec 2021 15:55:42 +0000 Qiantan Hong <qhong@mit.edu> wrote:
>
>>> Is (string-bytes (buffer-string)) not reliable?
>>> 
>>> Steve Berman
>> I’m not very familiar with different kinds of encoding,
>> but would some coding system result in final
>> writes longer than internal string-bytes?
>
> I'm afraid I don't know either.
>

Yes, if the coding system is not the UTF-8 variant used by Emacs
internally, the result won't be accurate for some scripts.  You need to
encode the contents of the buffer/string using the value of
buffer-file-coding-system, for example, and call string-bytes on it.

Re: Question: estimating size of a buffer when written to file?

[Stefan Monnier]

> Is there any reliable way to get (or conservatively estimate) 
> the size of a buffer when written to file?

How 'bout your first encode the buffer's content and then write it
instead of relying on the write to do the encoding?
This way you get to see the size trivially.

If you need this estimate faster (e.g. because you may end up not
writing the text at all depending on the result), then I think the only
way is to multiply the number of chars by the amplification factor that
is a property of the encoding (this is used in the C to allocate
a destination byte-array that's guaranteed to be large enough.  Not sure
if we currently expose it to ELisp but it would be easy to do).


        Stefan

Re: Question: estimating size of a buffer when written to file?

[Qiantan Hong]

> How 'bout your first encode the buffer's content and then write it
> instead of relying on the write to do the encoding?
> This way you get to see the size trivially.
I just found the function encode-coding-region, I never knew it exists, thanks!
The log will eventually gets written out so this is good enough.

BTW, after second thought, I think this is probably a premature optimization.

What about not care about locking at all in the kv-store APIs and let
package author do it?

On the other hand, I’m currently implementing a high level persistent-variable
API that runs in idle timer. I can easily lock/unlock only once every idle timer
event (instead of before/after every kv-put) and now the cost should be
negligible.