From: Eli Zaretskii <eliz@gnu.org>
To: Stefan Monnier <monnier@iro.umontreal.ca>
Cc: emacs-devel@gnu.org
Subject: Re: creating unibyte strings
Date: Fri, 22 Mar 2019 17:11:52 +0200 [thread overview]
Message-ID: <83sgvfq6yv.fsf@gnu.org> (raw)
In-Reply-To: <jwvh8bvf1kk.fsf-monnier+emacs@gnu.org> (message from Stefan Monnier on Fri, 22 Mar 2019 10:23:20 -0400)
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 22 Mar 2019 10:23:20 -0400
>
> >> I don't see what's subtle about "unibyte" strings, as long as you
> >> understand that these are strings of *bytes* instead of strings
> >> of *characters* (i.e. they're `int8[]` rather than `w_char_t[]`).
> >
> > That's the subtlety, right there. Handling such "strings" in Emacs
> > Lisp can produce strange and unexpected results for someone who is not
> > aware of the difference and its implications.
>
> But this has nothing to do with the modules API: it's not more tricky
> then when doing it purely in Elisp. Are you seriously suggesting we
> deprecate unibyte strings altogether?
We won't deprecate unibyte strings, but we decided long ago to
minimize their use.
> >> "Multibyte" strings are just as subtle (maybe more so even), yet we
> >> rightly don't hesitate to offer a primitive way to construct them.
> > Because we succeed to hide the subtleties in that case,
> > so the multibyte nature is not really visible on the Lisp level,
> > unless you try very hard to make it so.
>
> Then I don't know what subtleties you're talking about.
> Can you give some examples of the kinds of things you're thinking of?
String concatenation, for one. Regular expression search for another.
And those just the ones I thought about in the first 5 seconds.
> >> > Instead, how about doing that via vectors of byte values?
> >> What's the advantage? That seems even more convoluted: create a Lisp
> >> vector of the right size (i.e. 8x the size of your string on a 64bit
> >> system), loop over your string turning each byte into a Lisp integer
> >> (with the reverted API, this involves allocation of an `emacs_value`
> >> box), then pass that to `concat`?
> > That's one way, but I'm sure I can come up with a simpler one. ;-)
>
> I'm all ears.
Provide an Emacs primitive for that, then at least some of the
awkwardness is gone. And/or use records.
> >> It's probably going to be even less efficient than going through utf-8
> >> and back.
> > I doubt that. It's just an assignment. And it's a rare situation
> > anyway.
>
> Why do you think it's rare?
Because the number of Emacs features that require you to submit a
unibyte string is very small.
> It's pretty common to receive non-utf-8 byte streams from the external world.
> And when you do receive them, it can come at a very fast pace and become
> temporarily anything but rare.
Are you talking about text encoded in some non-UTF-8 encoding? If so,
it should be converted to UTF-8, and that will solve the problem. If
it isn't text, then what common use cases are you talking about?
> 2- the C side string contains text in latin-1, big5, younameit.
> The module API provides nothing convenient. Should we force our
> module to link to C-side coding-system libraries to convert to utf-8
> before passing it on to the Elisp, even though Emacs already has all
> the needed facilities? Really?
Yes, really. Why is that a problem? libiconv exists on every
platform we support, and is easy to use. Moreover, if you just want
to convert a native string into another native string, using Emacs
built-in en/decoding machinery is inconvenient, because it involves
more copying than necessary.
> 3- The C side string contains binary data (say PNG images).
> What does "arrange for it to be UTF-8" even mean?
Nothing, since in this case there's no meaning to "decoding".
next prev parent reply other threads:[~2019-03-22 15:11 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-11 18:12 Oddities with dynamic modules Eli Zaretskii
2018-10-12 14:29 ` Kaushal Modi
2019-02-10 20:23 ` Philipp Stephani
2019-02-11 15:45 ` Eli Zaretskii
2019-02-11 16:04 ` Yuri Khan
2019-03-21 20:04 ` Philipp Stephani
2019-03-21 20:17 ` Eli Zaretskii
2019-03-21 20:32 ` Philipp Stephani
2019-03-21 20:46 ` Eli Zaretskii
2019-03-21 20:51 ` Philipp Stephani
2019-03-21 20:12 ` Philipp Stephani
2019-03-21 20:25 ` Eli Zaretskii
2019-03-21 20:34 ` Philipp Stephani
2019-03-21 20:51 ` Eli Zaretskii
2019-03-21 20:58 ` Philipp Stephani
2019-03-22 1:26 ` creating unibyte strings (was: Oddities with dynamic modules) Stefan Monnier
2019-03-22 7:41 ` Eli Zaretskii
2019-03-22 12:33 ` creating unibyte strings Stefan Monnier
2019-03-22 13:27 ` Eli Zaretskii
2019-03-22 14:23 ` Stefan Monnier
2019-03-22 15:11 ` Eli Zaretskii [this message]
2019-03-22 15:37 ` Stefan Monnier
2019-03-22 15:54 ` Eli Zaretskii
2019-03-24 14:51 ` Elias Mårtenson
2019-03-24 17:10 ` Eli Zaretskii
2019-03-25 1:47 ` Elias Mårtenson
2019-03-25 3:41 ` Eli Zaretskii
2019-03-26 10:23 ` Elias Mårtenson
2019-03-26 11:12 ` Stefan Monnier
2019-03-22 8:20 ` Oddities with dynamic modules Eli Zaretskii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=83sgvfq6yv.fsf@gnu.org \
--to=eliz@gnu.org \
--cc=emacs-devel@gnu.org \
--cc=monnier@iro.umontreal.ca \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.