Oddities with dynamic modules

Oddities with dynamic modules

[Eli Zaretskii]

Having written the documentation of the module API, I couldn't help
but notice a few oddities about its repertory.  I list below the
issues that caused me to raise a brow, for the record:

 . Why do we have functions to access vectors, but not functions to
   access lists?  I always thought lists were more important for Emacs
   than vectors.  If we are asking users to use 'intern' to access
   'car' and 'cdr', why not 'aref' and 'aset'?

 . Why aren't there API functions to _create_ lists and vectors?

 . Using 'funcall' is unnecessarily cumbersome, because the function
   to be called is passed as an 'emacs_value'.  Why don't we have a
   variant that just accepts a name of a Lisp-callable function as a C
   string?

 . Why does 'intern' only handle pure ASCII symbol names?  It's not
   like supporting non-ASCII names is hard.

 . I could understand why equality predicates are not provided in the
   API, but I don't understand why we do provide 'eq' there.  Is it
   that much more important than the other predicates?

IOW, if the API was supposed to be minimal, it looks like it isn't;
and if it wasn't supposed to be minimal, then some important/popular
functions are strangely missing, for reasons I couldn't wrap my head
around.

Thanks.

Re: Oddities with dynamic modules

[Kaushal Modi]

On Thu, Oct 11, 2018 at 2:14 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Having written the documentation of the module API,

Thanks for writing up all that documentation!
https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=emacs-26&id=ce8b4584a3c69e5c4abad8a0a9c3781ce8c0c1f8

I am not in the capacity to comment on most of your questions as I am
using the Modules feature to get around my lack of C knowledge :)

>  . Why aren't there API functions to _create_ lists and vectors?
>
>  . Using 'funcall' is unnecessarily cumbersome, because the function
>    to be called is passed as an 'emacs_value'.  Why don't we have a
>    variant that just accepts a name of a Lisp-callable function as a C
>    string?

+1

I needed to create some sugar syntax in Nim (which compiles to C) to
get around that limitation:

proc MakeList*(env: ptr emacs_env; listArray: openArray[emacs_value]):
emacs_value =
## Return an Emacs-Lisp ``list``.
Funcall(env, "list", listArray)


proc MakeCons*(env: ptr emacs_env; consCar, consCdr: emacs_value): emacs_value =
## Return an Emacs-Lisp ``cons``.
Funcall(env, "cons", [consCar, consCdr])

It would be nice to have API for list (and cons).

>  . Why does 'intern' only handle pure ASCII symbol names?  It's not
>    like supporting non-ASCII names is hard.
>
>  . I could understand why equality predicates are not provided in the
>    API, but I don't understand why we do provide 'eq' there.  Is it
>    that much more important than the other predicates?

I had the same question too. I find equal more useful than eq.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Do., 11. Okt. 2018 um 20:13 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> Having written the documentation of the module API, I couldn't help
> but notice a few oddities about its repertory.  I list below the
> issues that caused me to raise a brow, for the record:
>
>  . Why do we have functions to access vectors, but not functions to
>    access lists?  I always thought lists were more important for Emacs
>    than vectors.  If we are asking users to use 'intern' to access
>    'car' and 'cdr', why not 'aref' and 'aset'?
>
>  . Why aren't there API functions to _create_ lists and vectors?

I guess these are mostly historical. These were introduced in
https://github.com/aaptel/emacs-dynamic-module/commit/016e8b6ffdfb861806957bb84c419a3d65caedb7,
but I don't remember the background.

>
>  . Using 'funcall' is unnecessarily cumbersome, because the function
>    to be called is passed as an 'emacs_value'.  Why don't we have a
>    variant that just accepts a name of a Lisp-callable function as a C
>    string?

Convenience is not a design goal of the module API. The primary design
goals are robustness, stability, simplicity, and minimalism.

>
>  . Why does 'intern' only handle pure ASCII symbol names?  It's not
>    like supporting non-ASCII names is hard.

Unfortunately it is, due to Emacs underspecifying encoding. If we can
manage to write an 'intern' function that accepts UTF-8 strings and
only UTF-8 strings, I'm all for it.

>
>  . I could understand why equality predicates are not provided in the
>    API, but I don't understand why we do provide 'eq' there.  Is it
>    that much more important than the other predicates?

Yes, it represents a fundamental property of objects.

>
> IOW, if the API was supposed to be minimal, it looks like it isn't;
> and if it wasn't supposed to be minimal, then some important/popular
> functions are strangely missing, for reasons I couldn't wrap my head
> around.

It is *mostly* minimal. A *completely* minimal API would not even have
integer and floating-point conversion functions, as those can be
written using the string functions. But that would be far less simple
and robust.
"eq" and "is_not_nil" are special in that they implement access to
fundamental object properties and can't fail, so they are fundamental
enough to deserve an entry in the module table.

The best source to answer the "why" questions is still the original
design document:
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 10 Feb 2019 21:23:18 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> >  . Using 'funcall' is unnecessarily cumbersome, because the function
> >    to be called is passed as an 'emacs_value'.  Why don't we have a
> >    variant that just accepts a name of a Lisp-callable function as a C
> >    string?
> 
> Convenience is not a design goal of the module API. The primary design
> goals are robustness, stability, simplicity, and minimalism.

I thought simplicity and convenience of use tramps simplicity of the
implementation, so it is strange to read arguments to the contrary.
IME, inconvenient interfaces are the main reason for them being
unstable, but that's me.

> >  . Why does 'intern' only handle pure ASCII symbol names?  It's not
> >    like supporting non-ASCII names is hard.
> 
> Unfortunately it is, due to Emacs underspecifying encoding. If we can
> manage to write an 'intern' function that accepts UTF-8 strings and
> only UTF-8 strings, I'm all for it.

What are the problems you have in mind?  After all, this is already
possible by means of 2 more function calls, as the example in the
manual shows.  Are there any problems to do the same under the hood,
instead of requiring users to do that explicitly in their module code?

> >  . I could understand why equality predicates are not provided in the
> >    API, but I don't understand why we do provide 'eq' there.  Is it
> >    that much more important than the other predicates?
> 
> Yes, it represents a fundamental property of objects.

How is that relevant?  Equality predicates are used very frequently
when dealing with Lisp objects; 'eq' is not different from others in
that respect.

> > IOW, if the API was supposed to be minimal, it looks like it isn't;
> > and if it wasn't supposed to be minimal, then some important/popular
> > functions are strangely missing, for reasons I couldn't wrap my head
> > around.
> 
> It is *mostly* minimal. A *completely* minimal API would not even have
> integer and floating-point conversion functions, as those can be
> written using the string functions. But that would be far less simple
> and robust.
> "eq" and "is_not_nil" are special in that they implement access to
> fundamental object properties and can't fail, so they are fundamental
> enough to deserve an entry in the module table.

I cannot follow this reasoning, sorry.  It sounds like you are saying
that the decision what to implement and what not justifies itself
because it's there.  All I can say is that as someone who wrote a
couple of lines of code in Emacs, the stuff that is in the API and the
omissions look quite arbitrary to me.

> The best source to answer the "why" questions is still the original
> design document:
> https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html

Which part(s) of that long document answer these questions, please?

Re: Oddities with dynamic modules

[Yuri Khan]

On Mon, Feb 11, 2019 at 10:46 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Convenience is not a design goal of the module API. The primary design
> > goals are robustness, stability, simplicity, and minimalism.
>
> I thought simplicity and convenience of use tramps simplicity of the
> implementation, so it is strange to read arguments to the contrary.
> IME, inconvenient interfaces are the main reason for them being
> unstable, but that's me.

A good middle ground is a minimalistic (but sufficient) API at the
host side, plus idiomatic convenience wrapper libraries for each
client language. (The latter need not be maintained by the host API
maintainer.)

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Mo., 11. Feb. 2019 um 17:04 Uhr schrieb Yuri Khan <yurivkhan@gmail.com>:
>
> On Mon, Feb 11, 2019 at 10:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > > Convenience is not a design goal of the module API. The primary design
> > > goals are robustness, stability, simplicity, and minimalism.
> >
> > I thought simplicity and convenience of use tramps simplicity of the
> > implementation, so it is strange to read arguments to the contrary.
> > IME, inconvenient interfaces are the main reason for them being
> > unstable, but that's me.
>
> A good middle ground is a minimalistic (but sufficient) API at the
> host side, plus idiomatic convenience wrapper libraries for each
> client language. (The latter need not be maintained by the host API
> maintainer.)

Yes, that's exactly what Daniel suggested in his initial design. It's
also what largely seems to be happening: people have written idiomatic
wrappers for Rust, Go, and probably other languages. Thus I'd say this
is working as inteded.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Mo., 11. Feb. 2019 um 16:46 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 10 Feb 2019 21:23:18 +0100
> > Cc: Emacs developers <emacs-devel@gnu.org>
> >
> > >  . Using 'funcall' is unnecessarily cumbersome, because the function
> > >    to be called is passed as an 'emacs_value'.  Why don't we have a
> > >    variant that just accepts a name of a Lisp-callable function as a C
> > >    string?
> >
> > Convenience is not a design goal of the module API. The primary design
> > goals are robustness, stability, simplicity, and minimalism.
>
> I thought simplicity and convenience of use tramps simplicity of the
> implementation, so it is strange to read arguments to the contrary.

Simplicity tends to be the opposite of convenience. Interface
simplicity definitely trumps convenience and implementation
simplicity.

>
> > >  . Why does 'intern' only handle pure ASCII symbol names?  It's not
> > >    like supporting non-ASCII names is hard.
> >
> > Unfortunately it is, due to Emacs underspecifying encoding. If we can
> > manage to write an 'intern' function that accepts UTF-8 strings and
> > only UTF-8 strings, I'm all for it.
>
> What are the problems you have in mind?  After all, this is already
> possible by means of 2 more function calls, as the example in the
> manual shows.  Are there any problems to do the same under the hood,
> instead of requiring users to do that explicitly in their module code?

If users want to have a truly generic "intern" function, they need to
do some legwork anyway because the signature of the "intern"
environment function doesn't allow embedded null bytes. It's also
unclear how non-Unicode symbols should be represented (if at all).
Given that almost all uses of "intern" will use ASCII symbols, it's
fine to restrict the API in this way. I've provided an example for a
wrapper function that allows at least interning of arbitrary Unicode
strings: https://phst.eu/emacs-modules#intern.

>
> > >  . I could understand why equality predicates are not provided in the
> > >    API, but I don't understand why we do provide 'eq' there.  Is it
> > >    that much more important than the other predicates?
> >
> > Yes, it represents a fundamental property of objects.
>
> How is that relevant?  Equality predicates are used very frequently
> when dealing with Lisp objects; 'eq' is not different from others in
> that respect.

I don't recollect the reasoning, but Daniel stated that "eq" is
strictly necessary, so you might want to ask him.

>
> > > IOW, if the API was supposed to be minimal, it looks like it isn't;
> > > and if it wasn't supposed to be minimal, then some important/popular
> > > functions are strangely missing, for reasons I couldn't wrap my head
> > > around.
> >
> > It is *mostly* minimal. A *completely* minimal API would not even have
> > integer and floating-point conversion functions, as those can be
> > written using the string functions. But that would be far less simple
> > and robust.
> > "eq" and "is_not_nil" are special in that they implement access to
> > fundamental object properties and can't fail, so they are fundamental
> > enough to deserve an entry in the module table.
>
> I cannot follow this reasoning, sorry.  It sounds like you are saying
> that the decision what to implement and what not justifies itself
> because it's there.  All I can say is that as someone who wrote a
> couple of lines of code in Emacs, the stuff that is in the API and the
> omissions look quite arbitrary to me.

Please see Daniel's original reasoning for the design. I'm not
claiming perfection for the module API, but it definitely strikes a
very good balance between minimalism and robustness.

>
> > The best source to answer the "why" questions is still the original
> > design document:
> > https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html
>
> Which part(s) of that long document answer these questions, please?

Mostly the first sentence "We want an ABI powerful enough to let C
modules interact with Emacs, but decoupled enough to let the Emacs
core evolve independently."

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 21 Mar 2019 21:04:07 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>
> 
> > > I thought simplicity and convenience of use tramps simplicity of the
> > > implementation, so it is strange to read arguments to the contrary.
> > > IME, inconvenient interfaces are the main reason for them being
> > > unstable, but that's me.
> >
> > A good middle ground is a minimalistic (but sufficient) API at the
> > host side, plus idiomatic convenience wrapper libraries for each
> > client language. (The latter need not be maintained by the host API
> > maintainer.)
> 
> Yes, that's exactly what Daniel suggested in his initial design. It's
> also what largely seems to be happening: people have written idiomatic
> wrappers for Rust, Go, and probably other languages.

Where's the wrapper for C?

> Thus I'd say this is working as inteded.

I still find it strange (and can already hardly remember what I wrote
several months ago, so perhaps try to respond sooner next time?).

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 21 Mar 2019 21:12:05 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > How is that relevant?  Equality predicates are used very frequently
> > when dealing with Lisp objects; 'eq' is not different from others in
> > that respect.
> 
> I don't recollect the reasoning, but Daniel stated that "eq" is
> strictly necessary, so you might want to ask him.
> [...]
> > > It is *mostly* minimal. A *completely* minimal API would not even have
> > > integer and floating-point conversion functions, as those can be
> > > written using the string functions. But that would be far less simple
> > > and robust.
> > > "eq" and "is_not_nil" are special in that they implement access to
> > > fundamental object properties and can't fail, so they are fundamental
> > > enough to deserve an entry in the module table.
> >
> > I cannot follow this reasoning, sorry.  It sounds like you are saying
> > that the decision what to implement and what not justifies itself
> > because it's there.  All I can say is that as someone who wrote a
> > couple of lines of code in Emacs, the stuff that is in the API and the
> > omissions look quite arbitrary to me.
> 
> Please see Daniel's original reasoning for the design.

I asked the questions after reading that, so please believe me that I
didn't find answers to my questions there.

> Mostly the first sentence "We want an ABI powerful enough to let C
> modules interact with Emacs, but decoupled enough to let the Emacs
> core evolve independently."

That means a judgment call, and I was questioning the judgment.
Saying that someone made a call doesn't explain why the decision was
what it was.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Do., 21. März 2019 um 21:17 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 21 Mar 2019 21:04:07 +0100
> > Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>
> >
> > > > I thought simplicity and convenience of use tramps simplicity of the
> > > > implementation, so it is strange to read arguments to the contrary.
> > > > IME, inconvenient interfaces are the main reason for them being
> > > > unstable, but that's me.
> > >
> > > A good middle ground is a minimalistic (but sufficient) API at the
> > > host side, plus idiomatic convenience wrapper libraries for each
> > > client language. (The latter need not be maintained by the host API
> > > maintainer.)
> >
> > Yes, that's exactly what Daniel suggested in his initial design. It's
> > also what largely seems to be happening: people have written idiomatic
> > wrappers for Rust, Go, and probably other languages.
>
> Where's the wrapper for C?

It seems like there isn't enough interest in modules written in C for
such a wrapper. Or there is a wrapper and I just don't know about it.
The module API is usable as-is, and writing and maintaining a wrapper
is nontrivial work. For other languages wrappers are necessary, but C
users can use the plain API directly, and that's often a reasonable
choice. For example, the FFI module
(https://github.com/tromey/emacs-ffi/blob/master/ffi-module.c) is
complex enough that a wrapper library probably wouldn't make it much
shorter.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Do., 21. März 2019 um 21:25 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Mostly the first sentence "We want an ABI powerful enough to let C
> > modules interact with Emacs, but decoupled enough to let the Emacs
> > core evolve independently."
>
> That means a judgment call, and I was questioning the judgment.
> Saying that someone made a call doesn't explain why the decision was
> what it was.
>

Well, feel free to CC Daniel and ask him directly. I agree with his
judgment call, but can't know what he might have been thinking
exactly.
TBH I haven't spent that much time on trying to figure out the exact
reasoning because potentially superfluous functions can't be removed
any more, so this question seems to be of largely historical interest.

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 21 Mar 2019 21:32:07 +0100
> Cc: Yuri Khan <yurivkhan@gmail.com>, Emacs developers <emacs-devel@gnu.org>
> 
> > > Yes, that's exactly what Daniel suggested in his initial design. It's
> > > also what largely seems to be happening: people have written idiomatic
> > > wrappers for Rust, Go, and probably other languages.
> >
> > Where's the wrapper for C?
> 
> It seems like there isn't enough interest in modules written in C for
> such a wrapper.

I think if we maintain that this is the job for a wrapper, we should
have such a wrapper in Emacs.

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 21 Mar 2019 21:34:39 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> potentially superfluous functions can't be removed any more, so this
> question seems to be of largely historical interest.

No, it's not only of historical interest, because we can add
functions.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Do., 21. März 2019 um 21:46 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 21 Mar 2019 21:32:07 +0100
> > Cc: Yuri Khan <yurivkhan@gmail.com>, Emacs developers <emacs-devel@gnu.org>
> >
> > > > Yes, that's exactly what Daniel suggested in his initial design. It's
> > > > also what largely seems to be happening: people have written idiomatic
> > > > wrappers for Rust, Go, and probably other languages.
> > >
> > > Where's the wrapper for C?
> >
> > It seems like there isn't enough interest in modules written in C for
> > such a wrapper.
>
> I think if we maintain that this is the job for a wrapper, we should
> have such a wrapper in Emacs.

I certainly wouldn't mind having such a wrapper in Emacs core.

Re: Oddities with dynamic modules

[Philipp Stephani]

Am Do., 21. März 2019 um 21:51 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 21 Mar 2019 21:34:39 +0100
> > Cc: Emacs developers <emacs-devel@gnu.org>
> >
> > potentially superfluous functions can't be removed any more, so this
> > question seems to be of largely historical interest.
>
> No, it's not only of historical interest, because we can add
> functions.

That's true, and I agree for those we should find some clearer critera
than "best judgment" or philosophical vague principles like
"simplicity."
Each addition should be discussed separately. Possible criteria could be:
1. Is it possible to obtain the functionality by calling existing
functions? ("completeness")
2. Is it very difficult to replicate the functionality with the
existing API, and the difficulty would be reduced significantly by
introducing a new function? ("simplicity")
3. Is there a huge performance benefit in introducing a specialized function?
If none of these are fulfilled, then the function should probably not
be added ("simplicity").
For example, I'd vote for adding timespec and bignum conversion
functions based on (2).

creating unibyte strings (was: Oddities with dynamic modules)

[Stefan Monnier]

>> > potentially superfluous functions can't be removed any more, so this
>> > question seems to be of largely historical interest.
>> No, it's not only of historical interest, because we can add
>> functions.

Which reminds me: could someone add to the module API a primitive to
build a *unibyte* string?  Currently it seems the only way to do that is
by building a multibyte string and then encoding it with utf-8, which is
both inefficient and risky (I know it's supposed to return exactly the
originally intended bytes, but it's a very round about way to do it
with lots of opportunity for bugs along the way).


        Stefan

Re: creating unibyte strings (was: Oddities with dynamic modules)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 21 Mar 2019 21:26:32 -0400
> 
> Which reminds me: could someone add to the module API a primitive to
> build a *unibyte* string?

I don't like adding such a primitive.  We don't want to proliferate
unibyte strings in Emacs through that back door, because manipulating
unibyte strings involves subtle issues many Lisp programmers are not
aware of.

Instead, how about doing that via vectors of byte values?  If there
are Emacs primitives that currently only accept strings, we could
extend them.

Re: Oddities with dynamic modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 21 Mar 2019 21:58:11 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > No, it's not only of historical interest, because we can add
> > functions.
> 
> That's true, and I agree for those we should find some clearer critera
> than "best judgment" or philosophical vague principles like
> "simplicity."
> [...]
> For example, I'd vote for adding timespec and bignum conversion
> functions based on (2).

I agree with the last proposal.

But I also think that we need a lot more convenience wrappers even for
existing functionalities.  Any non-trivial module whose code I ever
saw is a clear evidence to that, as they all introduce their own
wrappers for practically the same purposes.

Re: creating unibyte strings

[Stefan Monnier]

>> Which reminds me: could someone add to the module API a primitive to
>> build a *unibyte* string?
> I don't like adding such a primitive.  We don't want to proliferate
> unibyte strings in Emacs through that back door, because manipulating
> unibyte strings involves subtle issues many Lisp programmers are not
> aware of.

I don't see what's subtle about "unibyte" strings, as long as you
understand that these are strings of *bytes* instead of strings
of *characters* (i.e. they're `int8[]` rather than `w_char_t[]`).

"Multibyte" strings are just as subtle (maybe more so even), yet we
rightly don't hesitate to offer a primitive way to construct them.

> Instead, how about doing that via vectors of byte values?

What's the advantage?  That seems even more convoluted: create a Lisp
vector of the right size (i.e. 8x the size of your string on a 64bit
system), loop over your string turning each byte into a Lisp integer
(with the reverted API, this involves allocation of an `emacs_value`
box), then pass that to `concat`?

It's probably going to be even less efficient than going through utf-8
and back.

Think about cases where the module receives byte strings from the disk
or the network and need to pass that to `decode-coding-string`.
And consider that we might be talking about megabytes of strings.


        Stefan

Re: creating unibyte strings

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Mar 2019 08:33:02 -0400
> 
> >> Which reminds me: could someone add to the module API a primitive to
> >> build a *unibyte* string?
> > I don't like adding such a primitive.  We don't want to proliferate
> > unibyte strings in Emacs through that back door, because manipulating
> > unibyte strings involves subtle issues many Lisp programmers are not
> > aware of.
> 
> I don't see what's subtle about "unibyte" strings, as long as you
> understand that these are strings of *bytes* instead of strings
> of *characters* (i.e. they're `int8[]` rather than `w_char_t[]`).

That's the subtlety, right there.  Handling such "strings" in Emacs
Lisp can produce strange and unexpected results for someone who is not
aware of the difference and its implications.

> "Multibyte" strings are just as subtle (maybe more so even), yet we
> rightly don't hesitate to offer a primitive way to construct them.

Because we succeed to hide the subtleties in that case, so the
multibyte nature is not really visible on the Lisp level, unless you
try very hard to make it so.

> > Instead, how about doing that via vectors of byte values?
> 
> What's the advantage?  That seems even more convoluted: create a Lisp
> vector of the right size (i.e. 8x the size of your string on a 64bit
> system), loop over your string turning each byte into a Lisp integer
> (with the reverted API, this involves allocation of an `emacs_value`
> box), then pass that to `concat`?

That's one way, but I'm sure I can come up with a simpler one. ;-)

> It's probably going to be even less efficient than going through utf-8
> and back.

I doubt that.  It's just an assignment.  And it's a rare situation
anyway.

> Think about cases where the module receives byte strings from the disk
> or the network and need to pass that to `decode-coding-string`.
> And consider that we might be talking about megabytes of strings.

They don't need to decode, they just need to arrange for it to be
UTF-8.

Re: creating unibyte strings

[Stefan Monnier]

>> >> Which reminds me: could someone add to the module API a primitive to
>> >> build a *unibyte* string?
>> > I don't like adding such a primitive.  We don't want to proliferate
>> > unibyte strings in Emacs through that back door, because manipulating
>> > unibyte strings involves subtle issues many Lisp programmers are not
>> > aware of.
>> 
>> I don't see what's subtle about "unibyte" strings, as long as you
>> understand that these are strings of *bytes* instead of strings
>> of *characters* (i.e. they're `int8[]` rather than `w_char_t[]`).
>
> That's the subtlety, right there.  Handling such "strings" in Emacs
> Lisp can produce strange and unexpected results for someone who is not
> aware of the difference and its implications.

But this has nothing to do with the modules API: it's not more tricky
then when doing it purely in Elisp.  Are you seriously suggesting we
deprecate unibyte strings altogether?

>> "Multibyte" strings are just as subtle (maybe more so even), yet we
>> rightly don't hesitate to offer a primitive way to construct them.
> Because we succeed to hide the subtleties in that case,
> so the multibyte nature is not really visible on the Lisp level,
> unless you try very hard to make it so.

Then I don't know what subtleties you're talking about.
Can you give some examples of the kinds of things you're thinking of?

>> > Instead, how about doing that via vectors of byte values?
>> What's the advantage?  That seems even more convoluted: create a Lisp
>> vector of the right size (i.e. 8x the size of your string on a 64bit
>> system), loop over your string turning each byte into a Lisp integer
>> (with the reverted API, this involves allocation of an `emacs_value`
>> box), then pass that to `concat`?
> That's one way, but I'm sure I can come up with a simpler one. ;-)

I'm all ears.

>> It's probably going to be even less efficient than going through utf-8
>> and back.
> I doubt that.  It's just an assignment.  And it's a rare situation
> anyway.

Why do you think it's rare?
It's pretty common to receive non-utf-8 byte streams from the external world.
And when you do receive them, it can come at a very fast pace and become
temporarily anything but rare.

>> Think about cases where the module receives byte strings from the disk
>> or the network and need to pass that to `decode-coding-string`.
>> And consider that we might be talking about megabytes of strings.
> They don't need to decode, they just need to arrange for it to be
> UTF-8.

Three possibilities:
1- the C side string contains utf-8 text.
   The module API provides just the right operation, we're good to go.
2- the C side string contains text in latin-1, big5, younameit.
   The module API provides nothing convenient.  Should we force our
   module to link to C-side coding-system libraries to convert to utf-8
   before passing it on to the Elisp, even though Emacs already has all
   the needed facilities?  Really?
3- The C side string contains binary data (say PNG images).
   What does "arrange for it to be UTF-8" even mean?


-- Stefan


PS: The PNG case is not hypothetical at all, it's what prompted my request.

Re: creating unibyte strings

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 22 Mar 2019 10:23:20 -0400
> 
> >> I don't see what's subtle about "unibyte" strings, as long as you
> >> understand that these are strings of *bytes* instead of strings
> >> of *characters* (i.e. they're `int8[]` rather than `w_char_t[]`).
> >
> > That's the subtlety, right there.  Handling such "strings" in Emacs
> > Lisp can produce strange and unexpected results for someone who is not
> > aware of the difference and its implications.
> 
> But this has nothing to do with the modules API: it's not more tricky
> then when doing it purely in Elisp.  Are you seriously suggesting we
> deprecate unibyte strings altogether?

We won't deprecate unibyte strings, but we decided long ago to
minimize their use.

> >> "Multibyte" strings are just as subtle (maybe more so even), yet we
> >> rightly don't hesitate to offer a primitive way to construct them.
> > Because we succeed to hide the subtleties in that case,
> > so the multibyte nature is not really visible on the Lisp level,
> > unless you try very hard to make it so.
> 
> Then I don't know what subtleties you're talking about.
> Can you give some examples of the kinds of things you're thinking of?

String concatenation, for one.  Regular expression search for another.
And those just the ones I thought about in the first 5 seconds.

> >> > Instead, how about doing that via vectors of byte values?
> >> What's the advantage?  That seems even more convoluted: create a Lisp
> >> vector of the right size (i.e. 8x the size of your string on a 64bit
> >> system), loop over your string turning each byte into a Lisp integer
> >> (with the reverted API, this involves allocation of an `emacs_value`
> >> box), then pass that to `concat`?
> > That's one way, but I'm sure I can come up with a simpler one. ;-)
> 
> I'm all ears.

Provide an Emacs primitive for that, then at least some of the
awkwardness is gone.  And/or use records.

> >> It's probably going to be even less efficient than going through utf-8
> >> and back.
> > I doubt that.  It's just an assignment.  And it's a rare situation
> > anyway.
> 
> Why do you think it's rare?

Because the number of Emacs features that require you to submit a
unibyte string is very small.

> It's pretty common to receive non-utf-8 byte streams from the external world.
> And when you do receive them, it can come at a very fast pace and become
> temporarily anything but rare.

Are you talking about text encoded in some non-UTF-8 encoding?  If so,
it should be converted to UTF-8, and that will solve the problem.  If
it isn't text, then what common use cases are you talking about?

> 2- the C side string contains text in latin-1, big5, younameit.
>    The module API provides nothing convenient.  Should we force our
>    module to link to C-side coding-system libraries to convert to utf-8
>    before passing it on to the Elisp, even though Emacs already has all
>    the needed facilities?  Really?

Yes, really.  Why is that a problem?  libiconv exists on every
platform we support, and is easy to use.  Moreover, if you just want
to convert a native string into another native string, using Emacs
built-in en/decoding machinery is inconvenient, because it involves
more copying than necessary.

> 3- The C side string contains binary data (say PNG images).
>    What does "arrange for it to be UTF-8" even mean?

Nothing, since in this case there's no meaning to "decoding".

Re: creating unibyte strings

[Stefan Monnier]

[ Boy this discussion is really frustrating.  I should have just added
  the damn feature and moved on.  Now I'm stuck in this morass!  ]

>> But this has nothing to do with the modules API: it's not more tricky
>> then when doing it purely in Elisp.  Are you seriously suggesting we
>> deprecate unibyte strings altogether?
> We won't deprecate unibyte strings, but we decided long ago to
> minimize their use.

Minimize their use doesn't mean that the places where they are used are
less important.  Sometimes what you need is a unibyte string and nothing
else will do.

It also doesn't explain why you want to make it extra cumbersome for
modules whereas Elisp can still do it conveniently.

>> Then I don't know what subtleties you're talking about.
>> Can you give some examples of the kinds of things you're thinking of?
> String concatenation, for one.  Regular expression search for another.
> And those just the ones I thought about in the first 5 seconds.

I don't see in which way these are better hidden for multibyte strings
than they are for unibyte strings.

>> >> > Instead, how about doing that via vectors of byte values?
>> >> What's the advantage?  That seems even more convoluted: create a Lisp
>> >> vector of the right size (i.e. 8x the size of your string on a 64bit
>> >> system), loop over your string turning each byte into a Lisp integer
>> >> (with the reverted API, this involves allocation of an `emacs_value`
>> >> box), then pass that to `concat`?
>> > That's one way, but I'm sure I can come up with a simpler one. ;-)
>> I'm all ears.
> Provide an Emacs primitive for that, then at least some of the
> awkwardness is gone.

No matter the primitive you provide, it means that to build a unibyte
Elisp strings out of a C char[], you're suggesting we go through an
extra copy that uses up 8x the memory.

With such inefficient interfaces, the whole idea of writing modules
becomes completely unattractive: better write a separate application and
communicate via pipes (then you can get unibyte strings in the natural
way).

> And/or use records.

I don't understand what you mean by "use records".

>> >> It's probably going to be even less efficient than going through utf-8
>> >> and back.
>> > I doubt that.  It's just an assignment.  And it's a rare situation
>> > anyway.
>> Why do you think it's rare?
> Because the number of Emacs features that require you to submit a
> unibyte string is very small.

Maybe rare in terms of number of lines of code that will want to do.
But that doesn't mean rare in terms of number of times it'll be executed
for a specific user, so performance considerations should apply.

>> 2- the C side string contains text in latin-1, big5, younameit.
>>    The module API provides nothing convenient.  Should we force our
>>    module to link to C-side coding-system libraries to convert to utf-8
>>    before passing it on to the Elisp, even though Emacs already has all
>>    the needed facilities?  Really?
>
> Yes, really.  Why is that a problem?  libiconv exists on every
> platform we support, and is easy to use.  Moreover, if you just want
> to convert a native string into another native string, using Emacs
> built-in en/decoding machinery is inconvenient, because it involves
> more copying than necessary.

The idea is not to use Emacs as a C library for text conversion, but
that if you receive a latin-1 string and want to pass it to Emacs, it
makes a lot of sense to do:

    make_bytestring (s)
    
and later

    (decode-coding-string s)

then having to link with libiconv.

>> 3- The C side string contains binary data (say PNG images).
>>    What does "arrange for it to be UTF-8" even mean?
> Nothing, since in this case there's no meaning to "decoding".

My point exactly: what should be done instead?

The solution currently used for this existing case is to call make_string
on it (even though it's not a utf-8 string) and then pass it through
(encode-coding-string s 'utf-8) which is ridiculously inefficient
compared to what make_bytestring would do.


        Stefan

Re: creating unibyte strings

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 22 Mar 2019 11:37:59 -0400
> 
> [ Boy this discussion is really frustrating.  I should have just added
>   the damn feature and moved on.  Now I'm stuck in this morass!  ]

That's it, I'm outta here.

Re: creating unibyte strings

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

On Fri, 22 Mar 2019 at 23:14, Eli Zaretskii <eliz@gnu.org> wrote:


> Because the number of Emacs features that require you to submit a
> unibyte string is very small.


In the one native module that I have written, integration of GSSAPI, I end
up passing large byte buffers back and forth between Elisp and C. There
really is no efficient way to do this, so I ended up having to use vectors
and use extract_integer in a loop.

Am I misunderstanding this discussion, and there is a better way to pass
binary buffers to modules that I don't know about?

This is what the current code looks like, and it's not really pretty:
https://github.com/lokedhs/emacs-gssapi/blob/master/gssapi.c#L382

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1183 bytes --]

Re: creating unibyte strings

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Sun, 24 Mar 2019 22:51:16 +0800
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel <emacs-devel@gnu.org>
> 
> In the one native module that I have written, integration of GSSAPI, I end up passing large byte buffers back
> and forth between Elisp and C. There really is no efficient way to do this, so I ended up having to use vectors
> and use extract_integer in a loop.

Can you tell why the byte buffer needs to be exposed to Lisp in this
case?  IOW, what would a Lisp program using this module want to do
with these byte buffers?

Re: creating unibyte strings

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On Mon, 25 Mar 2019, 01:10 Eli Zaretskii, <eliz@gnu.org> wrote:

>
> Can you tell why the byte buffer needs to be exposed to Lisp in this
> case?  IOW, what would a Lisp program using this module want to do
> with these byte buffers?
>

In this particular case, I'm exposing GSSAPI, which is a rather low level
interface to Kerberos (well, it supports multiple systems, but it's mostly
used for Kerberos).

For example, one function is called gss-unwrap, and it takes an encrypted
byte array and decrypts it, returning a new byte array with the decrypted
content.

What the impact is depends on the Elisp code that uses GSS. In the case of
IMAP for example, it's not too bad, since only the initial handshake is
passed through these functions (IMAP encryption is handled by TLS, not
Kerberos). However, there are other uses where every single package is
passed through the wrap and unwrap functions.

Regards,
Elias

Regards,
Elias

>

[-- Attachment #2: Type: text/html, Size: 1702 bytes --]

Re: creating unibyte strings

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Mon, 25 Mar 2019 09:47:26 +0800
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel <emacs-devel@gnu.org>
> 
>  Can you tell why the byte buffer needs to be exposed to Lisp in this
>  case?  IOW, what would a Lisp program using this module want to do
>  with these byte buffers?
> 
> In this particular case, I'm exposing GSSAPI, which is a rather low level interface to Kerberos (well, it supports
> multiple systems, but it's mostly used for Kerberos).
> 
> For example, one function is called gss-unwrap, and it takes an encrypted byte array and decrypts it, returning
> a new byte array with the decrypted content. 
> 
> What the impact is depends on the Elisp code that uses GSS. In the case of IMAP for example, it's not too
> bad, since only the initial handshake is passed through these functions (IMAP encryption is handled by TLS,
> not Kerberos). However, there are other uses where every single package is passed through the wrap and
> unwrap functions. 

I guess I was asking why is this done in Lisp, not in C.  The
decrypted stuff is human-readable text, is it not?  Then why would a
Lisp program want to see the encrypted byte array?

Re: creating unibyte strings

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Mon, 25 Mar 2019 at 11:41, Eli Zaretskii <eliz@gnu.org> wrote:


> I guess I was asking why is this done in Lisp, not in C.  The
> decrypted stuff is human-readable text, is it not?  Then why would a
> Lisp program want to see the encrypted byte array?
>

No, it's not. This is called from the backend protocol implementation of
IMAP, for example. When Gnus wants to load a mailbox, it calls the IMAP
code (Elisp) which then calls into GSSAPI to perform the decryption of the
underlying packets.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 853 bytes --]

Re: creating unibyte strings

[Stefan Monnier]

>> I guess I was asking why is this done in Lisp, not in C.  The
>> decrypted stuff is human-readable text, is it not?  Then why would a
>> Lisp program want to see the encrypted byte array?
> No, it's not. This is called from the backend protocol implementation of
> IMAP, for example. When Gnus wants to load a mailbox, it calls the IMAP
> code (Elisp) which then calls into GSSAPI to perform the decryption of the
> underlying packets.

IIUC the reason why Lisp code sees the byte array is because the module
cannot hook itself directly into the underlying network connection, so
the imap connection returns the byte-array to your code which then
passes it on to the GSSAPI module, which then returns another byte-array
which might be sent back to the imap server, ...


        Stefan