Some dynamic module questions

Some dynamic module questions

[Helmut Eller]

1. Is there some way to pass unibyte-strings to module functions?

2. Why does initialize_environment copy all the function pointers all
   the time?  Would it be more efficient to use the same function table
   for all environments.

Helmut

Re: Some dynamic module questions

[Philipp Stephani]

Am Fr., 27. Nov. 2020 um 11:47 Uhr schrieb Helmut Eller
<eller.helmut@gmail.com>:
>
> 1. Is there some way to pass unibyte-strings to module functions?

Emacs 28 will have a make_unibyte_string function. Before that, you
have to resort to calling the unibyte-string function directly.

>
> 2. Why does initialize_environment copy all the function pointers all
>    the time?  Would it be more efficient to use the same function table
>    for all environments.

The environment structure holds some per-call state (the signal data),
so we need to create a new one for each call. We could in theory
optimize this by preinitializing an environment structure with the
function pointers (saving a few IP-relative memory accesses); I'm not
sure whether that's significantly faster though.

Re: Some dynamic module questions

[Helmut Eller]

On Fri, Nov 27 2020, Philipp Stephani wrote:

> Am Fr., 27. Nov. 2020 um 11:47 Uhr schrieb Helmut Eller
> <eller.helmut@gmail.com>:
>>
>> 1. Is there some way to pass unibyte-strings to module functions?
>
> Emacs 28 will have a make_unibyte_string function. Before that, you
> have to resort to calling the unibyte-string function directly.

But that's only for passing bytes from C to Lisp.  What about the other
direction?  Will there be something like copy_unibyte_string_contents?
Or preferably something that doesn't need copying and just gives a const
char* pointer to the bytes.

Helmut

Re: Some dynamic module questions

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]


Helmut Eller writes:

> But that's only for passing bytes from C to Lisp.  What about the other
> direction?  Will there be something like copy_unibyte_string_contents?
> Or preferably something that doesn't need copying and just gives a const
> char* pointer to the bytes.


unibytes string are actually vector of u8. you can convert to vector(use
vconcat from Lisp side)
first then read it using vec_get and vec_set.


-- 
Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]

Re: Some dynamic module questions

[Helmut Eller]

On Fri, Nov 27 2020, Zhu Zihao wrote:

> Helmut Eller writes:
>
>> But that's only for passing bytes from C to Lisp.  What about the other
>> direction?  Will there be something like copy_unibyte_string_contents?
>> Or preferably something that doesn't need copying and just gives a const
>> char* pointer to the bytes.
>
> unibytes string are actually vector of u8. you can convert to vector(use
> vconcat from Lisp side)
> first then read it using vec_get and vec_set.

Yes, that will be good enough for now. Thanks.

Helmut

Re: Some dynamic module questions

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Date: Fri, 27 Nov 2020 13:01:55 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > Emacs 28 will have a make_unibyte_string function. Before that, you
> > have to resort to calling the unibyte-string function directly.
> 
> But that's only for passing bytes from C to Lisp.  What about the other
> direction?

Why do you need that?  The concept of unibyte vs multibyte strings
exists only in Lisp; in C you just have a sequence of 'char' bytes,
and the way you interpret it depends on your C code.

If you have some specific use case where this looks like a necessity,
can you describe that use case?

Re: Some dynamic module questions

[Helmut Eller]

On Fri, Nov 27 2020, Eli Zaretskii wrote:

>> But that's only for passing bytes from C to Lisp.  What about the other
>> direction?
>
> Why do you need that?  The concept of unibyte vs multibyte strings
> exists only in Lisp; in C you just have a sequence of 'char' bytes,
> and the way you interpret it depends on your C code.
>
> If you have some specific use case where this looks like a necessity,
> can you describe that use case?

I'm writing (or trying to write) ELisp bindings for Cap’n Proto[1],
which is an RPC system along with a data interchange format.  Cap'n
Proto has (among other things[2]) these two data types:

  - Text: is always UTF-8 encoded and NUL-terminated.
  - Data: is a completely arbitrary sequence of bytes.

My plan is to map Text to Elisp (multibyte)strings and Data to unibyte
strings.

Helmut

[1] https://capnproto.org/
[2] https://capnproto.org/language.html

Re: Some dynamic module questions

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: p.stephani2@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 27 Nov 2020 13:59:51 +0100
> 
> I'm writing (or trying to write) ELisp bindings for Cap’n Proto[1],
> which is an RPC system along with a data interchange format.  Cap'n
> Proto has (among other things[2]) these two data types:
> 
>   - Text: is always UTF-8 encoded and NUL-terminated.
>   - Data: is a completely arbitrary sequence of bytes.
> 
> My plan is to map Text to Elisp (multibyte)strings and Data to unibyte
> strings.

Doesn't it work to create a unibyte string in Lisp and then pass it to
the module?

Re: Some dynamic module questions

[Eli Zaretskii]

> Date: Fri, 27 Nov 2020 15:29:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: p.stephani2@gmail.com, emacs-devel@gnu.org
> 
> >   - Text: is always UTF-8 encoded and NUL-terminated.
> >   - Data: is a completely arbitrary sequence of bytes.
> > 
> > My plan is to map Text to Elisp (multibyte)strings and Data to unibyte
> > strings.
> 
> Doesn't it work to create a unibyte string in Lisp and then pass it to
> the module?

Sorry, you are right: that doesn't work.

So you'd need to use vectors instead.

Re: Some dynamic module questions

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 558 bytes --]

> Sorry, you are right: that doesn't work.
> 
> So you'd need to use vectors instead.

Isn't that remarkably slow though? If you either pass a vector of
integers on the Lisp side and recreate it on the C side using Flength
and Farev or have a char array on the C side and create a Lisp array for
the Lisp side, I can see that operation to outweigh otherwise fast
foreign code easily. I've only experimented with encoding unibyte
strings to/from hex strings though when using cryptographic code and
that was slow enough to ruin the benefit of FFI.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Some dynamic module questions

[Stefan Monnier]

>> Emacs 28 will have a make_unibyte_string function. Before that, you
>> have to resort to calling the unibyte-string function directly.
> But that's only for passing bytes from C to Lisp.  What about the other
> direction?  Will there be something like copy_unibyte_string_contents?

Indeed, that would make a lot of sense to add alongside `make_unibyte_string`.


        Stefan

Re: Some dynamic module questions

[Eli Zaretskii]

> Date: Fri, 27 Nov 2020 14:54:06 +0100
> From: Vasilij Schneidermann <mail@vasilij.de>
> Cc: eller.helmut@gmail.com, p.stephani2@gmail.com, emacs-devel@gnu.org
> 
> > So you'd need to use vectors instead.
> 
> Isn't that remarkably slow though? If you either pass a vector of
> integers on the Lisp side and recreate it on the C side using Flength
> and Farev or have a char array on the C side and create a Lisp array for
> the Lisp side, I can see that operation to outweigh otherwise fast
> foreign code easily. I've only experimented with encoding unibyte
> strings to/from hex strings though when using cryptographic code and
> that was slow enough to ruin the benefit of FFI.

I don't yet see why would you need to send data to a module.  Why not
generate that data in the module code itself, based on some Lisp you
get from Emacs?

Re: Some dynamic module questions

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

> I don't yet see why would you need to send data to a module.  Why not
> generate that data in the module code itself, based on some Lisp you
> get from Emacs?

I don't see why you would need to question my approach: Not all data can
be known in advance and it would greatly diminish the utility of a
foreign function interface to limit itself to strictly dealing with
static data.

In any case, Stefan answered the question satisfactorily: It does make
sense to have an equivalent API to transfer unibyte data back to Lisp
code.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Some dynamic module questions

[Eli Zaretskii]

> Date: Fri, 27 Nov 2020 16:21:14 +0100
> From: Vasilij Schneidermann <mail@vasilij.de>
> Cc: eller.helmut@gmail.com, p.stephani2@gmail.com, emacs-devel@gnu.org
> 
> > I don't yet see why would you need to send data to a module.  Why not
> > generate that data in the module code itself, based on some Lisp you
> > get from Emacs?
> 
> I don't see why you would need to question my approach:

Because adding an interface doesn't come for free.  Someone will need
to code it, test, document it, etc.  That is why the modules interface
allows only a relatively small set of objects across.

> In any case, Stefan answered the question satisfactorily: It does make
> sense to have an equivalent API to transfer unibyte data back to Lisp
> code.

So now you get to choose whose answer to accept.

Re: Some dynamic module questions

[Philipp Stephani]

Am Fr., 27. Nov. 2020 um 16:26 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > Date: Fri, 27 Nov 2020 16:21:14 +0100
> > From: Vasilij Schneidermann <mail@vasilij.de>
> > Cc: eller.helmut@gmail.com, p.stephani2@gmail.com, emacs-devel@gnu.org
> >
> > > I don't yet see why would you need to send data to a module.  Why not
> > > generate that data in the module code itself, based on some Lisp you
> > > get from Emacs?
> >
> > I don't see why you would need to question my approach:
>
> Because adding an interface doesn't come for free.  Someone will need
> to code it, test, document it, etc.  That is why the modules interface
> allows only a relatively small set of objects across.

That is correct, but in this case the costs are outweighed by the
benefits. Byte arrays are a ubiquitous fundamental data type (similar
to integers or timestamps), and Emacs modules need good support for
them. For better or worse, Emacs has been using unibyte strings to
represent binary data for a long time (cf. bindat, image), so also
supporting them in modules for this purpose doesn't really introduce
new uses.
Unibyte support has now been requested at least three times
independently (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23487,
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34873, this thread),
showing how much users care about it.

Re: Some dynamic module questions

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Unibyte support has now been requested at least three times

What are the jobs people would like to do using this feature?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Some dynamic module questions

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Unibyte support has now been requested at least three times
> independently (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23487,
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34873, this thread),
> showing how much users care about it.

Sorry; I haven't followed this thread -- but if you're talking about
making unibyte strings in modules, this was addded in
12175a339e2a2214fdd0ab4e16d8d8b1e92a78d3.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Some dynamic module questions

[Helmut Eller]

On Thu, Dec 03 2020, Lars Ingebrigtsen wrote:

> Philipp Stephani <p.stephani2@gmail.com> writes:
>
>> Unibyte support has now been requested at least three times
>> independently (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23487,
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34873, this thread),
>> showing how much users care about it.
>
> Sorry; I haven't followed this thread -- but if you're talking about
> making unibyte strings in modules, this was addded in
> 12175a339e2a2214fdd0ab4e16d8d8b1e92a78d3.

My question is mostly if copy_string_contents can also be used for
unibyte strings or if there should be a separate
copy_unibyte_string_contents.

Encoding unibyte strings in UTF-8 probably doesn't make much sense, or
does it?  And adding a trailing NUL byte is not useful for binary data.

Helmut

Re: Some dynamic module questions

[Philipp Stephani]

Am Do., 3. Dez. 2020 um 09:42 Uhr schrieb Helmut Eller <eller.helmut@gmail.com>:
>
> On Thu, Dec 03 2020, Lars Ingebrigtsen wrote:
>
> > Philipp Stephani <p.stephani2@gmail.com> writes:
> >
> >> Unibyte support has now been requested at least three times
> >> independently (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23487,
> >> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34873, this thread),
> >> showing how much users care about it.
> >
> > Sorry; I haven't followed this thread -- but if you're talking about
> > making unibyte strings in modules, this was addded in
> > 12175a339e2a2214fdd0ab4e16d8d8b1e92a78d3.
>
> My question is mostly if copy_string_contents can also be used for
> unibyte strings or if there should be a separate
> copy_unibyte_string_contents.
>
> Encoding unibyte strings in UTF-8 probably doesn't make much sense, or
> does it?  And adding a trailing NUL byte is not useful for binary data.

I agree that there should be a copy_unibyte_string_contents as well.

Re: Some dynamic module questions

[Stefan Monnier]

> I agree that there should be a copy_unibyte_string_contents as well.

An alternative would be to have a function like `copy_string_contents`
which either returns bytes or a utf-8-encoded sequence of chars
depending on whether the string was unibyte or not, and where it
additionally returns an indication of which was the case.


        Stefan

Re: Some dynamic module questions

[Philipp Stephani]

Am Do., 3. Dez. 2020 um 16:13 Uhr schrieb Stefan Monnier
<monnier@iro.umontreal.ca>:
>
> > I agree that there should be a copy_unibyte_string_contents as well.
>
> An alternative would be to have a function like `copy_string_contents`
> which either returns bytes or a utf-8-encoded sequence of chars
> depending on whether the string was unibyte or not, and where it
> additionally returns an indication of which was the case.
>

Yes, that would be a viable alternative. (It would need a different
name, though, "copy_string_contents" is already taken.)