Q: BLV for function slots + BL obarray/hmap for symbol lookup?

Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

1. Is there some special reason why BLV are implemented only for value
slots and not for function slots? Would it be inneficient, too
complicated, or just considered as not needed?

2. Would it be possible, not too inneficient, complicated etc, if obarray
or hashmap, whichever is used for symbol lookup by Emacs, could be defined
as BLV, so that Emacs would lookup symbols similar as it does for buffer
local variables?

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

> 1. Is there some special reason why BLV are implemented only for value
> slots and not for function slots? Would it be inneficient, too
> complicated, or just considered as not needed?

35 years of experience suggest it's not needed (thank god).
It's quite easy to instead do something like

    (defun foo (..)
      (call foo-function ..))

and then use a buffer-local setting for `foo-function` to get the same result.

> 2. Would it be possible, not too inneficient, complicated etc, if obarray
> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
> as BLV, so that Emacs would lookup symbols similar as it does for buffer
> local variables?

35 years of experience suggest it's not needed (thank god).
Also, I think this could lead to quite unexpected semantics.


        Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 1. Is there some special reason why BLV are implemented only for value
>> slots and not for function slots? Would it be inneficient, too
>> complicated, or just considered as not needed?
>
> 35 years of experience suggest it's not needed (thank god).
> It's quite easy to instead do something like
>
>     (defun foo (..)
>       (call foo-function ..))
>
> and then use a buffer-local setting for `foo-function` to get the same result.

Yeah, I know, it is exactly what I do know: defvar-local some-fn, and then save a
function in variable. However there is a fundamental restriction: if
there is already a foo defined, I can't use that name.

Beside, the obvious namespace problem, there is a thing I would like to
do, which I can't. If I would like to slightly modify a function, I can
either re-install my own version and do some conditional check to do own
thing or return original computation, or I could advice. Both of those
options have global effect in entire Emacs. It might be costly in terms
of performance, so in that case, I might use my own version just in some
buffer, leaving rest unnaffected.

>> 2. Would it be possible, not too inneficient, complicated etc, if obarray
>> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
>> as BLV, so that Emacs would lookup symbols similar as it does for buffer
>> local variables?
>
> 35 years of experience suggest it's not needed (thank god).
It can also just mean that for 35 years people haven't being doing what
some people would like to do now? :) 

Can you give me anything on technical side of quesion? 

> Also, I think this could lead to quite unexpected semantics.

Ok, there you have something interesting. In which way? If you have time
to expand on that one. It is already possible to use custo obarray, but
we have to tell Emacs which obarray to use. I would like it to sort of
happen automatically if there is a buffer local one. Sort of. No idea
what would happen if there were more than one local :-). 

To explaing more:

I would like to have a private namespace, in this case it would provide
for a buffer-local namespae, if I can call it so.

It would be just one level: buffer-local or global. 

I don't want to write too long, but I would like to have a way to skip
prefixes in names and to be able to use ordinary elisp semantics of
defun, defmacro etc, but that work only on some "local" level. Buffer
local would be really nice. I could otherwise define my own DSL, but I
don't want to do that.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

> Ok, there you have something interesting. In which way? If you have time
> to expand on that one. It is already possible to use custo obarray, but
> we have to tell Emacs which obarray to use. I would like it to sort of
> happen automatically if there is a buffer local one. Sort of. No idea
> what would happen if there were more than one local :-). 

I have the impression that your needs are a bit vague ;-)
E.g. which buffer should be used to resolve the "buffer-localness"?
The one when `intern` is called or the one when the already-interned
symbol is used?

Maybe you'd be interested in the effort to add some kind of namespace
mechanism by adding some "prefix rewrite" rules used by the reader, so
you could say that the "f:" prefix maps to "formi-" and then a
symbol like "f:dable" is read as "formi-dable"?


        Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Ok, there you have something interesting. In which way? If you have time
>> to expand on that one. It is already possible to use custo obarray, but
>> we have to tell Emacs which obarray to use. I would like it to sort of
>> happen automatically if there is a buffer local one. Sort of. No idea
>> what would happen if there were more than one local :-). 
>
> I have the impression that your needs are a bit vague ;-)

I didn't want to write too long last time.

I have been experimenting with a little framework last 3 days, and I am
quite sure I need some kind of namespace, buffer local would be perfect
in this case. I don't wish to post any code on a public list atm, untill
I am done with it.

A short sketch is:

I have derived elisp mode, and I setup a sort of a "workbook"
with some predefined environment. There can be several "workbooks" open
at a time, each one is a mini-application, domain specific. User writes
elisp code, but since it is a domain specific, there may (and will) be
lots of variables and functions with same name, so they have to 
be insulated from each other. They also won't do sense in rest of
Emacs, but Emacs will need to be able to read and eval each workbook.

I also don't wish to force prefix usage since it would be very tedious,
and bit contraproductive in context of a simple domain specific
framework. I could impose restriction on one workbook at a time, but
there is still risk of stepping over some Emacs function.

> E.g. which buffer should be used to resolve the "buffer-localness"?
> The one when `intern` is called or the one when the already-interned
> symbol is used?

I am not sure I understand what you mean here. The rule would be same as
for buffer local vars, but they would work on symbol level instead of
variable slot level, at least conceptually. If "local obarray" is
declafed in a buffer, all interning would go to that one, like all
set/qset set's the local vaalue if there is one. The locally interned
symbol would automatically shadow the global one, and in buffers where
there is no local obarray the global one would be used. Maybe obarray is
not needed, maybe we could just somehow "make-local-symbol" instead of
"make-local-variable"? I don't know, I asked because I guess person(s)
who implemented the mechanism probably had the thought themselves and
decided for some reason to go just for variable slot and not entire
symbol. 

> Maybe you'd be interested in the effort to add some kind of namespace
> mechanism by adding some "prefix rewrite" rules used by the reader, so
> you could say that the "f:" prefix maps to "formi-" and then a
> symbol like "f:dable" is read as "formi-dable"?

I haven't been monitoring "the effort", so I can't tell much about it,
but I did initially had thought of doing my own name-mangling. Since the
framework has a luxury of user pressing on a "run" button to run the
workbook, I actually eval the buffer myself, so I can rewrite
declarations myself before I hand them to Emacs for eval. However, the
debugger would see rewritten names. I don't wish that. Also there is
effort of writing pre-processor/compiler whatever, which I am not sure I
wish to invest in.

If there was some mechanism to tell Emacs that symbol is buffer local,
it would come out for free so to say.

I was looking a bit in eval.c and lread.c how bvl work, but I don't
understand it enough so I could hack myself this, so that is why I
asked. Since there is already a way to switch between obarrays, I
thought it might have been an easier route. But any way to tell Emacs
that a symbol is buffer local, and eval respecting this, would work.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

>> E.g. which buffer should be used to resolve the "buffer-localness"?
>> The one when `intern` is called or the one when the already-interned
>> symbol is used?
>
> I am not sure I understand what you mean here. The rule would be same as
> for buffer local vars, but they would work on symbol level instead of
> variable slot level, at least conceptually.

E.g. when we `load` a file, the `current-buffer` is the one which
happened to be current when the `load` function was called.
Usually that's when `intern` is executed.  So if you just naively make
`intern` use `current-buffer` to decide what to do, you may get very
unpredictable behavior.

Have you looked at the different attempts to add some kind of
namespace support to Emacs?  Some of them may be particularly well
suited to your DSL situation.  When writing our history of ELisp paper,
we found: Fakespaces, Namespaces, with-namespace, Codex, Names, and
Nameless (the last two are in GNU ELPA).

> If "local obarray" is declafed in a buffer, all interning would go to
> that one, like all set/qset set's the local vaalue if there is one.

But how do you control which buffer is current when the `intern` takes place?

Also, have you considered something like

    (setq-local obarray (obarray-copy obarray))


-- Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> E.g. which buffer should be used to resolve the "buffer-localness"?
>>> The one when `intern` is called or the one when the already-interned
>>> symbol is used?
>>
>> I am not sure I understand what you mean here. The rule would be same as
>> for buffer local vars, but they would work on symbol level instead of
>> variable slot level, at least conceptually.
>
> E.g. when we `load` a file, the `current-buffer` is the one which
> happened to be current when the `load` function was called.
> Usually that's when `intern` is executed.  So if you just naively make
> `intern` use `current-buffer` to decide what to do, you may get very
> unpredictable behavior.

I am still not sure I understand the problem 100%, but I understand it
partially I think :). Thanks.

> Have you looked at the different attempts to add some kind of
> namespace support to Emacs?  Some of them may be particularly well
> suited to your DSL situation.  When writing our history of ELisp paper,
> we found: Fakespaces, Namespaces, with-namespace, Codex, Names, and
> Nameless (the last two are in GNU ELPA).

Once, some years ago it was very popular with namespaces, so I have
heard about some, read some rants on Wiki pages, but I didn't use Elisp
that much, so I never checked them out. Back then namespaces used to pop
up as completion frameworks does nowadays.

This is the first time I feel I would have real use for a namespace (I
peronally don't mind typing prefixes). Names package look like something
I'll investigate and see if it works or the approach you suggest below.

>> If "local obarray" is declafed in a buffer, all interning would go to
>> that one, like all set/qset set's the local vaalue if there is one.
>
> But how do you control which buffer is current when the `intern` takes place?
>
> Also, have you considered something like
>
>     (setq-local obarray (obarray-copy obarray))

You mean to copy Emacs global obarray into local variable, the other way
around so to say, and use that one all the way? I didn't :-). Feels a bit
drastic to a big global environment, but it might work, I'll have to try
it. Thanks for the suggestion(s).

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Arthur Miller <arthur.miller@live.com> writes:

>>> If "local obarray" is declafed in a buffer, all interning would go to
>>> that one, like all set/qset set's the local vaalue if there is one.
>>
>> But how do you control which buffer is current when the `intern` takes place?
>>
>> Also, have you considered something like
>>
>>     (setq-local obarray (obarray-copy obarray))
>
> You mean to copy Emacs global obarray into local variable, the other way
> around so to say, and use that one all the way? I didn't :-). Feels a bit
> drastic to a big global environment, but it might work, I'll have to try
> it. Thanks for the suggestion(s).

Just as curiosa to confirm, yes that worked, rather well :-). That was actually
what I asked for if you think, I just didn't thought of copying over the
obarray.

(setq-local obarray (copy-sequence obarray))
(defvar mysymbol "A SYMBOL")
(defun myfunc () "HELLO")

nysymbol and myfunc were avialable only in that buffer.

Don't know how efficient it is to copy entire obarray, in possibly
serveral buffers, but for the intended purpose, as a buffer-local
namespace works well. Thanks. 

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Michael Heerdegen]

Arthur Miller <arthur.miller@live.com> writes:


> (setq-local obarray (copy-sequence obarray))
                       ^^^^^^^^^^^^^

I guess this is something different than with the originally suggested
`obarray-copy'.

I once was told that obarray is not just simply an array of existing
symbols, e.g. for me currently

 (length obarray) ==> 15121

 (let ((i 0)) (mapatoms (lambda (_) (cl-incf i))) i) ==> 66002


Michael.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>
>> (setq-local obarray (copy-sequence obarray))
>                        ^^^^^^^^^^^^^
>
> I guess this is something different than with the originally suggested
> `obarray-copy'.
>
> I once was told that obarray is not just simply an array of existing
> symbols

You mean I got a shallow copy of the obarray instead of a deep copy?

Where do I find this obarray-copy? :-) I don't see it in 27.1.

>  (length obarray) ==> 15121

Yes, I get the same apocalytpic number, surprise :).

>  (let ((i 0)) (mapatoms (lambda (_) (cl-incf i))) i) ==> 66002

I get 45416.

This is beyond my knowledge about how obarrays in Emacs are made. I just
looked quickly in the source and see that make-obarray is in principle
just a wrapper for (make-vector size 0), which then calls into
make_vector in C code, so obviouslu the vector is allocated elsewhere
since the length is a prime number. I guess it is just the number of
predefined slots in the vector for hashing, not the actually numer of
symbols in the array? I guess just buckets for chaining? I am not sure
how to interpret the second number from mapatoms, collisions? :) But I
am probably wrong, just speculating.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Philipp]

> Am 30.05.2021 um 04:57 schrieb Arthur Miller <arthur.miller@live.com>:
> 
> Michael Heerdegen <michael_heerdegen@web.de> writes:
> 
>> Arthur Miller <arthur.miller@live.com> writes:
>> 
>> 
>>> (setq-local obarray (copy-sequence obarray))
>>                       ^^^^^^^^^^^^^
>> 
>> I guess this is something different than with the originally suggested
>> `obarray-copy'.
>> 
>> I once was told that obarray is not just simply an array of existing
>> symbols
> 
> You mean I got a shallow copy of the obarray instead of a deep copy?

I wouldn't call it a shallow copy; you get a weird object that doesn't behave as expected:

(let ((a (obarray-make 1)))
  (intern "u" a)
  (intern "v" a)
  (let ((b (copy-sequence a)))
    (unintern "u" b))
  (let ((syms ()))
    (mapatoms (lambda (s) (push s syms)) a)
    syms))

=> (v)

Here, uninterning "u" from the "copy" has also removed it from the original.


> 
> Where do I find this obarray-copy? :-) I don't see it in 27.1.

It doesn't exist yet, you'll need to write it.  For example:

(defun obarray-copy (ob)
  (let ((r (obarray-make (obarray-size ob))))
    (obarray-map (lambda (sym) (obarray-put (symbol-name sym) r))) ob)
    r))

> 
>> (length obarray) ==> 15121
> 
> Yes, I get the same apocalytpic number, surprise :).
> 
>> (let ((i 0)) (mapatoms (lambda (_) (cl-incf i))) i) ==> 66002
> 
> I get 45416.
> 
> This is beyond my knowledge about how obarrays in Emacs are made.

See the node "Creating Symbols" in the ELisp manual, which explains the details.  (Probably too well, since they are really implementation details.)

For the full picture, check the definition of 'struct Lisp_Symbol' in lisp.h and the obarray functions in lread.c.

Obarrays are rather weird objects that are not really well supported by most Elisp functions.  You can only work with them reliably using the dedicated obarray functions (intern, unintern, mapatoms, the functions from obarray.el).

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

>>>     (setq-local obarray (obarray-copy obarray))
[...]
> (setq-local obarray (copy-sequence obarray))

obarray-copy != copy-sequence


        Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>     (setq-local obarray (obarray-copy obarray))
> [...]
>> (setq-local obarray (copy-sequence obarray))
>
> obarray-copy != copy-sequence

My Emacs 27.1 says void function obarray-copy, but copy-sequence 
worked. Aren't obarray just vectors?

For your previous mail; yes I am quite aware this is very use-case
specific solution. Anything done elsewhere, outside that particular
buffer after the copy is performed will not be visible in that buffer,
as well as no definition will escape to rest of the Emacs, so this
buffer can only be used to change state of this particular buffer and
nothing else, and that can be quite brittle. For the bad and good. I
haven't experimented enough yet, just a bit, I am not sure how it will
work with Emacs state internally, gc? etc.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

Arthur Miller [2021-05-30 04:30:24] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>>     (setq-local obarray (obarray-copy obarray))
>> [...]
>>> (setq-local obarray (copy-sequence obarray))
>>
>> obarray-copy != copy-sequence
> My Emacs 27.1 says void function obarray-copy,

Indeed, it's not provided.

> but copy-sequence worked.

Not really, no.  You just haven't noticed the breakage yet.

>  Aren't obarray just vectors?

No, they're very special vectors.  E.g.:

    (let ((oa (make-vector 1 nil)))
      (intern "foo" oa)
      (intern "bar" oa)
      (intern "baz" oa)
      oa)

    ==>

      [baz]

Yet, `foo` and `bar` are definitely still in there.

> For your previous mail; yes I am quite aware this is very use-case
> specific solution. Anything done elsewhere, outside that particular
> buffer after the copy is performed will not be visible in that buffer,
> as well as no definition will escape to rest of the Emacs, so this
> buffer can only be used to change state of this particular buffer and
> nothing else, and that can be quite brittle. For the bad and good. I
> haven't experimented enough yet, just a bit, I am not sure how it will
> work with Emacs state internally, gc? etc.

AFAIK I think it can be made to work, yes.
I don't expect any problem from "Emacs state internally" or the GC.

The only source of trouble I can foresee is if "normal code" ends up
running while your obarray is the one held in the global `obarray` var.
This is because "normal code" will occasionally load files (via
`require` or autoloads, typically) and that can quickly lead to
confusion.


        Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Arthur Miller [2021-05-30 04:30:24] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>>>     (setq-local obarray (obarray-copy obarray))
>>> [...]
>>>> (setq-local obarray (copy-sequence obarray))
>>>
>>> obarray-copy != copy-sequence
>> My Emacs 27.1 says void function obarray-copy,
>
> Indeed, it's not provided.
>
>> but copy-sequence worked.
>
> Not really, no.  You just haven't noticed the breakage yet.

Shallow copy, as Michael pointed it out? I'll see to update my Emacs.

>>  Aren't obarray just vectors?
>
> No, they're very special vectors.  E.g.:
>
>     (let ((oa (make-vector 1 nil)))
>       (intern "foo" oa)
>       (intern "bar" oa)
>       (intern "baz" oa)
>       oa)
>
>     ==>
>
>       [baz]
>
> Yet, `foo` and `bar` are definitely still in there.

Yes, manual says use size 0 when making an obarray, so I understand the
space is allocated elsewhere, when interning is happening.

>> For your previous mail; yes I am quite aware this is very use-case
>> specific solution. Anything done elsewhere, outside that particular
>> buffer after the copy is performed will not be visible in that buffer,
>> as well as no definition will escape to rest of the Emacs, so this
>> buffer can only be used to change state of this particular buffer and
>> nothing else, and that can be quite brittle. For the bad and good. I
>> haven't experimented enough yet, just a bit, I am not sure how it will
>> work with Emacs state internally, gc? etc.
>
> AFAIK I think it can be made to work, yes.
> I don't expect any problem from "Emacs state internally" or the GC.
>
> The only source of trouble I can foresee is if "normal code" ends up
> running while your obarray is the one held in the global `obarray` var.
> This is because "normal code" will occasionally load files (via
> `require` or autoloads, typically) and that can quickly lead to
> confusion.
>
Allright, thanks for the clarifications.

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Stefan Monnier]

> Just as curiosa to confirm, yes that worked, rather well :-). That was actually
> what I asked for if you think, I just didn't thought of copying over the
> obarray.
>
> (setq-local obarray (copy-sequence obarray))
> (defvar mysymbol "A SYMBOL")
> (defun myfunc () "HELLO")
>
> nysymbol and myfunc were avialable only in that buffer.
>
> Don't know how efficient it is to copy entire obarray, in possibly
> serveral buffers, but for the intended purpose, as a buffer-local
> namespace works well. Thanks.

In terms of efficiency: the global obarray holds about 50k-100k symbols,
every symbols occupies 6 words (i.e. 48 bytes on a 64bit system), so the
copy should allocate about 3-5MB of memory.

That's not very large if you only expect to make a couple of such
obarrays in any Emacs session.  In any case, you may encounter more
serious difficulties along the way, such as the fact that reading `nil`
will not return the same symbol as when you read `()`.

It *can* be a solution for a situation like a DSL, but in any case it
needs to be used with a lot of care (e.g. you don't want to end up
`require`ing (or more generally `load`ing) a normal ELisp package in
a context where `obarray` refers to such a copy).


        Stefan

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Clément Pit-Claudel]

On 5/22/21 5:46 PM, Arthur Miller wrote:
> 2. Would it be possible, not too inneficient, complicated etc, if obarray
> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
> as BLV, so that Emacs would lookup symbols similar as it does for buffer
> local variables?

We have debated doing this in Flycheck with an explicit obarray (properties of Flycheck checkers are stored as symbol properties).

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> On 5/22/21 5:46 PM, Arthur Miller wrote:
>> 2. Would it be possible, not too inneficient, complicated etc, if obarray
>> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
>> as BLV, so that Emacs would lookup symbols similar as it does for buffer
>> local variables?
>
> We have debated doing this in Flycheck with an explicit obarray (properties of Flycheck checkers are stored as symbol properties).

Ok, cool, interesting. What have you come up with then? Is there some
public mail archive where discussion is archived?

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Clément Pit-Claudel]

On 5/23/21 4:23 PM, Arthur Miller wrote:
> Clément Pit-Claudel <cpitclaudel@gmail.com> writes:
> 
>> On 5/22/21 5:46 PM, Arthur Miller wrote:
>>> 2. Would it be possible, not too inneficient, complicated etc, if obarray
>>> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
>>> as BLV, so that Emacs would lookup symbols similar as it does for buffer
>>> local variables?
>>
>> We have debated doing this in Flycheck with an explicit obarray (properties of Flycheck checkers are stored as symbol properties).
> 
> Ok, cool, interesting. What have you come up with then? Is there some
> public mail archive where discussion is archived?

https://github.com/flycheck/flycheck/issues/1762#issuecomment-749826601

Re: Q: BLV for function slots + BL obarray/hmap for symbol lookup?

[Arthur Miller]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> On 5/23/21 4:23 PM, Arthur Miller wrote:
>> Clément Pit-Claudel <cpitclaudel@gmail.com> writes:
>> 
>>> On 5/22/21 5:46 PM, Arthur Miller wrote:
>>>> 2. Would it be possible, not too inneficient, complicated etc, if obarray
>>>> or hashmap, whichever is used for symbol lookup by Emacs, could be defined
>>>> as BLV, so that Emacs would lookup symbols similar as it does for buffer
>>>> local variables?
>>>
>>> We have debated doing this in Flycheck with an explicit obarray (properties of Flycheck checkers are stored as symbol properties).
>> 
>> Ok, cool, interesting. What have you come up with then? Is there some
>> public mail archive where discussion is archived?
>
> https://github.com/flycheck/flycheck/issues/1762#issuecomment-749826601

Thank you.