Advice on ST-Object data-structure setup

Advice on ST-Object data-structure setup

[ken.dickey]

Greetings,

I am looking at porting a toy Smalltalk-in-Scheme implementation to 
Guile.

[Note https://github.com/KenDickey/Crosstalk ].

The idea is for simplicity first, then making into a module and 
potentially evolving Smalltalk into a supported, compiled ",language" in 
the Guile way.

The first bit of business is to get the mechanics working.  For this I 
need to have an St-Object which is basically a Vector but which answers 
#false to vector? and #true to st-object?

[Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]

I note that one might change the T7 type-tag from 0x0d (vector) to 0x6d 
(currently unused) with a quick bitwise-or, but would like some advice 
on the best way to do this or some alternate.  I would prefer to work in 
Scheme, not C, BTW.

Any hints appreciated.

Thanks much,
-KenD

Re: Advice on ST-Object data-structure setup

[Basile Starynkevitch]

On 4/16/24 23:14, ken.dickey@whidbey.com wrote:
> Greetings,
>
> I am looking at porting a toy Smalltalk-in-Scheme implementation to 
> Guile.
>
> [Note https://github.com/KenDickey/Crosstalk ].
>

On github I (Basile Starynkevitch) am https://github.com/bstarynk and 
contributing to https://github.com/RefPerSys (see also 
http://refpersys.org/ ...)

> The idea is for simplicity first, then making into a module and 
> potentially evolving Smalltalk into a supported, compiled ",language" 
> in the Guile way.
>
> The first bit of business is to get the mechanics working.  For this I 
> need to have an St-Object which is basically a Vector but which 
> answers #false to vector? and #true to st-object?
>
> [Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]
>
> I note that one might change the T7 type-tag from 0x0d (vector) to 
> 0x6d (currently unused) with a quick bitwise-or, but would like some 
> advice on the best way to do this or some alternate.  I would prefer 
> to work in Scheme, not C, BTW.


I think it depends upon your goals; I assume your x86-64 computer is 
running some Linux OS. :

Do you want to code in Smalltalk as quickly as possible? Then install 
GNU smalltalk. https://www.gnu.org/software/smalltalk/ (it is free 
software, a bit old, but you can study the source code).

Do you care about efficiency? Then you want to generate code. A 
possibility is to generate C or C++ code at runtime (in some temporary 
file) then to compile that file into a temporary plugin (forking some 
gcc command) and dlopen that plugin. This in practice can be done many 
dozen of thousands of time in the same process (see 
https://github.com/bstarynk/misc-basile/blob/master/manydl.c for an 
example). See also https://arxiv.org/abs/1109.0779 and some old 
(unmaintained) open source code on 
http://starynkevitch.net/Basile/gcc-melt/ that you'll need to adapt.

If you want to generate efficient machine code directly and accept to 
use a C++ library for that, consider using https://asmjit.com/ (this 
requires knowledge of the x86-64 instruction set).

If you want to generate quickly and portably some (less efficient) 
machine code, use GNU lightning 
<https://www.gnu.org/software/lightning/>. This library generates 
machine code, but in a more abstract and portable (and less efficient) 
way than asmjit.

If you want to take advantage of most optimization passes of a recent 
GCC <https://gcc.gnu.org/> compiler (ie GCC 13 at least) link to its 
libgccjit <https://gcc.gnu.org/onlinedocs/jit/> library (which is the 
GCC compiler packaged as a library). Code generation is as slow as GCC 
can be, but the generation can be optimized by GCC so the generated code 
can be really efficient.

My pet open source project is the RefPerSys ("Reflexive Persistent 
System") inference engine (in C++, GPLv3+ licensed) on 
https://github.com/RefPerSys/RefPerSys/ while it is not Smalltalk, it 
shares a lot of features with it (so I think you could borrow a lot of 
code from it):

RefPerSys has immutable values and mutable objects (I call RefPerSys 
value something which is either an immutable value or a mutable object, 
or null; it fits in a 64 bits word). It is capable of dumping the entire 
heap in JSON textual files - see 
https://github.com/RefPerSys/RefPerSys/tree/master/persistore (and also 
in C++ code), and is loading the entire heap at startup (from a set of 
textual files). It has a precise (multi-threaded) garbage collector and 
generate code at runtime.

Immutable values can be scalars (integers, boxed doubles, strings, ....) 
or composite (finite set of objects, finite tuple of objects, ....). the 
null pointer is interpreted as a lack of value.

RefPerSys objects are mutable, each containing a C++ mutex to serialize 
access or modification of the object. The object model is Smalltalk like 
(or ObjVLisp like): each object has a class (which is an object itself). 
Most objects are persistent (so dumped and reloaded) but some are 
temporary (dumped as null). Each object contains a mutable sequence of 
components (RefPerSys values) and a mutable finite dictionary of 
attribute pairs. An attribute pair has a key (which is a RefPerSys 
object) with a non-null RefPerSys value associated to it.  (This is like 
in Smalltalk). In addition to components and attributes, a RefPerSys 
object may carry a payload (some C++ class) for extra data.

RefPerSys is work in progress (we are seeking some ITEA consortium 
interested by it, or even informal contributors). See also 
http://refpersys.org/ecai2020-highlight-refpersys.pdf and 
http://starynkevitch.net/Basile/refpersys-design.pdf

RefPerSys is currently lacking a syntax (this is being discussed right 
now,mid April 2024, on 
https://framalistes.org/sympa/info/refpersys-forum ), but you could add 
to it some Smalltalk parsing code (e.g. from GNU smalltalk). For that, 
improve its parsrepl_rps.cc C++ file.

Hoping this could be useful to some of you.

Thanks for reading.

Regards from near Paris in France.

-- 
Basile Starynkevitch<basile@starynkevitch.net>
(only mine opinions / les opinions sont miennes uniquement)
8 rue de la Faïencerie, 92340 Bourg-la-Reine, France
web page: starynkevitch.net/Basile/
See/voir:https://github.com/RefPerSys/RefPerSys

Advice on ST-Object data-structure setup

[Basile Starynkevitch]

(sorry, resending as plain text to guile-user)


On 4/16/24 23:14, ken.dickey@whidbey.com wrote:
> Greetings,
>
> I am looking at porting a toy Smalltalk-in-Scheme implementation to 
> Guile.
>
> [Note https://github.com/KenDickey/Crosstalk ].
>

On github I (Basile Starynkevitch) am https://github.com/bstarynk and 
contributing to https://github.com/RefPerSys (see also 
http://refpersys.org/ ...)

> The idea is for simplicity first, then making into a module and 
> potentially evolving Smalltalk into a supported, compiled ",language" 
> in the Guile way.
>
> The first bit of business is to get the mechanics working.  For this I 
> need to have an St-Object which is basically a Vector but which 
> answers #false to vector? and #true to st-object?
>
> [Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]
>
> I note that one might change the T7 type-tag from 0x0d (vector) to 
> 0x6d (currently unused) with a quick bitwise-or, but would like some 
> advice on the best way to do this or some alternate.  I would prefer 
> to work in Scheme, not C, BTW.


I think it depends upon your goals; I assume your x86-64 computer is 
running some Linux OS. :

Do you want to code in Smalltalk as quickly as possible? Then install 
GNU smalltalk. https://www.gnu.org/software/smalltalk/ (it is free 
software, a bit old, but you can study the source code).

Do you care about efficiency? Then you want to generate code. A 
possibility is to generate C or C++ code at runtime (in some temporary 
file) then to compile that file into a temporary plugin (forking some 
gcc command) and dlopen that plugin. This in practice can be done many 
dozen of thousands of time in the same process (see 
https://github.com/bstarynk/misc-basile/blob/master/manydl.c for an 
example). See also https://arxiv.org/abs/1109.0779 and some old 
(unmaintained) open source code on 
http://starynkevitch.net/Basile/gcc-melt/ that you'll need to adapt.

If you want to generate efficient machine code directly and accept to 
use a C++ library for that, consider using https://asmjit.com/ (this 
requires knowledge of the x86-64 instruction set).

If you want to generate quickly and portably some (less efficient) 
machine code, use GNU lightning 
<https://www.gnu.org/software/lightning/>. This library generates 
machine code, but in a more abstract and portable (and less efficient) 
way than asmjit.

If you want to take advantage of most optimization passes of a recent 
GCC <https://gcc.gnu.org/> compiler (ie GCC 13 at least) link to its 
libgccjit <https://gcc.gnu.org/onlinedocs/jit/> library (which is the 
GCC compiler packaged as a library). Code generation is as slow as GCC 
can be, but the generation can be optimized by GCC so the generated code 
can be really efficient.

My pet open source project is the RefPerSys ("Reflexive Persistent 
System") inference engine (in C++, GPLv3+ licensed) on 
https://github.com/RefPerSys/RefPerSys/ while it is not Smalltalk, it 
shares a lot of features with it (so I think you could borrow a lot of 
code from it):

RefPerSys has immutable values and mutable objects (I call RefPerSys 
value something which is either an immutable value or a mutable object, 
or null; it fits in a 64 bits word). It is capable of dumping the entire 
heap in JSON textual files - see 
https://github.com/RefPerSys/RefPerSys/tree/master/persistore (and also 
in C++ code), and is loading the entire heap at startup (from a set of 
textual files). It has a precise (multi-threaded) garbage collector and 
generate code at runtime.

Immutable values can be scalars (integers, boxed doubles, strings, ....) 
or composite (finite set of objects, finite tuple of objects, ....). the 
null pointer is interpreted as a lack of value.

RefPerSys objects are mutable, each containing a C++ mutex to serialize 
access or modification of the object. The object model is Smalltalk like 
(or ObjVLisp like): each object has a class (which is an object itself). 
Most objects are persistent (so dumped and reloaded) but some are 
temporary (dumped as null). Each object contains a mutable sequence of 
components (RefPerSys values) and a mutable finite dictionary of 
attribute pairs. An attribute pair has a key (which is a RefPerSys 
object) with a non-null RefPerSys value associated to it.  (This is like 
in Smalltalk). In addition to components and attributes, a RefPerSys 
object may carry a payload (some C++ class) for extra data.

RefPerSys is work in progress (we are seeking some ITEA consortium 
interested by it, or even informal contributors). See also 
http://refpersys.org/ecai2020-highlight-refpersys.pdf and 
http://starynkevitch.net/Basile/refpersys-design.pdf

RefPerSys is currently lacking a syntax (this is being discussed right 
now,mid April 2024, on 
https://framalistes.org/sympa/info/refpersys-forum ), but you could add 
to it some Smalltalk parsing code (e.g. from GNU smalltalk). For that, 
improve its parsrepl_rps.cc C++ file.

Hoping this could be useful to some of you.

Thanks for reading.

Regards from near Paris in France.

-- 
Basile Starynkevitch<basile@starynkevitch.net>
(only mine opinions / les opinions sont miennes uniquement)
8 rue de la Faïencerie, 92340 Bourg-la-Reine, France
web page: starynkevitch.net/Basile/
See/voir:https://github.com/RefPerSys/RefPerSys

Re: Advice on ST-Object data-structure setup

[ken.dickey]

Basile,

Thank you much for the thoughtful reply.

> On 4/16/24 23:14, ken.dickey@whidbey.com wrote:
..
>> I am looking at porting a toy Smalltalk-in-Scheme implementation to 
>> Guile.

On 2024-04-17 00:10, Basile Starynkevitch wrote:
..
> I think it depends upon your goals; I assume your x86-64 computer is 
> running some Linux OS. :

Not x86.  Aarch64 & RISCV64. I think of x86 as "rust belt". But, yes, 
Linux.

> Do you want to code in Smalltalk as quickly as possible? Then install 
> GNU smalltalk. https://www.gnu.org/software/smalltalk/ (it is free 
> software, a bit old, but you can study the source code).

Thanks. I have been mainly using Cuis Smalltalk for a couple of decades 
now, and am quite happy with it.

[Noter https://en.wikipedia.org/wiki/Draft:Cuis_Smalltalk ]

> Do you care about efficiency? Then you want to generate code.

I guess I need to explain a bit.  The OpenSmalltalk runtime (transpiled 
to C from Smalltalk) is quite nice and very portable.  I did the initial 
port of the stack-interpreter VM to Aarch64 and riscv64.  I have been 
looking lately at the Bee Smalltalk VM which runs directly on x86 
without a VM.

   [ https://github.com/OpenSmalltalk/opensmalltalk-vm
     https://github.com/powerlang/powerlang ]

My concern is that the OpenSmalltalk VM is quite complex and understood 
by very few people.  I would like a backup "taproot" for Cuis and Squeak 
Smalltalks.

I have also a long term interest in bootstrapping. microkernels, and 
multicore SOCs.

So my exploration here is really an interest in a complete bootstrap via 
mes Scheme, which is closely related in spirit to Guile.

  [ https://www.gnu.org/software/mes/ ]

To back up further,  I am an old hacker guy playing around and 
exploring.  I have Scheme runtimes in my list of sins and Guile looks 
interesting enough to play with.

HTH.

Thanks once more for the helpful suggestions,
-KenD

RE: Advice on ST-Object data-structure setup

[Maxime Devos]

>is to get the mechanics working.  For this I 
> need to have an St-Object which is basically a Vector but which 
> answers #false to vector? and #true to st-object?
>
> [Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]
>
> I note that one might change the T7 type-tag from 0x0d (vector) to 
> 0x6d (currently unused) with a quick bitwise-or, but would like some 
> advice on the best way to do this or some alternate.  I would prefer 
> to work in Scheme, not C, BTW.

It would be simplest to define a record type with a single field, let’s say ‘<st-vectorlike>’ with a field ‘actual-vector’(*)

(*) plus whatever fields are required to make it satisfy ‘st-object?’

The objects of type <st-vectorlike> are by construction not vectors (so, they don’t satisfy vector?). However, you can easily define

;; st-vector-ref: rename this to the equivalent of vector-ref in Smalltalk,
;; and replace ‘define’ by what would be the equivalent in Smalltalk
(define (st-vector-ref v n)
  ;; st-vector-like-actual-vector: field getter
  ;; vector-ref: standard Scheme procedure
  (vector-ref (st-vector-like-actual-vector v) n))

and the like.

Best regards,
Maxime Devos.

RE: Advice on ST-Object data-structure setup

[Maxime Devos]

LICENSE says ‘Copyright (c) 2016’, but some files date from 2024. It’s a bit inconsistent … Do you mean that changes >2016 are public domain, or not covered by BSD license, or the inconsistency is an oversight?

Re: Advice on ST-Object data-structure setup

[ken.dickey]

On 2024-04-17 09:34, Maxime Devos wrote:

> LICENSE says 'Copyright (c) 2016', but some files date from 2024. It's a bit inconsistent ... Do you mean that changes >2016 are public domain, or not covered by BSD license, or the inconsistency is an oversight?

Ah. Thanks. 

I used Larceny Scheme for the initial development and am just starting
to update the code to use in Guile.  Guile code broken so far. 

Just an oversight.  Open source license. 

-KenD

Re: Advice on ST-Object data-structure setup

[Mikael Djurfeldt]

Hi Ken,

Have you looked at Guile's OOP system GOOPS?

Chances are that you would get many required mechanisms for free. You get
classes that can be instantiated. You could possibly implement messages as
generic functions. The generic messages/generic functions already support
keyword arguments, etc.

Best regards,
Mikael

On Tue, Apr 16, 2024 at 11:15 PM <ken.dickey@whidbey.com> wrote:

> Greetings,
>
> I am looking at porting a toy Smalltalk-in-Scheme implementation to
> Guile.
>
> [Note https://github.com/KenDickey/Crosstalk ].
>
> The idea is for simplicity first, then making into a module and
> potentially evolving Smalltalk into a supported, compiled ",language" in
> the Guile way.
>
> The first bit of business is to get the mechanics working.  For this I
> need to have an St-Object which is basically a Vector but which answers
> #false to vector? and #true to st-object?
>
> [Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]
>
> I note that one might change the T7 type-tag from 0x0d (vector) to 0x6d
> (currently unused) with a quick bitwise-or, but would like some advice
> on the best way to do this or some alternate.  I would prefer to work in
> Scheme, not C, BTW.
>
> Any hints appreciated.
>
> Thanks much,
> -KenD
>
>

Re: Advice on ST-Object data-structure setup

[Mikael Djurfeldt]

Then, of course, mapping language onto another complex system is perhaps
seldom a good idea. Still, it is one way to create a new type of object.
And, generic functions on such objects could still provide for a nice
implementation of Smalltalk, even if you can't map the OOP systems onto
eachother.

On Sat, Apr 20, 2024 at 11:49 AM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> Hi Ken,
>
> Have you looked at Guile's OOP system GOOPS?
>
> Chances are that you would get many required mechanisms for free. You get
> classes that can be instantiated. You could possibly implement messages as
> generic functions. The generic messages/generic functions already support
> keyword arguments, etc.
>
> Best regards,
> Mikael
>
> On Tue, Apr 16, 2024 at 11:15 PM <ken.dickey@whidbey.com> wrote:
>
>> Greetings,
>>
>> I am looking at porting a toy Smalltalk-in-Scheme implementation to
>> Guile.
>>
>> [Note https://github.com/KenDickey/Crosstalk ].
>>
>> The idea is for simplicity first, then making into a module and
>> potentially evolving Smalltalk into a supported, compiled ",language" in
>> the Guile way.
>>
>> The first bit of business is to get the mechanics working.  For this I
>> need to have an St-Object which is basically a Vector but which answers
>> #false to vector? and #true to st-object?
>>
>> [Note https://github.com/KenDickey/Crosstalk/blob/master/st-kernel.scm ]
>>
>> I note that one might change the T7 type-tag from 0x0d (vector) to 0x6d
>> (currently unused) with a quick bitwise-or, but would like some advice
>> on the best way to do this or some alternate.  I would prefer to work in
>> Scheme, not C, BTW.
>>
>> Any hints appreciated.
>>
>> Thanks much,
>> -KenD
>>
>>

Re: Advice on ST-Object data-structure setup

[ken.dickey]

On 2024-04-20 02:49, Mikael Djurfeldt wrote:

> Have you looked at Guile's OOP system GOOPS?

Thanks, Mikael.

I have implemented a number of object systems, including CLOS style 
systems such as GOOPS.

The mismatch here is in the overhead of the multimethod dispatch.

I would like something fairly efficient -- and without using caches if 
possible.

You probably know/have seen this, but back to basics...

If you look at the `behavior` function in 
https://github.com/KenDickey/Crosstalk/blob/master/guile-st-kernel.scm 
you will see that the basic dispatch mechanism is (primLookup: (behavior 
self) selectorSym) where the lookup is just a hash-table lookup with 
suitable failure handling.

(define (primLookup: methodDict symbol)
   (hashtable-ref methodDict
                  symbol
                  (lambda (self . rest-args)
                    (send-failed self symbol rest-args)))
                    ;; (make-messageSend self symbol rest-args)))
)

The natural way to interoperate with Scheme data types is to use type 
tag as an array index into a vector of functions which return a suitable 
behavior/hash-table.

For generic Smalltalk objects, this is just the first slot of a vector 
of value slots.  For typical Scheme objects, this is a method-dictionary 
for that kind of object.  So Scheme vectors, numbers, #true, closures, 
and so forth just work as expected when viewed as Smalltalk Arrays, 
numbers, true, BlockClosures..

So after optimization, dispatch mechanics is just mask the tag, shift, 
index, hash, apply.

The hard parts have to do with thisContext (call/cc interactions) 
keeping a proper debugging context, and Smalltalk`become`.

It appears that Guile has evolved to have suitable low-level mechanisms 
to be able to do this.

I find things out by writing code.  "If it works, it must be possible."

So my investigation is to get a basic Smalltalk working and then get 
into the compilation and runtime mechanics to see if I can make this 
efficient enough.

Starting with a CLOS style dispatch is a bit further from where I want 
to end up.

Thanks again for the thoughtful comments,
-KenD