unofficial mirror of guile-devel@gnu.org 
 help / color / mirror / Atom feed
* Announcement stis-engine
@ 2021-08-08 20:27 Stefan Israelsson Tampe
  2021-08-14  4:09 ` Stefan Israelsson Tampe
  2021-08-23 17:31 ` Stefan Israelsson Tampe
  0 siblings, 2 replies; 4+ messages in thread
From: Stefan Israelsson Tampe @ 2021-08-08 20:27 UTC (permalink / raw)
  To: Guile User, guile-devel

I will start the development of stis-engine that is based on fibers and
will experiment with applications that separate into different network
components.

Among things I plan to add serializers, deserializers, zipping and zero mq
networking that works
with fibers. E.g. we will stream things with fiber pipelines, use non
blocking zero mq calls, avoid
allocating large bytevectors, and use as little stack as possible in the
code.

This code depends on stis-data and optionally guile-persists

References:
https://gitlab.com/tampe/stis-engine
https://gitlab.com/tampe/guile-persist
https://gitlab.com/tampe/stis-data/


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Announcement stis-engine
  2021-08-08 20:27 Announcement stis-engine Stefan Israelsson Tampe
@ 2021-08-14  4:09 ` Stefan Israelsson Tampe
       [not found]   ` <c8cec86b-0c32-ce1d-106c-367f2ed7bf6e@posteo.de>
  2021-08-23 17:31 ` Stefan Israelsson Tampe
  1 sibling, 1 reply; 4+ messages in thread
From: Stefan Israelsson Tampe @ 2021-08-14  4:09 UTC (permalink / raw)
  To: Guile User, guile-devel

Okej the system starts to shape up. After some intensive hacking I managed
to make a toolbox for piping binary and scheme streams.
Currently it can pipe a list of 1000 000 elements typically around 1-4s.

see https://gitlab.com/tampe/stis-engine


On Sun, Aug 8, 2021 at 10:27 PM Stefan Israelsson Tampe <
stefan.itampe@gmail.com> wrote:

> I will start the development of stis-engine that is based on fibers and
> will experiment with applications that separate into different network
> components.
>
> Among things I plan to add serializers, deserializers, zipping and zero mq
> networking that works
> with fibers. E.g. we will stream things with fiber pipelines, use non
> blocking zero mq calls, avoid
> allocating large bytevectors, and use as little stack as possible in the
> code.
>
> This code depends on stis-data and optionally guile-persists
>
> References:
> https://gitlab.com/tampe/stis-engine
> https://gitlab.com/tampe/guile-persist
> https://gitlab.com/tampe/stis-data/
>


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Announcement stis-engine
       [not found]     ` <CAGua6m1-vUgEk18Zb7cUw-Rp5kAz+Rg=LDF+0c26JBMghC2-zQ@mail.gmail.com>
@ 2021-08-18 23:37       ` Stefan Israelsson Tampe
  0 siblings, 0 replies; 4+ messages in thread
From: Stefan Israelsson Tampe @ 2021-08-18 23:37 UTC (permalink / raw)
  To: Zelphir Kaltstahl, Guile User, guile-devel

[-- Attachment #1: Type: text/plain, Size: 2279 bytes --]

HI all,

I added a new gem, it' a c based serializer and deserializer of guile data
which is relative feature rich and will ping pong
between scheme and c-land in order to be able to be fiber friendly. So
currently the benchmarks are for the following copy
operations,

,time (load (dump (make-bytevector N 1))
1s for ~ 1GB (test of memcpy)

,time (load (dump (make-vector N 1))
1s for ~ 100MB (will be less pipelined)

,time (load (dump (iota N))
1s for ~ 50MB (will create in total N cons cells with 5 bytes)

The scheme based version of these serializers and deserializers was able to
perform 5MB per second, but thats with much higher allocation overhead

This means that we match pythons dump and load functions. I will add those
to my fibers piping tool

Currently supported data structures
all numbers yes even bignums
all immediates  ('(), booleans characters and whatnot)
bytevectors
strings (wide strings and narrow strings)
symbols
keywords
pairs
vectors

the location of the c-code is in
guile-persist:   https://gitlab.com/tampe/guile-persist

(use-modules (persist persistance)) => atom-load-bv and atom-dump-bv is
defined.

This library is used by python-on-guile and I will replace the current dump
and load with this in the end. Also this tool
will be added to the fiber piping system I am working with in stis-engine.
















On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe <
stefan.itampe@gmail.com> wrote:

> Next is to add support for
> 1. self referential data structures like #1 = '(1 2 3 #1), string
> compression
> 2. Structs
> 3 .Goops Classes and Objects
>
> Then I will add
> 1. Lambdas
> and then Continuations to be able to serialize (if you turn off unboxing
> optimizationsas in guile)
>
> After that a toolbox to customize tserializing just as with python
>
> Note all this is supported in python-on-guile, just that the new code is
> better suited for proper streaming
> and the code is much nicer in my personal view so you do have all these
> features already if you use pythons
> dumps and loads commands together with string->fpipe and fpipe->string.
>
> The new thing is essentially stream compression and streams to zmq's.
> Stream compression wll come to
> python as well as the zlib python module is very rudementary
>
>

[-- Attachment #2: Type: text/html, Size: 3536 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Announcement stis-engine
  2021-08-08 20:27 Announcement stis-engine Stefan Israelsson Tampe
  2021-08-14  4:09 ` Stefan Israelsson Tampe
@ 2021-08-23 17:31 ` Stefan Israelsson Tampe
  1 sibling, 0 replies; 4+ messages in thread
From: Stefan Israelsson Tampe @ 2021-08-23 17:31 UTC (permalink / raw)
  To: Guile User, guile-devel

Have just added serialisations of the following to the c-code serializer,
1) struct and classes
2) hasmaps of type eq? eqv? and equal?
3) lambdas

These are tricky to get right, especially hashmaps looked like it was
impossible, my solution is a bit imprecise, but can
be improved upon.

So now this works,

define h1 (make-hash-table))
(hash-set! h1 "/home/stis/src/guile-persist/a.scm" 0)

(define h2 (make-hash-table))
(hashq-set! h2 0 "/home/stis/src/guile-persist/a.scm")

(define (f x) (lambda (y) (+ x y)))

(pk ((atom-load-bv (atom-dump-bv (f 1) #:j 1 #:eq-map h2 #:equal-map h1
#:do-lam #t)
         #:j   1
         #:map h2) 2))

On Sun, Aug 8, 2021 at 10:27 PM Stefan Israelsson Tampe <
stefan.itampe@gmail.com> wrote:

> I will start the development of stis-engine that is based on fibers and
> will experiment with applications that separate into different network
> components.
>
> Among things I plan to add serializers, deserializers, zipping and zero mq
> networking that works
> with fibers. E.g. we will stream things with fiber pipelines, use non
> blocking zero mq calls, avoid
> allocating large bytevectors, and use as little stack as possible in the
> code.
>
> This code depends on stis-data and optionally guile-persists
>
> References:
> https://gitlab.com/tampe/stis-engine
> https://gitlab.com/tampe/guile-persist
> https://gitlab.com/tampe/stis-data/
>


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-08-23 17:31 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-08 20:27 Announcement stis-engine Stefan Israelsson Tampe
2021-08-14  4:09 ` Stefan Israelsson Tampe
     [not found]   ` <c8cec86b-0c32-ce1d-106c-367f2ed7bf6e@posteo.de>
     [not found]     ` <CAGua6m1-vUgEk18Zb7cUw-Rp5kAz+Rg=LDF+0c26JBMghC2-zQ@mail.gmail.com>
2021-08-18 23:37       ` Stefan Israelsson Tampe
2021-08-23 17:31 ` Stefan Israelsson Tampe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).