From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Israelsson Tampe Newsgroups: gmane.lisp.guile.devel,gmane.lisp.guile.user Subject: Re: Announcement stis-engine Date: Thu, 19 Aug 2021 01:37:02 +0200 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000f254f105c9dde9d6" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="23271"; mail-complaints-to="usenet@ciao.gmane.io" To: Zelphir Kaltstahl , Guile User , guile-devel Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Thu Aug 19 01:37:44 2021 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mGV7v-0005n1-Tq for guile-devel@m.gmane-mx.org; Thu, 19 Aug 2021 01:37:44 +0200 Original-Received: from localhost ([::1]:33032 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mGV7u-0006qU-7D for guile-devel@m.gmane-mx.org; Wed, 18 Aug 2021 19:37:42 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:52980) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mGV7W-0006qI-Am; Wed, 18 Aug 2021 19:37:18 -0400 Original-Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]:36419) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mGV7T-000334-Pj; Wed, 18 Aug 2021 19:37:18 -0400 Original-Received: by mail-pj1-x1031.google.com with SMTP id u13-20020a17090abb0db0290177e1d9b3f7so10100489pjr.1; Wed, 18 Aug 2021 16:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=xy1uUhpAiDSOWTos7Ug4gcuyrYKNE/NdqaKAk05EwtA=; b=rf6aCZZofyRjvBl1ytO6UEL4WfcaNVD9ECDQhllVy3TALHoP7GgG00z6KBE7wtzv8J yhCNSyBej1Y1dgqST+TLECZtgjB3WwPSkzrhVeg5UXL6A9DDknaB08wfXm2DJZi21KnC MegyPj5DWSJSAOZ85aR4bTKm4tANtqlZngANQWQtRd94eaqq1QNChffY2jkBENzRiPsd K8mmz/Nih6vwShK6xQYns+ElFpsxyn2GMWM3KFh5PG5dY5dSxcIbgiR2slgBcuRBFc65 skjT89yyV7t5PDWmBLKjr9c6ldZnnV/0dmH59T63YGPYSvgiJhnYT/gPS99ffctA+fNs S+oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=xy1uUhpAiDSOWTos7Ug4gcuyrYKNE/NdqaKAk05EwtA=; b=CJVJ1WQzEmEshqUubEtknjf73DlBo+s+Kj9sIH1iTa+wVuu2jIf5sjdP5ZUSZBZkVz 2JDMYAM5hZh+EOr9sSj1MmDyOW4h/Uos3mwo6Ba9Sks8Lv2Q3b33t7qL32JnLT8n01I8 N2gdkP2iaat08rTcIEK3hMRZ98C9KJqn+PdjTGfn7M681D+zC9YO+CKgPpRjQ/iVf48C Ewe6CfqmEFKFM2LTvEjm+wz+nSa+FwdYZXflTBEF/X6OlbAH0xYUg/NJSsD7UlqpTWim 4bYvXFTiQA6aX+SeUJ85j9kbXUsJv9pBFRlhpWBFwt03wEwJuctY4QvUsN2C2PA/n5yn wkiw== X-Gm-Message-State: AOAM5317aFqz/w6m8IUbT7UjOzBC0+pyijBAqlToxbfUnR5Fpe3OoHLN 5NFPMSqcmPJkPM1wfXBNoqcTAoDCwIKOQD5r4Kg= X-Google-Smtp-Source: ABdhPJxrWJa1MY9kq68EURPiTEQxycrZovs2nIjmLuuiXi7DP5prCe4iE7vVWO2if5PwHjTVj/876uciE0VpMABLFak= X-Received: by 2002:a17:90a:9f91:: with SMTP id o17mr11517501pjp.29.1629329833480; Wed, 18 Aug 2021 16:37:13 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::1031; envelope-from=stefan.itampe@gmail.com; helo=mail-pj1-x1031.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.io gmane.lisp.guile.devel:20827 gmane.lisp.guile.user:17687 Archived-At: --000000000000f254f105c9dde9d6 Content-Type: text/plain; charset="UTF-8" HI all, I added a new gem, it' a c based serializer and deserializer of guile data which is relative feature rich and will ping pong between scheme and c-land in order to be able to be fiber friendly. So currently the benchmarks are for the following copy operations, ,time (load (dump (make-bytevector N 1)) 1s for ~ 1GB (test of memcpy) ,time (load (dump (make-vector N 1)) 1s for ~ 100MB (will be less pipelined) ,time (load (dump (iota N)) 1s for ~ 50MB (will create in total N cons cells with 5 bytes) The scheme based version of these serializers and deserializers was able to perform 5MB per second, but thats with much higher allocation overhead This means that we match pythons dump and load functions. I will add those to my fibers piping tool Currently supported data structures all numbers yes even bignums all immediates ('(), booleans characters and whatnot) bytevectors strings (wide strings and narrow strings) symbols keywords pairs vectors the location of the c-code is in guile-persist: https://gitlab.com/tampe/guile-persist (use-modules (persist persistance)) => atom-load-bv and atom-dump-bv is defined. This library is used by python-on-guile and I will replace the current dump and load with this in the end. Also this tool will be added to the fiber piping system I am working with in stis-engine. On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe < stefan.itampe@gmail.com> wrote: > Next is to add support for > 1. self referential data structures like #1 = '(1 2 3 #1), string > compression > 2. Structs > 3 .Goops Classes and Objects > > Then I will add > 1. Lambdas > and then Continuations to be able to serialize (if you turn off unboxing > optimizationsas in guile) > > After that a toolbox to customize tserializing just as with python > > Note all this is supported in python-on-guile, just that the new code is > better suited for proper streaming > and the code is much nicer in my personal view so you do have all these > features already if you use pythons > dumps and loads commands together with string->fpipe and fpipe->string. > > The new thing is essentially stream compression and streams to zmq's. > Stream compression wll come to > python as well as the zlib python module is very rudementary > > --000000000000f254f105c9dde9d6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
HI all,

I added a new gem, it' a c = based serializer and deserializer of guile data which is relative feature r= ich and will ping pong=C2=A0
between scheme and c-land in order t= o be able to be fiber friendly. So currently=C2=A0the benchmarks are for th= e following copy
operations,

,time (load= (dump (make-bytevector N 1))
1s for ~ 1GB (test of memcpy)
=

,time (load (dump (make-vector N 1))
1s = for ~ 100MB (will be less pipelined)

,time (load (dump (iota N))
1s for ~ 50MB (will create in tota= l N cons cells with 5 bytes)

The scheme base= d version of these serializers=C2=A0and deserializers was able to perform 5= MB per second, but thats with much higher allocation overhead
This means that we match pythons dump and load functions. I wil= l add those to my fibers piping tool

Currently sup= ported data structures
all numbers yes even bignums
all= immediates=C2=A0 ('(), booleans characters and whatnot)
byte= vectors
strings (wide strings and narrow strings)
symbo= ls
keywords
pairs
vectors

the location of the c-code is in=C2=A0
guile-persist:=C2=A0= =C2=A0https://gitlab.co= m/tampe/guile-persist

(use-modules (persist pe= rsistance)) =3D> atom-load-bv and atom-dump-bv is defined.
This library is used by python-on-guile and I will replace the = current dump and load with this in the end. Also this tool
will b= e added to the fiber piping system I am working with in stis-engine.
<= div>




<= div>




<= div>




<= /div>
On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe <stefan.itampe@gmail.com> wrote:
=
Next is to add support for
1. self referentia= l data structures=C2=A0like #1 =3D '(1 2 3 #1), string compression
2. Structs
3 .Goops Classes and Objects

=
Then I will add
1. Lambdas
and then Co= ntinuations to be able to serialize (if you=C2=A0turn off unboxing optimiza= tionsas=C2=A0in guile)

After that a toolbox to customize tserializing=C2=A0just as with pyth= on

Note all this is supported in python-on-guile, = just that the new code is better suited for proper streaming=C2=A0
and the code is much nicer in my personal view so you do have all these f= eatures already if you use pythons
dumps and loads commands toget= her with string->fpipe and fpipe->string.=C2=A0=C2=A0

<= /div>
The new thing is essentially stream compression and streams to zm= q's. Stream compression wll come to=C2=A0
python as well as t= he zlib python module is very rudementary

--000000000000f254f105c9dde9d6--