From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Israelsson Tampe Newsgroups: gmane.lisp.guile.user,gmane.lisp.guile.devel Subject: Anouncement stis-engine 0.1 Date: Tue, 31 Aug 2021 02:08:54 +0200 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14845"; mail-complaints-to="usenet@ciao.gmane.io" To: Guile User , guile-devel Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Tue Aug 31 02:09:28 2021 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mKrLC-0003gg-KF for guile-user@m.gmane-mx.org; Tue, 31 Aug 2021 02:09:26 +0200 Original-Received: from localhost ([::1]:47438 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mKrLB-0003Hd-92 for guile-user@m.gmane-mx.org; Mon, 30 Aug 2021 20:09:25 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:53094) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mKrKx-0003G6-2b; Mon, 30 Aug 2021 20:09:11 -0400 Original-Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]:39485) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mKrKu-0006V8-MI; Mon, 30 Aug 2021 20:09:10 -0400 Original-Received: by mail-pj1-x102f.google.com with SMTP id mj9-20020a17090b368900b001965618d019so671726pjb.4; Mon, 30 Aug 2021 17:09:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=3jwAo10MHWw6Al86sq8VbpMqOp/u8L9VEcpOZtmum14=; b=UWy+mineniBC6zGW+6ouNUZjeor5bRNiKR70n/Vbh0hUgRRtJhjKYbThBL56vV04zv SQJWM4WhdxVnp+rRhYZLlq1IyN1gU/EpCmYEHK7FLg6gpYnRcTKCwwJgDx2GDKmUAjWP V6NOFxkj70eSz2S7yh2wKZ0DWCQrejJb+91VF2lSsE3UrsNtwjOah8EfuUtDjIt53kuk o2afvI2kCn+or4c1R90pegdaWv+PErCU/LzUl30YW7MOag9xZmRBfTwpjC5P4HUrFlMX maPNcc+ar5pDGEeGTyWufmX5GQfk7m06K10pqJFOmTm44ZJNTs+SAelXa69eee6HzXIg gdgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=3jwAo10MHWw6Al86sq8VbpMqOp/u8L9VEcpOZtmum14=; b=q+bshaI5cBZB15xP7TzwibPtcqQyWzg5f7amh+FFVezoAmmEaYwIFgFALSdy7njBXt 5Xe/b+P36Y1ItkTps6kO/hDkOYPnTSWay62i3W29hHODe83KbEtUYANjEsMHjPmLmhiN Uv4lLxtJJckfhbaWfKYZtbXxsALLPCFbR0nZCG07nKIFmpqixMeNDRVUNrkf0dCSJbQw 9P+wM1bsjK+TIhFy/eTkQkNY/S8LYvaUK6vO6S8CVzr5j2rsaTHHvhysv1/V41H8K6cc u/1idpPtbBEqQ7jN3tTkUWAwux/G2gl6f0Vb6HnMmVUINZ90lB7r8/btwhJB4nvn7vfI 7Obw== X-Gm-Message-State: AOAM532cGvIv+jzPTExKxc8wvdnE0/G3Iy735YKmtAw9LuChlrOD54GW PQG56fdgKUOYIfNN7XYZPKGUvsSWdg+GKP6JHDKOL9IvABI= X-Google-Smtp-Source: ABdhPJwkpy//GbhxUu2aA3MAzat73MzBI3gMJ1G247UZQOtnMR4B4cfb05sFl3Lz774mFlr9JU5aEirWpsrn0pELEto= X-Received: by 2002:a17:90a:9f91:: with SMTP id o17mr1850682pjp.29.1630368545372; Mon, 30 Aug 2021 17:09:05 -0700 (PDT) Received-SPF: pass client-ip=2607:f8b0:4864:20::102f; envelope-from=stefan.itampe@gmail.com; helo=mail-pj1-x102f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:17704 gmane.lisp.guile.devel:20831 Archived-At: Hi guilers! I am now happy to announce the first version 0.1 stis-engine. With this tool you can manage data streams with quite good throughput. Example: Consider a case with streaming objects over the network and allow to have a prefix of the stream where you control what kind of serializer are used and if streamed zipping is done. Below is the server and client implementation in module/fibers/stis-parser/q.scm. 1 400MB big bytevector transports from the client to the server and back with about one second. I did the transport over a zmq thread communication link. We reach quite high throughput as we only do logic in the header of the stream so we can copy the buffers themselves when we can instead of copying the bytes. The sizes of the buffers is quite large as to make sure that the fiber overhead is not too large The main constraint is from copying bytes, just the serializer and deserialiser used to copy an object would be 10x the speed. But this is for simple data structures like bytevectors. Already a list of numbers instead of bytevector the serialisation and deserialisation starts to dominate as bytestream operations is essentially memmove memcpy and those are insanely optimized. Some other examples are audio and image streams. you can glue stream operations together in guile and have close to C speed for large amount of data. Here are the code: ;;We can make an abstractions as such: (define-fpipe-construct #:scm (zmq->atom socket) (zmq->fpipe socket) ; read from the network to a byte stream (let ((it) ; pick up the first byte (it1 ...) means the bytes prefix (opt? (= 1 (logand it 1))) (text? (= 2 (logand it 2)))) (fpipe-skip 1) ; skip the prefix (when opt? (uncompress-from-fpipe-to-fpipe)) ; id the stream was compressed decompress (if text? ; if text the transport is in cleartext e.g. scheme (fpipe->scm) (fpipe->c-atom)))) (define-fpipe-construct #:scm (atom->zmq socket) (let ( it ;; This picks a scheme object (header . payload) (header (car it)) (opt? (memq 'compress header)) (text? (memq 'text header)) (tag (logior (if opt? 1 0) (if text? 2 0)))) (if text? (scm->fpipe) (c-atom->fpipe)) (when opt? (compress-from-fpipe-to-fpipe #:level 3)) (fpipe-prepend tag) ; prepend the stream with the tag ... (fpipe->zmq socket))) ;;Now we can make a server and a client out like so, (define* (make-client address ip-server? #:key (context #f)) (define ctx (if context context (zmq-init))) (define socket (zmq-socket ctx ZMQ_REQ)) (if ip-server? (zmq-bind socket address) (zmq-connect socket address)) ;; here we construct the scheme pipeline using the abstraction (define-values (ch1 ch2) (fpipe-construct (atom->zmq socket) (zmq->atom socket))) (define action (fpipe-schemer ch1 ch2)) (lambda* (message #:key (compress? #f)) (cdr (action (cons (if compress? '(compress) '()) message))))) ;; the server: (define* (run-server server-lambda address ip-server? #:key (context #f)) (define ctx (if context context (zmq-init))) (define socket (zmq-socket ctx ZMQ_REP)) (define (lam x) (call-with-values server-lambda (lambda* (message #:key (compress? #f)) (cons (if compress? '(compress) '()) message)))) (if ip-server? (zmq-bind socket address) (zmq-connect socket address)) ;; the pipeline using the abstractions (define-values (ch1 ch2) (fpipe-construct (zmq->atom socket) (fpipe-map server-lambda) (atom->zmq socket))) (define schemer (fpipe-schemer ch1 ch2)) (spawn-fiber (lambda () (let lp () (schemer %fpipe-eof%) (lp))) #:parallel? #f))