From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Israelsson Tampe Newsgroups: gmane.lisp.guile.devel Subject: reader musings Date: Wed, 8 Jun 2022 01:07:58 +0200 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000008b765705e0e3a965" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8413"; mail-complaints-to="usenet@ciao.gmane.io" To: guile-devel Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Wed Jun 08 01:48:20 2022 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nyivr-0001ry-EE for guile-devel@m.gmane-mx.org; Wed, 08 Jun 2022 01:48:19 +0200 Original-Received: from localhost ([::1]:44748 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nyivq-0003FJ-9o for guile-devel@m.gmane-mx.org; Tue, 07 Jun 2022 19:48:18 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35814) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nyiJ5-0003E4-E3 for guile-devel@gnu.org; Tue, 07 Jun 2022 19:08:15 -0400 Original-Received: from mail-ed1-x52f.google.com ([2a00:1450:4864:20::52f]:35348) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nyiJ3-0005R8-9Y for guile-devel@gnu.org; Tue, 07 Jun 2022 19:08:15 -0400 Original-Received: by mail-ed1-x52f.google.com with SMTP id x5so19769280edi.2 for ; Tue, 07 Jun 2022 16:08:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=CGEk9LB2b7PwX+apHN6wDCZNPqqD40inAhdTQNNyjIE=; b=ICXU6OhtxYh9GAcLHcZ74eLGriAGdcohU0VeJSZ/5IRcLLY5ZrcvZNUUbqhglr3pAc mCaRECEyoBeN5pUdaNVsTbhX32U3qtDyS0B+DOneUPmC+SJ73gr6tnfq2upKBh4XdbM/ Gkc2XIZwIVU1TKjGiI8rwdK80ACmy94ZjgxU0oVB4+vn83b8/BtXFXelYlPKdazcZzjS J4m9c4hPl11S4aLbrUS3Z/uINPvNnrKT0j2JjTUsPUxiOScEIpbmL0EsOZY2NqMgWpLF V/FRrhosyp+tSDUMKg/6+01MnFYX90w58LVHQ2VniHvob5rBH1ofZGlogdb7u3KrDvQf l67w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=CGEk9LB2b7PwX+apHN6wDCZNPqqD40inAhdTQNNyjIE=; b=hr4cpgvef43Zw5lDXL9RnrbLhecfPqzRbNOBWQIhbw0fpWJCDsVcftDEH3MzloMd8L +KoyV2imsxLeR1DGaU+tEnQEQcT+wUIVw9hATBOOCfZj2JL+vS/qblzpIDmqcI6S1/HO 50DAx430Irb2bP5PVN4lLcQkjyCaczzPFi0Wf2TecaXzpRwA7e92aavl6P6U4u1K90+W 0/JQ9bq3hxKBmbCjP53fDJ3sh/gXjP/5GA78Jxh2jLHwljDwPL/l1Iml7ncE0VXCkBpu ga9Mp9M80pS6GgMsKAWGEpK0fxdVyXG1Heham2m+ojBG3lFbWvzuwHi6daCBO/14abFZ XALQ== X-Gm-Message-State: AOAM532txyp9QaI+AANDH/ZqEsFz14HJEXoVnEhFoqpq0qC62bipTxKF ohzimyQ+BY324FveROEnbee5MSP0D6qwMqX5TIs1aBKFGECPzg== X-Google-Smtp-Source: ABdhPJwm2J4rZkmCjBtaPUUDjdtlNaQynt7O/EyQ3HmCVcnZM5PhkhtnUNG0WxZFYuHUDv3Nw35MFtsp6USIbsAwJlA= X-Received: by 2002:aa7:ce84:0:b0:42d:ce51:8c6e with SMTP id y4-20020aa7ce84000000b0042dce518c6emr37103546edv.10.1654643290239; Tue, 07 Jun 2022 16:08:10 -0700 (PDT) Received-SPF: pass client-ip=2a00:1450:4864:20::52f; envelope-from=stefan.itampe@gmail.com; helo=mail-ed1-x52f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.io gmane.lisp.guile.devel:21219 Archived-At: --0000000000008b765705e0e3a965 Content-Type: text/plain; charset="UTF-8" After finishing a fast writer that balances the use of C and scheme in such a way so that you do not need to spend a long time in C, and all custom hooks are on the scheme level meaning a much more customizable and hackable reader as well as performant. Why not everything in scheme. Well C code can be dramatically faster even if we jit and do some assembly magic's still taking advantage of SIMD instructions we do not have much yet to play with. So in this project we make hefty use of this. For example if we want to write latin1 to utf8 we know that in most cases all letters are ascii with some instances of text with a few special symbols outside the ascii set. And there are some texts that all letters are outside. Anyhow if we have ascii text we can read in 16 bytes and in one instruction check if all are ascii, if not go to a special loop and spend time there until it looks like you are back to ascii text. Basically this. The tricky thing is to handle alignment and end of stream (or shorter strings) also trying to be smart to get some speed increase. Anyhow with this we can write more than 2G ascii characters / s which is kind of cool. Tonn's of similar optimizations makes this write tool speedy as hell. For example a great amount of effort was put into writing real numbers in not just decimal form, but also in hex octal and binary representations similar to what you get from number->string. The end result is more than 20M of reals that can be printed. Efforts have also been made to write out lists and vectors very fast if they contain mostly atoms. For more deep tree like structures with just a small number of leafs at each node the writer is just on par with guile's internal writer written entirely in C. Now we have the advanced features in scheme and mostly the atoms are C - ified (like numbers). What's still missing are some code to handle bigints for which we make use of the number->string function. Guile's internal writer does not handle floats well, it just converts the float to double and then prints out using the double printer. Now in this write tool we put effort into writing out proper presentations of floats meaning that 1.2 will not be written as 1.200003734676 which leads to ugly imprecise and inefficient management of float's. Over to the reader. Reading reals in decimal form is 3X faster than guile's string->number function and reading a bytevector of reals is 5X faster (because guile has a high dispatch overhead). But we added a flag to the writer so that we can specify that all numbers will be printed in hex form, also doubles and floats. ANd now the reader can read in those values and do that so that a bytevector of such coding is 50X faster than guile's vanilla reader. And not only this, we do not lose precision by writing and reading numbers. Now this is of limited use, as a binary representation is usually a better alternative, but still, if you just want to dump a data structure, you get quite an improvement. Code is at: https://gitlab.com/tampe/guile-persist/-/tree/master/ice-9 https://gitlab.com/tampe/guile-persist/-/tree/master/src/write I'm wondering if I should make a C library for other projects to take advantage of this work. Happy hacking --0000000000008b765705e0e3a965 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
After finishing a fast writer that balances the use of C a= nd scheme in such a way so that you do not need to spend a long time in C, = and all custom hooks are on the scheme level meaning a much more customizab= le and hackable reader as well as performant.

Why not ev= erything=C2=A0in scheme. Well C code can be dramatically faster even if we = jit and do some assembly magic's still taking advantage of SIMD instruc= tions we do not have much yet to play with. So in this project we make heft= y use of this. For example if we want to write latin1 to utf8 we know that = in most cases all letters are ascii with some instances of text with a few = special symbols outside the ascii set. And there are some texts that all le= tters are outside. Anyhow if we have ascii text we can read in 16 bytes and= in one instruction check if all are ascii, if not go to a special loop and= spend time there until=C2=A0it looks like you are back to ascii text. Basi= cally this. The tricky thing is to handle alignment and end of stream (or s= horter strings) also trying to be smart to get some speed increase. Anyhow = with this we can write more than 2G ascii characters / s which is kind of c= ool.

Tonn's of similar optimizations makes thi= s write tool speedy as hell. For example a=C2=A0great amount of effort was = put into writing real numbers in not just decimal form, but also in hex oct= al and binary representations similar to what you get from number->strin= g.=C2=A0 The end result is more than 20M of reals that can be printed. Effo= rts have also been made to write out lists and vectors very fast if they co= ntain mostly atoms. For more deep tree like structures with just a small nu= mber of leafs at each node the writer is just on par with guile's inter= nal writer written entirely in C. Now we have the advanced features in sche= me and mostly the atoms are C - ified (like numbers). What's still miss= ing are some code to handle bigints for which we make use of the number->= ;string function.

Guile's internal writer does= not handle floats well, it just converts the float to double and then prin= ts out using the double printer. Now in this write tool we
put ef= fort into writing out proper presentations of floats meaning that 1.2 will = not be written as 1.200003734676=C2=A0 which leads to ugly imprecise and in= efficient management of float's.=C2=A0

Over to= the reader. Reading reals in decimal form is 3X faster than guile's st= ring->number function and reading a bytevector of reals is 5X faster (be= cause guile has a high dispatch overhead). But we added a flag to the write= r so that we can specify that all numbers will be printed in hex form, also= doubles and floats. ANd now the reader can read in those values and do tha= t so that a bytevector of such coding is 50X faster than guile's vanill= a reader. And not only this, we do not lose precision=C2=A0by writing and r= eading numbers. Now this is of limited use, as a binary representation is= =C2=A0 usually a better alternative, but still, if you just want to dump a = data structure, you get quite an improvement.=C2=A0

Code is at:

I'm wondering if I should ma= ke a C library for other projects to take advantage of this work.

Happy hacking

--0000000000008b765705e0e3a965--