From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: "Dr. Arne Babenhauserheide" Newsgroups: gmane.lisp.guile.devel,gmane.lisp.guile.user Subject: Re: Guile Potluck 2021 Date: Sat, 27 Feb 2021 17:43:51 +0100 Message-ID: <87lfb98nqw.fsf@web.de> References: <1144e4d9920845eb4f49e6e099a306e85735bd01.camel@yahoo.com> <87tupx97ey.fsf@web.de> <10585499-c6ef-4cce-b576-60940b3aa4d9@www.fastmail.com> <87o8g58nsv.fsf@web.de> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="10696"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.15; emacs 27.1 Cc: guile-user , guile-devel@gnu.org To: Linus =?utf-8?Q?Bj=C3=B6rnstam?= Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Sat Feb 27 17:44:14 2021 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lG2hS-0002ez-5i for guile-devel@m.gmane-mx.org; Sat, 27 Feb 2021 17:44:14 +0100 Original-Received: from localhost ([::1]:56564 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lG2hR-0002Hm-8u for guile-devel@m.gmane-mx.org; Sat, 27 Feb 2021 11:44:13 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50434) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lG2hE-0002Gq-TD; Sat, 27 Feb 2021 11:44:00 -0500 Original-Received: from mout.web.de ([212.227.15.14]:57997) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lG2hC-0007UE-Lo; Sat, 27 Feb 2021 11:44:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1614444233; bh=KVyFpZJwFgPGw1wDI800ta/oh7pfHurf2v/HG7I7hQU=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=IvxhavQeK8j4WpYng1pKKjinoInnL6h8N5zzuKOi/b9tFR2tyvtTg9ypGQJzC8apY 32MKjvzDVJPf0M5L4LTRJzZP8VmzsRMeSUUkCeC9ntb1KSc/f8x6FmyOUR1PVegzKa FgMJ9IRlAX7IRt4jWAG4C8rn+ZnIXpPCkwHrMD0Y= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Original-Received: from fluss ([84.149.81.26]) by smtp.web.de (mrweb005 [213.165.67.108]) with ESMTPSA (Nemesis) id 1MBSB9-1l5pxe2yDf-00D1Kx; Sat, 27 Feb 2021 17:43:53 +0100 In-reply-to: <87o8g58nsv.fsf@web.de> X-Provags-ID: V03:K1:/7D+YYTvf6k1GO/ir4tm87nfafRxLVca5ewrs6FdiPRWbq66xtF XxvH4swvsQ2DcQpLXjnfQTyHeVY0PFkWJx/7BUDvIGN0wa5Orb/QqzC7pWJfOpane0WYkAh AT1j5XGuYbMQB/IQ9qMZTN+3a+NEAb0MrAw6zU6nPYSSSwbHMzk3jv+PFW0OsASZoEiJtW1 7JLK8S+DUKOA8EAfAnFEw== X-UI-Out-Filterresults: notjunk:1;V03:K0:0P4FNEwSbBA=:w+o4qXdRo2M/3eFB8qv2VW a4dUbOvnUO2TL3z2LJldlv9MRJbGqtWyvaOvyB9URs7ziniAYbmMXj3ec/JM9ZKOoVtpGd+xL TU5rJUjiKKek9KXduYXSQTpqmNjwveiSTrm/qLt6xomQ8XZZz9w4bbPjs2OCfLBM/vc0rHwMg 2R0245wnyNalhk+mxGxnFCOzNeANiYPV25g/6kQii3SzqJTBKEuAvmvll0MoIe3VNp3ICd6WI oUubyraMZsQjkK/CdryHy1a0jv/b1hEmrQ1oASWUWVI3AvOOBddh/0H6x3f5LXUGSUvBA6YVt xUJUUzBWgKng1jwQoN28ZqQG4/79eIP0EzCezJG2LiEmQ0kMqeJ75+arlOTHrJRQCM97rfqOS gupo62ZZ/vuZpD1/Cmdoqh4OQSSsY8HS/GC6HqIVNFXEDI0e3loUI2mqXHti1qtpkb4H3Clca ikQEic9JmuU8ABy1qjW4ElcVyeEYLQ+ExgU2qpDJHGJJH2L1ZvjbTiRjbcBNyRl1rkL6cq0DR ck045/FYeMBvOXJ5pZBEHDADteomnLhwGYd9OyKHG34RhXd5Yb5mQhDNg/D5Ppn463K4nFcId PezKv0cbYL5324uGFo+Hcb05baDNlHiLPkX92kfkI2QlXYbTu2jG08aUnaE2Sz6ygKpsinrhd OdeVaL5KqtPV83oD0MAFYWKJfhHiEx/2arEFiLCM9j39Eq9dPSkVWFUz7hem750dQl2Kd3VTN IRnFMCC5GF5y9g2QeNh2z4zK3muIUe19XRWVytkdxjbp9o8RGQHGcAcMdJwyyDXIYX1ZNpY9 Received-SPF: pass client-ip=212.227.15.14; envelope-from=arne_bab@web.de; helo=mout.web.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.io gmane.lisp.guile.devel:20679 gmane.lisp.guile.user:17326 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable PS: Still it is important to get this code fast, because it is the fallback for all situations where I cannot cheat and only calculate a subset of the trust, and it is required at startup. Dr. Arne Babenhauserheide writes: > Hi Linus, > > Linus Bj=C3=B6rnstam writes: >> I had a look and there is quite a lot you can do. Just out of >> curiosity: how much is GC time? You are generating a lot of >> intermediate data. Instead of vector-append! maybe use something like >> the new vectorlist srfi? If you show me a flame graph and give me some >> test data I can have a go! > > Thank you! > > I expect GC time to be roughly half during the trust calculation (much > more during import), because this fills up two cores. > > I can=E2=80=99t easily give you a flame graph (except if you can tell me = how to > generate it), but I now pushed a version in pure Scheme (instead of > wisp) that runs a huge test: > > hg clone https://hg.sr.ht/~arnebab/wispwot > cd wispwot > ./run-wispwot.scm --test > > This imports 200k trust edges (which takes a lot of time) and then does > three full trust calculations (which each take around 70s). > > The most important limitation for optimization is that the memory use > must stay reasonable with 64k IDs which each have 1000 trust edges. This > memory use is what the current code is optimized for (that=E2=80=99s why = there > are all those u8vectors and u16vectors). > > For the math: at 64 million trust edges, this should currently require > (in theory) less than 200MiB of memory for the actual trust structure (2 > byte for the id, 1 byte for the trust). > > Going up to 300MiB for time-optimization would still be viable, but it > shouldn=E2=80=99t go beyond that (there are algorithmic options to reduce= the > amount of work done per update, so the runtime of the current code is > important but it is not a life-and-death situation). > > Best wishes, > Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJEBAEBCAAuFiEE801qEjXQSQPNItXAE++NRSQDw+sFAmA6dsgQHGFybmVfYmFi QHdlYi5kZQAKCRAT741FJAPD6yX/EADSdf23Ugo9zisazIViZaONQGtnethNtlcf K/89y8vtYPg0fqlIj3Ir7cEULaL++ezpptMmvR15p8eMgwsMBJ4vuZeChXWlE8ii mXc02+ScBhqkwWIIgXnKcdiWFcmD1Ec+meHpdj6Mscm7TA696oYKOKpT0bjRxplg J8o0UeFv/IHF35AjyfATNIGq0zWvlK1jvO+sUEf/J+c8qaNj83Q51dGA6xy9hyjW nL/JXoYJ0bpDBxJAtfJnf3xpZJdhB+5cCrxt6VS2wlsgjo3nucGsAKlpSmqIwrNd hT97OLt8iNJcyRxn1YVpKtz8SZgdMTOkxdpiWUJClmAvKw3ucqMY4c7KkLIYVWYu AWZ+cmAQJlbXU9mWqaE7sUp2vnvR7/MNGwtfSKVTsOWneuEpjPYFEIHyRYxnYcw2 CwUoN55OUyEebmupj3SKmvwVcRdYS0vOqGfgVwir9X+JIr/Rnlg/zYMWJoiYntv6 wtChBeEjmBWCw7w4KKkBraHdF8gFG2ERpcJc8ln7Ot07sstW6f9Cp9xKRXOclyXd yIDTj4INaKw6LBG8/jWT0djgMGeru1PK5M6K1cbYRnTIIeZ8p6tEKZZTCSHmPEX6 FoHNt2wvHP+znbk0K2LbZCbvxmh1Wd9nuppCAZVl5uVpZKceAr9M0q1qUoRyPJGI d9URPYLZhojEBAEBCAAuFiEE3Si95tmHXKvOSosd3M8NswvBBUgFAmA6dsgQHGFy bmVfYmFiQHdlYi5kZQAKCRDczw2zC8EFSD/sBACRsq0/1LB+Qat1qmUFP6LgPMXO o6pYjwufnayPT+cNzUwKgZ2gmQfwtlM6g1wu7OcYUOCz80NTjLQuW+4IqCO+CRO9 c1nNs5mV1cY9ipCtAUexVT0FeNoOCoRMfT/dWb7Zw+vwwQRlmg0593eB61PuSYmq z+8o6wt7mbN/R0a/Gg== =mB9i -----END PGP SIGNATURE----- --=-=-=--