From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: "Dr. Arne Babenhauserheide" Newsgroups: gmane.lisp.guile.user,gmane.lisp.guile.devel Subject: Re: Guile Potluck 2021 Date: Sat, 27 Feb 2021 17:42:40 +0100 Message-ID: <87o8g58nsv.fsf@web.de> References: <1144e4d9920845eb4f49e6e099a306e85735bd01.camel@yahoo.com> <87tupx97ey.fsf@web.de> <10585499-c6ef-4cce-b576-60940b3aa4d9@www.fastmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="5798"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.15; emacs 27.1 Cc: guile-user , guile-devel@gnu.org To: Linus =?utf-8?Q?Bj=C3=B6rnstam?= Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Sat Feb 27 17:43:02 2021 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lG2gI-0001QM-RL for guile-user@m.gmane-mx.org; Sat, 27 Feb 2021 17:43:02 +0100 Original-Received: from localhost ([::1]:55258 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lG2gH-0001iR-Rg for guile-user@m.gmane-mx.org; Sat, 27 Feb 2021 11:43:01 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50134) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lG2g6-0001iG-H8; Sat, 27 Feb 2021 11:42:50 -0500 Original-Received: from mout.web.de ([212.227.15.3]:58375) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lG2g4-0006zk-NB; Sat, 27 Feb 2021 11:42:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1614444163; bh=qYdelG68VYWAEsrXMpFeScl1myqiFiBzObn+ZmyGL5o=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=PbPvkp2teP/JuYwvHH1ZJJjahLvRxiNYS4Zxc4W5YOG/rZsJ+Is3bJAYidcoWnd1h kHwm/zIpEycJmk9D4E2n3ISK6bR4QgHt5kUasplUym2jQD4X6f4GfGQ47CLekG0Wkh 5gvBAAQjt8Wgjeh9wUcuUobCVkbQqGvcSrsVY7Uo= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Original-Received: from fluss ([84.149.81.26]) by smtp.web.de (mrweb003 [213.165.67.108]) with ESMTPSA (Nemesis) id 0MZlNu-1lVvP311Hq-00LTuh; Sat, 27 Feb 2021 17:42:43 +0100 In-reply-to: <10585499-c6ef-4cce-b576-60940b3aa4d9@www.fastmail.com> X-Provags-ID: V03:K1:ocI2ZFMQKkpIm7VfLUamrVsS5Pa/d3IoRPqCi+jdVQR3F09GCU3 6RQ8hTV1ygiShrdrdk//T0sp3utJ0aXpiUqoWrrIdMsa/avTSDHJug/s7qtOIkbuta/vN2J vm28zZ3TDeNdL1+4WSQVMxPakZEQ+Kg4l1hdPnUWvEWceR6rQXlC9flux4x2oO+vIMvU5fa Lise5fM/7EWttS9PO+evA== X-UI-Out-Filterresults: notjunk:1;V03:K0:GfhYK99bph0=:1Cuh6/dF6jEdyKZrpY8ehG 8gsEUN8CkvlhZMet7JkRmca0S227qV/g8vzaJahAd3fY/NwGHl9VIgxcUMY7gjfL4f5kWsqeO yS0HPnQS2rP/YBaund2YH0FaKzxob/6mnrNY0WQO9NFMypE2UtabPIokBoGVQcBdVh6uPFfZI eNxuFKEg+B3yZhWjs4a9hVQuDcnQ6uds7elE0TvG3n0UQX2IjzlI6bactU8quKmdq/SH38vbs eKBm6BFiu/VdrgwaNDTnnuRwRDdHxfQBsIIse4DMrYcT3UnxQ1ruYjyCaoOGeqmpgD7EgdW+t 2b0qOy1/+mtbh5RCfXgPRpoDudfdn00SafAsU7A7mctJOjY3Sgh4rjOA/Br7PLzRfqmgxmPYv 42U6NKpqviXLWeyGf4FQpcuc2K4kySAXEpr77eJY5aMUPmiA3+gxV9lQmXQAdGOfPi3mMvyl7 V9/JFbr5CWghYGiz8p2/jEiKJMunRfMznEN5olswBwbX7kDIsPjEmogRG9ygkLGr85eYkN/xQ QDzc/g88t8McrnnE96Cf2yBbzYQgCkhHszgTgSr8EBajVTlUYfQbWNInPhqz/olhiR80nXKOf UYVu42cYzR4BQPUePzoQ9F3xfThHZXQ2fjRhB3bn0aWAqVYrudlDc4ewwBuuqH9PRccqHf9Zj nbTEwpSofWnPI6A8lbCqRVPWFvJ0XqaLLzCenFpJaD8Og8vZHEwC3IBHzHeos5m3u/I7mhrzq GtA/DIwBmiy/E3GjQr14o/vWZpAIK/4IZbreb7FjGzcHy7FE8eUvTE7bjx4mXH8DkczacMzg Received-SPF: pass client-ip=212.227.15.3; envelope-from=arne_bab@web.de; helo=mout.web.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:17325 gmane.lisp.guile.devel:20678 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Linus, Linus Bj=C3=B6rnstam writes: > I had a look and there is quite a lot you can do. Just out of > curiosity: how much is GC time? You are generating a lot of > intermediate data. Instead of vector-append! maybe use something like > the new vectorlist srfi? If you show me a flame graph and give me some > test data I can have a go! Thank you! I expect GC time to be roughly half during the trust calculation (much more during import), because this fills up two cores. I can=E2=80=99t easily give you a flame graph (except if you can tell me ho= w to generate it), but I now pushed a version in pure Scheme (instead of wisp) that runs a huge test: hg clone https://hg.sr.ht/~arnebab/wispwot cd wispwot ./run-wispwot.scm --test This imports 200k trust edges (which takes a lot of time) and then does three full trust calculations (which each take around 70s). The most important limitation for optimization is that the memory use must stay reasonable with 64k IDs which each have 1000 trust edges. This memory use is what the current code is optimized for (that=E2=80=99s why th= ere are all those u8vectors and u16vectors). For the math: at 64 million trust edges, this should currently require (in theory) less than 200MiB of memory for the actual trust structure (2 byte for the id, 1 byte for the trust). Going up to 300MiB for time-optimization would still be viable, but it shouldn=E2=80=99t go beyond that (there are algorithmic options to reduce t= he amount of work done per update, so the runtime of the current code is important but it is not a life-and-death situation). Best wishes, Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJEBAEBCAAuFiEE801qEjXQSQPNItXAE++NRSQDw+sFAmA6doIQHGFybmVfYmFi QHdlYi5kZQAKCRAT741FJAPD61xlEACeVJDe+uMNrU/vEQamvTKq1Cu4GiCQbUlb j2o6Rma3Io+9bNbA41xJ3iUOMZnrjRBUdNGx/81NzTZz5RyiUGgowtDzG6yovs4Z eDW1BXytr7EW2EE27TsohTzPXE+iWJh+hnqLvN+HHzOOJczI9URd0rwqaJoOGqvK CNHF+gwFUwnGnjYsEv2mP0vo/I7tXuyAa2O7NNvZmTkxb2+BTxwxebC2+PakfWW1 5oow68Zu257vAl1eWJUQJs7ByS8fT5gexz8C5n2cVMihO6XYBb6a0Aj3NLhmQSwn NEDufuB/uxCmcxf3UalE0eek2uEM2G628J0lYubbHn+HReW6qwox3S2SsmR2TEm1 Z36Uo6SbjIz1lAWtvVPEgD730xHVA/sVqz4qk8XH5LETuU1J/rbmTpFOc5cveoiu 57pKG2kxNMUo2Q2PXzleJu8IfmeWjyecE/1rVXdiTFSyfcBVD8ZwOKr3ho3/iSEe 9IHh/z2v/dsJ0DvDmHK1FAM6HOKpgH2bSQTn+IyMv6XIPP/NMZbCjmhPz21lFLQN U9WUUmoD0FPf5sA6RLUuyJ4wdUTZ1gBkXSg9CVhzIngTJMBzhwk7uPFYT9E23yLK juxbnGSE44uYOAo8lqJJ/G2W+ppE3WtQX34Xo4S7eTTvPoe1B0UqZkYv6f7ZDhvj c1mophiLHYjEBAEBCAAuFiEE3Si95tmHXKvOSosd3M8NswvBBUgFAmA6doIQHGFy bmVfYmFiQHdlYi5kZQAKCRDczw2zC8EFSA9cA/0d2iANOzodP2n0qLW1LjKXfOOd RRI8ZVgkdsJSwLjTteBozc7yFBIcNmQjXG1+RcdYjLQ9DjwQCPaQFY9j4hv/W/hV x3zw1S6MubZ7bcB1MmfkBfnah92I/F7QYZ3tTi47LHWL4awn4Qy1llU20evAyjQJ G+PpqJ+fxk2NAEn67Q== =9cF9 -----END PGP SIGNATURE----- --=-=-=--