From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
To: "Linus Björnstam" <linus.bjornstam@veryfast.biz>
Cc: guile-user <guile-user@gnu.org>, guile-devel@gnu.org
Subject: Re: Guile Potluck 2021
Date: Sat, 27 Feb 2021 17:43:51 +0100 [thread overview]
Message-ID: <87lfb98nqw.fsf@web.de> (raw)
In-Reply-To: <87o8g58nsv.fsf@web.de>
[-- Attachment #1: Type: text/plain, Size: 2053 bytes --]
PS: Still it is important to get this code fast, because it is the
fallback for all situations where I cannot cheat and only calculate
a subset of the trust, and it is required at startup.
Dr. Arne Babenhauserheide <arne_bab@web.de> writes:
> Hi Linus,
>
> Linus Björnstam <linus.bjornstam@veryfast.biz> writes:
>> I had a look and there is quite a lot you can do. Just out of
>> curiosity: how much is GC time? You are generating a lot of
>> intermediate data. Instead of vector-append! maybe use something like
>> the new vectorlist srfi? If you show me a flame graph and give me some
>> test data I can have a go!
>
> Thank you!
>
> I expect GC time to be roughly half during the trust calculation (much
> more during import), because this fills up two cores.
>
> I can’t easily give you a flame graph (except if you can tell me how to
> generate it), but I now pushed a version in pure Scheme (instead of
> wisp) that runs a huge test:
>
> hg clone https://hg.sr.ht/~arnebab/wispwot
> cd wispwot
> ./run-wispwot.scm --test
>
> This imports 200k trust edges (which takes a lot of time) and then does
> three full trust calculations (which each take around 70s).
>
> The most important limitation for optimization is that the memory use
> must stay reasonable with 64k IDs which each have 1000 trust edges. This
> memory use is what the current code is optimized for (that’s why there
> are all those u8vectors and u16vectors).
>
> For the math: at 64 million trust edges, this should currently require
> (in theory) less than 200MiB of memory for the actual trust structure (2
> byte for the id, 1 byte for the trust).
>
> Going up to 300MiB for time-optimization would still be viable, but it
> shouldn’t go beyond that (there are algorithmic options to reduce the
> amount of work done per update, so the runtime of the current code is
> important but it is not a life-and-death situation).
>
> Best wishes,
> Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]
next prev parent reply other threads:[~2021-02-27 16:43 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <be71b3d5db9c43723c749f108dbd1ada7785daf2.camel.ref@yahoo.com>
2021-02-18 17:24 ` Guile Potluck 2021 Mike Gran
2021-02-20 17:38 ` Alex Sassmannshausen
2021-02-21 14:20 ` Dr. Arne Babenhauserheide
2021-02-21 17:21 ` Daniel Tornabene
2021-02-22 5:59 ` Aleix Conchillo Flaqué
2021-02-26 23:19 ` Aleix Conchillo Flaqué
2021-02-27 1:08 ` Mike Gran
2021-02-27 9:39 ` Dr. Arne Babenhauserheide
2021-02-27 10:14 ` Linus Björnstam
2021-02-27 16:42 ` Dr. Arne Babenhauserheide
2021-02-27 16:43 ` Dr. Arne Babenhauserheide [this message]
2021-03-02 7:08 ` Dr. Arne Babenhauserheide
2021-03-01 11:34 ` pukkamustard
2021-03-01 21:14 ` Linus Björnstam
2021-03-01 21:37 ` Aleix Conchillo Flaqué
2021-03-02 0:43 ` Dr. Arne Babenhauserheide
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lfb98nqw.fsf@web.de \
--to=arne_bab@web.de \
--cc=guile-devel@gnu.org \
--cc=guile-user@gnu.org \
--cc=linus.bjornstam@veryfast.biz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).