From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Freja Nordsiek Newsgroups: gmane.lisp.guile.user Subject: Re: How to make GNU Guile more successful Date: Sun, 16 Jul 2017 10:11:57 +0000 Message-ID: <96F78925-F832-40C0-B94A-92E64B34B0EA@gmail.com> References: <87lgtajpkc.fsf@web.de> <87h8ychh9h.fsf@elektro.pacujo.net> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1500199954 15200 195.159.176.226 (16 Jul 2017 10:12:34 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 16 Jul 2017 10:12:34 +0000 (UTC) Cc: "guile-user@gnu.org" To: Marko Rauhamaa Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Sun Jul 16 12:12:29 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWgXN-0003gZ-6t for guile-user@m.gmane.org; Sun, 16 Jul 2017 12:12:29 +0200 Original-Received: from localhost ([::1]:44693 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dWgXS-00085M-Pd for guile-user@m.gmane.org; Sun, 16 Jul 2017 06:12:34 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:41596) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dWgX2-00085H-7F for guile-user@gnu.org; Sun, 16 Jul 2017 06:12:09 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dWgWz-0005AW-0I for guile-user@gnu.org; Sun, 16 Jul 2017 06:12:08 -0400 Original-Received: from mail-wr0-x22c.google.com ([2a00:1450:400c:c0c::22c]:36341) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dWgWy-00059r-MP for guile-user@gnu.org; Sun, 16 Jul 2017 06:12:04 -0400 Original-Received: by mail-wr0-x22c.google.com with SMTP id v60so18043776wrc.3 for ; Sun, 16 Jul 2017 03:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:in-reply-to:references:mime-version:content-transfer-encoding :subject:to:cc:from:message-id; bh=g9wvSQa/HJ++3QuQRBfQK+V3e/00D1goP076YtwjvXE=; b=QA365Gdeo5FwZC6LMP3ErXOKPiMp9SJ2DNosLXsiuWiPcmP8NDvWJ51YMpZ63Hd56q v3IEZYOw1QcobkOU84W7061zR3V3aQ/LIGlNRDiQbW1J+8XR26NBZAh/Sm0kSSSTHjfY iGr1gNaVL6ijMYugpSWcGA7jlMtMBfxAJ/4P9zskYB9DIiP/hI5VnXMWMjsFBW7O2mYP AeyogepTcRzrCKw5ZrV4jt/11nGka6QJGTQhWAQAOg4HF5Sritaytkuy+Rr66OblyDKo Tavnfwey6u8lw0AsUnYpOp5TtirmEcqJURxt5mi3GaI50Bc2N6RNt7yJkW0n3TaDi8wO qzQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:references:mime-version :content-transfer-encoding:subject:to:cc:from:message-id; bh=g9wvSQa/HJ++3QuQRBfQK+V3e/00D1goP076YtwjvXE=; b=f4vNWdddPaxs4wYKNMe9iVWeMTF6Ncmv//NEMTdW0vEhTiNm5j3th+JTFAAngpWOX0 inERlehO7qc7csu7N6khex7zDgWyAFWeQUeh1ggqeMiGpXb4wVoCkEvp81+6h/LHS9LW h//ElwNpvp/tmz9ICqCFLhI3aLeEZ9caQhdGQdu8DT3Fx5FwFA9n5V4Tox1Pio8tbjUf JPUKHtNsXRpSgQARTxbqBNt6LY2bqPqe3JhiWAmLWtFaSSwEz+DUziHx3eTG9FfreFyf 9fYlYO2P6puRH42rycvFTPt6916sFpmb45a128gdl0TGNAULfDYu5zpSbaAHK13ejUto kZYA== X-Gm-Message-State: AIVw113HxZWyMoZJHCNy9m6n/lgDDWnFJUELUXRsC/M1gAYWM729VoQ4 THGytXB+KadlIA== X-Received: by 10.223.150.74 with SMTP id c10mr8346299wra.124.1500199923671; Sun, 16 Jul 2017 03:12:03 -0700 (PDT) Original-Received: from android-5415d01cc55ceda2.local (dslc-082-083-039-055.pools.arcor-ip.net. [82.83.39.55]) by smtp.gmail.com with ESMTPSA id m196sm6391448wma.17.2017.07.16.03.12.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Jul 2017 03:12:02 -0700 (PDT) In-Reply-To: <87h8ychh9h.fsf@elektro.pacujo.net> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::22c X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:13938 Archived-At: I checked the implementation of bytecectors and SRFI-4 in Guile and they ar= e definitely not scanned for pointers=2E But I would say hacking them is no= t a good general solution for this problem=2E They are good and natural dat= a structures for large arrays of numerical data that are standard signed/un= signed integers of various fixed sizes and IEEE floating point numbers, or = structures/unions of these types=2E Using them for things other than that o= r strings/byte-arrays could be error prone, messy, and performance poor=2E Freja Nordsiek On July 16, 2017 11:18:18 AM GMT+02:00, Marko Rauhamaa wrote: >Freja Nordsiek : > >> If I was to hazard a reason for why Guile gets very slow when loading >> 20 GB or more (may or may not be related to it being buggy and >> crashy), my guesses would be a lot of the data when loaded into Guile >> was allocated such that the GC scans it for pointers (using >> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would >vastly >> increase the amount of memory the GC needs to scan every time it >runs=2E > >Good point! > >If you didn't to any C programming, what kind of native Guile data >structures are good for such large random-access storage? At least >arrays haven't specifically been documented for such GC optimization: >l#Arrays>=2E > >Maybe bytevectors would do: anual/html_node/Bytevectors=2Ehtml#Bytevectors>=2E > >Of course constantly encoding to and decoding from a bytevector using >scheme code might be very slow without the help of some binary bulk >formatting facilities for the data records=2E > > >Marko