From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Freja Nordsiek Newsgroups: gmane.lisp.guile.user Subject: Re: How to make GNU Guile more successful Date: Sun, 16 Jul 2017 10:30:25 +0200 Message-ID: References: <87lgtajpkc.fsf@web.de> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1500193871 17504 195.159.176.226 (16 Jul 2017 08:31:11 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 16 Jul 2017 08:31:11 +0000 (UTC) Cc: "guile-user@gnu.org" To: linasvepstas@gmail.com Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Sun Jul 16 10:31:04 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWexE-0004D4-EH for guile-user@m.gmane.org; Sun, 16 Jul 2017 10:31:04 +0200 Original-Received: from localhost ([::1]:44480 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dWexJ-0000OI-VJ for guile-user@m.gmane.org; Sun, 16 Jul 2017 04:31:09 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:59632) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dWewg-0000E0-Dc for guile-user@gnu.org; Sun, 16 Jul 2017 04:30:31 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dWewf-0000mI-9o for guile-user@gnu.org; Sun, 16 Jul 2017 04:30:30 -0400 Original-Received: from mail-pf0-x22f.google.com ([2607:f8b0:400e:c00::22f]:36643) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dWewf-0000kt-1s for guile-user@gnu.org; Sun, 16 Jul 2017 04:30:29 -0400 Original-Received: by mail-pf0-x22f.google.com with SMTP id q86so63393797pfl.3 for ; Sun, 16 Jul 2017 01:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mNTEt4cWdA4rqaYHXse8fnvAzhw0NbWuq++gz50udM0=; b=Pe+AHIDx4OpuAwbNh5QjnDdBYKHsVSbufCOgGu1Tr6P07iEazOpX5snZCDs72gQ9vN svclHvVDR6iRGd9SIQUnk37jY2UVcnWLff+09P9+JaQ+RkgvOp+6dBhypyCz/NqbbAIy lrB1tDSSu6xsPNiunYA5DApMkIUa/atceMJCzmGR3+hHlgt24t3mCOH+u7Dh/9MA4BVr 0SsnoIAOMcWlXmu6M9Or8+r67+dp3Ev8kuivNjGl95kB0fIgPrPhjAPkfkhQoHJDD/JS +oMDkxkel1WhHHTv52viKfVFHmM3pZTn91cFu0s5YgnPf97ZfnB2vdtZ9pRXqRR3U+O0 sDtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mNTEt4cWdA4rqaYHXse8fnvAzhw0NbWuq++gz50udM0=; b=kZ+chLHI6EPAmrZiAxq07NKyqxbnbi3sTzDz3gFpS9B1mEo4J+ULpeVQhQGp9XKmvi Sv7vPzUtsDlBTJyx/t39QOvmDH2KzxMdX8YkirpHv3Fzi4sGZosEfZZ4RttU1j8Dx30d SG3Vdk0nNI4sGhHbg/CDCJqh+0QKRYzs8WvZ3K+juZmjMen/s+Pan9n0jDe5tlr3mIce VgoAlxoNWY6Ezs+mPV4LY9uagjlsTD+RJXDBefnpFMXyINXIyaXXyMfRWAmlJJX+5/Ed XJI2eRKWyvrjM0TqMnRtiJpkPomDeEhMsbnoHib8IgQiaGz245T4AnB15U4cYwtvAKk1 o9jg== X-Gm-Message-State: AIVw113CCeL4Sx/OC/GXGy8C+aUHSFBO/OgIPEaNtsEHE7b8yVeE9mb7 uT6hqh/rI/knVpfOc3aP3apbMjQGkA== X-Received: by 10.84.130.6 with SMTP id 6mr10865557plc.156.1500193826334; Sun, 16 Jul 2017 01:30:26 -0700 (PDT) Original-Received: by 10.100.151.234 with HTTP; Sun, 16 Jul 2017 01:30:25 -0700 (PDT) In-Reply-To: X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::22f X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:13935 Archived-At: If I was to hazard a reason for why Guile gets very slow when loading 20 GB or more (may or may not be related to it being buggy and crashy), my guesses would be a lot of the data when loaded into Guile was allocated such that the GC scans it for pointers (using scm_gc_malloc instead of scm_gc_malloc_pointerless) which would vastly increase the amount of memory the GC needs to scan every time it runs. Depending on the data types and what is in them, it may be needless for the GC to run through the bulk of the data looking for pointers and this might be a fixable problem. For example, it generally isn't necessary to scan inside strings for pointers so if that is being done, there is something in Guile to fix. If there are really pointers in it (say it is a lot of and/or big lists, vectors, hash tables, etc.) then the GC really does need to scan them, which suggests a different kind of data structure would work around the problem. This is not always doable, and even if doable could take a lot of programmer time. It seems that Go programmers have run into this with very large maps already (see https://github.com/golang/go/issues/9477 and https://groups.google.com/forum/#!topic/golang-nuts/pHYverdFcLc ). No idea how this relates to being buggy or crashy. Freja Nordsiek On Fri, Jul 14, 2017 at 11:54 PM, Linas Vepstas wrote: > On Mon, Feb 13, 2017 at 2:28 PM, Panicz Maciej Godek > wrote: > >> >> someone >> responded critically: "are there out of the box libraries to estimate a >> zero inflated negative >> binomial regression model in guile". Of course, if I knew what a >> zero-inflated >> negative binomial regression model, I could deliver an implementation by >> just explaining >> the notions used in that phrase. > > > Caution: the message below sounds negative. Sorry, I use guile daily and > almost exclusively now. So there ... > > Lack of decent science libraries for scheme is a major stumbling block, for > me. Simply having sine and cosine is not enough. I got excited (a decade > ago) when I realized that guile supported GnuMP, and then rapidly deflated > when I realized it only supported integers and rationals in GnuMP .. I work > with arbitrary-precision floats. Or, I did back then. > > Maybe more important is making guile work well with large-RAM setups. > Currently, I do data analysis, every day, in guile, on datasets that take > 20GB or 40GB -- my current one is 110GB when loaded in RAM, and guile > starts getting buggy, crashy and slow when working at that size. > Sometimes, it starts calling GC half-a-dozen times per second, for no > apparent reason, eating up 6 cores (or more!) doing nothing but GC. Why? > Who knows? Who can tell? > > Yes, I have a machine with 256 GB RAM and a few dozen cores, and SSD's that > hold the data, but every time guile crashes, I have to wait an hour for the > data to reload. I can live with it, but its a dirty secret I would not > share with guile wannabe users. > > String handling in guile is a disaster area: If I give it a > 10-megabyte-long string in utf8, it promptly tries to convert all of that > string in utf32, for utterly pointless reasons. This just makes it slow. > > There are still bugs between GC and the compiler: if call (eval "(some > stuff) (other stuff)") the compiler will try to compile that string (after > it was converted ti utf32!) and if GC happens to run at just that moment, > guile crashes or hangs. These bugs need to be fixed. > > So although its a good start, there's a lot of work left until it can get > to "the next level". And that work can't happen until guile is more > popular. So it's very much chicken-and-egg scenario. > > --linas