From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Linas Vepstas Newsgroups: gmane.lisp.guile.user Subject: Re: Guile bugs Date: Sat, 9 Sep 2017 18:02:26 -0500 Message-ID: References: <87lgtajpkc.fsf@web.de> <87h8y7ruuz.fsf_-_@gnu.org> <877ez384eu.fsf@elektro.pacujo.net> <87k2333qx9.fsf@gmail.com> <87pocvc5ya.fsf@elektro.pacujo.net> <87poculu2v.fsf@gnu.org> <87inimd9yy.fsf@elektro.pacujo.net> <87k232cu3p.fsf@netris.org> <877ez2co3j.fsf@elektro.pacujo.net> <87wp57mrmb.fsf@elektro.pacujo.net> Reply-To: linasvepstas@gmail.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1504998218 14675 195.159.176.226 (9 Sep 2017 23:03:38 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 9 Sep 2017 23:03:38 +0000 (UTC) Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= , Guile User To: Marko Rauhamaa Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Sun Sep 10 01:03:33 2017 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqomN-0002Xf-Av for guile-user@m.gmane.org; Sun, 10 Sep 2017 01:03:11 +0200 Original-Received: from localhost ([::1]:51120 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqomU-0005wf-Da for guile-user@m.gmane.org; Sat, 09 Sep 2017 19:03:18 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:41936) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqom5-0005wF-8h for guile-user@gnu.org; Sat, 09 Sep 2017 19:02:54 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dqom3-0000dc-Nv for guile-user@gnu.org; Sat, 09 Sep 2017 19:02:53 -0400 Original-Received: from mail-lf0-x22e.google.com ([2a00:1450:4010:c07::22e]:38524) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dqom0-0000c5-RX; Sat, 09 Sep 2017 19:02:49 -0400 Original-Received: by mail-lf0-x22e.google.com with SMTP id q132so11608529lfe.5; Sat, 09 Sep 2017 16:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=EzhkWqcs6rHF6jwwWqOkU8HXloKk0xXxoi17nPklzuk=; b=nhVKPSWBPJV6shHqINpUrLJd5FTQkwyR89v5MlPHF6cJIPxTRbvTE09Y30iLq3fjlx jBHGq3CJVSvhfimoA+wxdVHU3/QsY3DFKjOVLH6t9Vq+KVyL2JpgvNg2fF90p9/rjMed i7zLbWnUziWh/iB1sew+UlnCf4FQtdQxxwOryEP2QxYjVRNWswGv+gDbgs63W02IFwug lG/ej/mzLNmTHloyVesBRqM3KXBxmGIfJGhVRKZIXMMtk9P0/V+ETQ3K7nJm/tz5AkYB XAm5v/j8DGqjTlRdgd7g338gDqcF1Jd/1QQlDxMbPM+UBU3LQETgezOeGj9WGp9mGvGy CbTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=EzhkWqcs6rHF6jwwWqOkU8HXloKk0xXxoi17nPklzuk=; b=bly5wo1mhkWxrk4ItldhXECD1qqCoY/ZRJFrKY4EF3EZc4oO0oleBNS+nEIeT8cnTN rgKzzb45hc5OsW0W67vkwPIhuauhQtenftMCeJs3EwBHK23EaWiPU+P2ZoB7rpIyTBgo C4gsvibpjYN5tR8XYMBvOTYFov6RySxk9Ew0oErCGMdJnz1Jrfeth3c7VNJc7goKqqIA 3moKKCpsRRSSqYmSnamzPs4diMroSrWj2hUVEx8xx789g/riXq7hFxB8SZ7607NkK1ul d3NfRRuJiPd4PMmiwtFvQA55+XaBsO/GECRix6q4OLnHvmilxBf7ZdtQ7q9fdSBrLtyw pp5Q== X-Gm-Message-State: AHPjjUjA+NImYSez56HTbkzZasnEBHn0bjkibw4sWPaiyhmDhIfDqOXU 9mWOW4Uljzh0dZP1Rmwe/32MZJCC9w== X-Google-Smtp-Source: AOwi7QA6d7VZfxylLzVKpk+quBGj5noe2+H9TbAcMmwt7KX5I+gfUBtaDKkxjZGj5yvpl9DJmG7TG91PGPvdjAE9OlQ= X-Received: by 10.46.87.5 with SMTP id l5mr2518952ljb.128.1504998167527; Sat, 09 Sep 2017 16:02:47 -0700 (PDT) Original-Received: by 10.25.44.200 with HTTP; Sat, 9 Sep 2017 16:02:26 -0700 (PDT) In-Reply-To: <87wp57mrmb.fsf@elektro.pacujo.net> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4010:c07::22e X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:14104 Archived-At: Hi Marko, On Sat, Sep 9, 2017 at 5:31 PM, Marko Rauhamaa wrote: > Linas Vepstas : > > So if you used GC_malloc_atomic() in your code, then gc will NOT scan > > that region for pointers. guile-2.2 does this correctly for strings (I > > checked) because strings will never contain pointers. I have not > > checked other uses. > > The question about mmap(2) is addressed more directly by the README > under : > > Any objects not intended to be collected must be pointed to either > from other such accessible objects, or from the registers, stack, > data, or statically allocated bss segments. [...] > > WARNING: pointers inside memory allocated by the standard malloc are > not seen by the garbage collector. Thus objects pointed to only from > such a region may be prematurely deallocated. It is thus suggested > that the standard malloc be used only for memory regions, such as I/O > buffers, that are guaranteed not to contain pointers to garbage > collectible memory. > > [...] > > WARNING: the collector does not guarantee to scan thread-local > storage (e.g. of the kind accessed with pthread_getspecific). The > collector does scan thread stacks > > Now, the question is, can you make your multigigabyte allocation go into > these excluded memory segments? Are you still hit by the pathological GC > behavior you reported? > What, my app, specifically? Uh yeah. All of my c++ objects go through plain-old malloc and should be 100% invisible to guile and to bdwgc and that's perfect. I do have a layer where guile interacts with the C++ code, and all the various bits and pieces are correctly declared everywhere. That code is about 8-10 years old, its been maintained all that time, its been in use all that time, and works without issues. "works without issues" means this: A typical run for me uses some 50-100 threads, maybe 1/2 of these in guile. Due to lock contention, in practice only 3 to 10 of these threads ever get scheduled, and this is fine, and not unexpected. My algos are the opposite of "embrassingly parallel", so low parallelism is expected. Each processing run goes for a few days to weeks or over a month, uses 2 to 50GB RAM, does not leak memory, does not crash, its stable, predictable, does what its supposed to do, terminates cleanly. It does sometimes run much slower than expected (about 1/2x or 1/4x) and I don't know why. I don't usually monitor GC, but when I do, it seems that a lot of cpu time gets wasted there. Sometimes other people also notice this, bitch at me, and propose silly solutions. I've been ignoring that as lower-priority. Only now it's risen up and is biting hard, and I don't understand how or why gc gets triggered in guile. For my app, it seems that it is running far, far too often, and is severely slowing/preventing forward progress. --linas > > > Marko > -- *"The problem is not that artificial intelligence will get too smart and take over the world," computer scientist Pedro Domingos writes, "the problem is that it's too stupid and already has." *