From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Pip Cet Newsgroups: gmane.emacs.devel Subject: Re: Concurrent GC via `fork` (was: Opportunistic GC) Date: Mon, 8 Mar 2021 16:25:57 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="5728"; mail-complaints-to="usenet@ciao.gmane.io" Cc: eliz@gnu.org, emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Mar 08 17:27:44 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lJIjP-0001N7-7y for ged-emacs-devel@m.gmane-mx.org; Mon, 08 Mar 2021 17:27:43 +0100 Original-Received: from localhost ([::1]:36420 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lJIjO-0008Rz-9Y for ged-emacs-devel@m.gmane-mx.org; Mon, 08 Mar 2021 11:27:42 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37948) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lJIic-0007z2-54 for emacs-devel@gnu.org; Mon, 08 Mar 2021 11:26:54 -0500 Original-Received: from mail-oi1-x22e.google.com ([2607:f8b0:4864:20::22e]:46544) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lJIiI-0003pv-QI; Mon, 08 Mar 2021 11:26:53 -0500 Original-Received: by mail-oi1-x22e.google.com with SMTP id f3so11513494oiw.13; Mon, 08 Mar 2021 08:26:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Hm/qhUT34m5MT50Mv+QyD0AheFY0+/p65Gu/bY1gM3g=; b=WUE9zxiQ0h8xB0BhNpV/c/V8aEcpaNqSx7v1apTxWhniA54+XjYEIbB9LkuXcQx7t1 T2boH12R0/vx0HW9piiC85WteRzMi+K2qU6ZU6S5QlObCXMTV+/PvMKoSU2HlbLYK+NE PmmmD5CSUzDo1i+8Nz/ac+unxaFXIMwBEf1y+KYUnR555stnAvcO4hYX1Yr+WDcS55a3 VXRB/41+GJaH3QQFeETkWlhObyxIDESgxRsZRvaqTpbUQgHd65fc3y9MMxhsq53ZgISE oxI17ceVixXxCyErNkdiX6nRNqn9/8Yh0A9UKZDIFmxdOYoSSWVsYWEkvF7FsbF+KJw0 merw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Hm/qhUT34m5MT50Mv+QyD0AheFY0+/p65Gu/bY1gM3g=; b=b0TIbEwpkpFXZbe9UODRS6INInjyAXjgCnoxj0gozPnvVUnH9gIEETnSNXZPvT/ZVh Wffdgyr+5NeBWNGP3/xDpbttnYtr1UOw6MYwEogII3AC13B+fIged7HAMm5+duAKsWBI xGv+foH9jCOVGWi9XWxtZaz/03X1HqHZM4p+Yr9LEaGM+IyQdKYINMZZ+ysCKCtNbShq Ia7cvcXO8JflZ7o/jICMT8U5yuX8j/ZONJcYjMmX//yITAT+r7g5/1uz7VoyniAzN7cT se00bj5GwB6UtotFAiDqehFnPO2hN16cSV7RXw5bRz4f553n9+kV0fB4oeMejl7zYuTR +HMQ== X-Gm-Message-State: AOAM532My/ClymiCICUL4E1mq83eQREvteMd6jjShbluIUstfpTrDZ9R KnUa1Avjfa8/yn1lZXpX7u87WEHf49LApWCgOyI= X-Google-Smtp-Source: ABdhPJyuud32cMKGINNX/WqyQ+xFC+BEPCj/xnA2/wUuR8lWivqbJhk3655y+JzHKUhCzF+0z0PEEesJ+FamJe88FXU= X-Received: by 2002:aca:4c0f:: with SMTP id z15mr14190510oia.44.1615220793279; Mon, 08 Mar 2021 08:26:33 -0800 (PST) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::22e; envelope-from=pipcet@gmail.com; helo=mail-oi1-x22e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:266204 Archived-At: On Mon, Mar 8, 2021 at 4:03 PM Stefan Monnier wrote: > > Just a random idea: What if we exploit the fact most people have more > > than one CPU core these days, garbage-collect in a separate fork()ed > > process, then do only the sweeping in the main process? > > Sounds like a good idea. Thanks. Sorry for derailing your thread, I should have chosen a new subject. > A concurrent GC which relies on `fork` to take the snapshot. > I don't think we'll be able to always use it, so the synchronous code > path will have to stay, Absolutely. And I do want to mention weak hash tables: they're a problem, because they can resurrect unreachable objects. Worse, while normal weak-key-strong-value hash tables only resurrect objects when you maphash, weak-value hash tables can do so for gethash as well, so that code path would become more expensive and more difficult to translate into native code. > but I think it's definitely worth working on it. I've got it working for conses, I think (as a proof of concept, synchronously, with an extra waitpid()). > It will probably require moving the markbits out of the objects in order > to work well (which would probably be beneficial in the synchronous GC > case as well), but other than that I'd expect it should be possible to > get pretty good performance out of it in the "normal" case: The normal case, for me, is Emacs freeing a lot of memory in a GC cycle, and keeping only a relatively small working set. That should work well with the way markbits are written only for objects that are to be kept. > - most pages should stay shared between the two processes (because the > GC process should basically only modify its stack and its markbit > pages, and the parent process should hopefully not have enough time to > modify a large fraction of its working set). > - the "normal" case I envision is one where there's a fair bit of extra > RAM left in the system and a few CPU cores waiting to help Emacs. Indeed. Apart from hating threads and loving processes, I love nice levels, and I think this would be a good use case for them. For example, I hear there's a new CPU out there that has two separate sets of cores for "efficiency" and "performance", and it would be great if Emacs ran an "efficiency" garbage collection in the background rather than a "performance" garbage collection synchronously. > > Eli, would this qualify as a "small" GC change, and thus be vetoed? > > I like the idea, but its inclusion can't be discussed without > a proof-of-concept patch. I think that's the sensible approach, yes. I certainly wasn't hoping to hear anything _positive_ before submitting a patch. Pip