From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: Larger GC thresholds for non-interactive Emacs Date: Fri, 17 Jun 2022 18:53:17 -0400 Message-ID: References: <87wndndbhq.fsf@gmail.com> <83bkuzznws.fsf@gnu.org> <877d5mqmkh.fsf@localhost> <83y1y2utnd.fsf@gnu.org> <87r13up587.fsf@localhost> <83o7yyur0l.fsf@gnu.org> <87leu2p3nu.fsf@localhost> <83leu2uewn.fsf@gnu.org> <87r13qv701.fsf@localhost> <83bkuursya.fsf@gnu.org> <87h74l9jk8.fsf@localhost> <83bkutqb3z.fsf@gnu.org> <9778F176-E724-4E61-B0FB-327BCDD316C0@acm.org> <87sfo4epeo.fsf@localhost> <87bkurrc5e.fsf@localhost> <87bkur72b7.fsf@gnus.org> <874k0j40e7.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="3347"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: Ihor Radchenko , Mattias =?windows-1252?Q?Engdeg?= =?windows-1252?Q?=E5rd?= , Eli Zaretskii , Tim Cross , rms@gnu.org, Alan Mackenzie , emacs-devel To: Lars Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Jun 18 00:54:36 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o2KrK-0000hi-RQ for ged-emacs-devel@m.gmane-mx.org; Sat, 18 Jun 2022 00:54:35 +0200 Original-Received: from localhost ([::1]:47410 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o2KrJ-0005Nq-QW for ged-emacs-devel@m.gmane-mx.org; Fri, 17 Jun 2022 18:54:33 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:54456) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o2KqE-0004Qn-2r for emacs-devel@gnu.org; Fri, 17 Jun 2022 18:53:26 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:40519) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o2KqB-0001IN-3J; Fri, 17 Jun 2022 18:53:24 -0400 Original-Received: from pmg2.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 3C67D80922; Fri, 17 Jun 2022 18:53:20 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg2.iro.umontreal.ca (Proxmox) with ESMTP id 97EA180009; Fri, 17 Jun 2022 18:53:18 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1655506398; bh=b6iDyT8r1mB1FPFMslblEPf9DuNFB3J2gQ8d5C++u1Q=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=OGQ4qnem0JVL1SGkyWpAu6JPVSnNOuSJhFnWZl7E5cdjaZUJTcpjefchZ83ex3tVy v5jcEOm1uML7snHz5NQdgYyhxE+pyN27aDkoxWtGBVuAK/fd/E9R233vzvj6+TnTNE gsWzMJ0FglRoVzpanxD8P1JOLmlHy4TTY6m+D4bt/kfR2O4K3OrQw0urwThom+6d0k FvfmhmuewxmZM8o0GuuOLbGhm8TAteL26uYgz0QbnZb0AdLtmITxNU0uyhasatZMRV MrUYBdPeFzSnjRxYMLoKp+4HZxRo8buNAaxWUGpvC9EGO1kbIxikvjdDtD210K/aiw Pk9DCoUGMOPfg== Original-Received: from alfajor (unknown [45.72.221.51]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 3C2601200A1; Fri, 17 Jun 2022 18:53:18 -0400 (EDT) In-Reply-To: <874k0j40e7.fsf@gnus.org> (Lars Ingebrigtsen's message of "Fri, 17 Jun 2022 20:20:48 +0200") Received-SPF: pass client-ip=132.204.25.50; envelope-from=monnier@iro.umontreal.ca; helo=mailscanner.iro.umontreal.ca X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:291319 Archived-At: >> This suggests that for batch jobs maybe we should bump up >> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0. > Yup. But do you mean in general? I.e., -batch would set that variable > to 2.0? Yup. > Would there be any likely major repercussions That's the question we should investigate as well. > -- i.e., jobs that used to run fine would run out of memory? I added the patch below to try and see a bit more what's going on: diff --git a/src/alloc.c b/src/alloc.c index 55e18ecd77e..d954e928eed 100644 --- a/src/alloc.c +++ b/src/alloc.c @@ -6245,6 +6245,11 @@ garbage_collect (void) consing_until_gc = gc_threshold = consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0); + fprintf (stderr, "GC-%d p=%.1f total=%.1fM thresold=%.1fM\n", + getpid (), XFLOAT_DATA (Vgc_cons_percentage), + total_bytes_of_live_objects () / 1048576.0, + consing_until_gc / 1048576.0); + /* Unblock *after* re-setting `consing_until_gc` in case `unblock_input` signals an error (see bug#43389). */ unblock_input (); and then I ran rm **/*.elc src/bootstrap-emacs lisp/loaddefs.el make -j24 BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage 2.0)' The first thing I see is that with the number of GCs where p=0.1 outweighs those with p=2.0 by a factor 10, so clearly the number of GCs is significantly reduced. I also looked for the number of GCs performed in a given process: % grep 'p=2.0' build-log | sed 's/ .*//' | sort | uniq -c 1 GC-15817 1 GC-15837 1 GC-15841 1 GC-15842 1 GC-15985 1 GC-15990 1 GC-15991 1 GC-15993 1 GC-15995 1 GC-15998 18 GC-16009 18 GC-16021 18 GC-16023 18 GC-16026 18 GC-16028 18 GC-16042 18 GC-16045 18 GC-16048 18 GC-16056 18 GC-16058 1 GC-16062 1 GC-16064 18 GC-16067 17 GC-16079 This says that only 24 processes performed a GC with p=2.0, so the vast majority of the byte-compilation processes finished without performing any GC at all (well, maybe not quite: see below). It's also odd how most performed 0 GC, then 12 performed 1 GC, then 1 performed 17 GCs and 11 performed exactly 18 GCs. What's so magical about 18? Then I looked at those that performed 18 GCs and they all look quite similar: % grep GC-16026 ./+make-2.0.log GC-16026 p=0.1 total=0.5M thresold=0.8M GC-16026 p=2.0 total=3.9M thresold=7.8M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M GC-16026 p=2.0 total=4.1M thresold=9.5M The first GC seems to happen before we set percentage to 2.0 (so apparently all compilation processes performed at least one GC before we set percentage to 2.0 and then the majority of them performed no further GC before exiting). [ So if we set percentage to 2.0 a bit earlier than what happens with `--eval` we may gain yet a bit more time. ] 9.5M is actually exactly 10000000 bytes so I suspect these come from either `cedet/semantic.el` or `international/mule-cmds.el` which both set gc-cons-threshold to (max gc-cons-threshold 10000000). Stefan