From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lars Ingebrigtsen Newsgroups: gmane.emacs.devel Subject: Compilation speed Date: Thu, 06 Aug 2020 17:20:13 +0200 Message-ID: <87bljnby36.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26613"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Aug 06 17:21:10 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k3hhe-0006oy-A9 for ged-emacs-devel@m.gmane-mx.org; Thu, 06 Aug 2020 17:21:10 +0200 Original-Received: from localhost ([::1]:34656 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k3hhd-0000UO-D2 for ged-emacs-devel@m.gmane-mx.org; Thu, 06 Aug 2020 11:21:09 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:48936) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k3hgv-0008TH-Uk for emacs-devel@gnu.org; Thu, 06 Aug 2020 11:20:25 -0400 Original-Received: from quimby.gnus.org ([2a01:4f9:2b:f0f::2]:45416) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k3hgs-0001gS-Tm for emacs-devel@gnu.org; Thu, 06 Aug 2020 11:20:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:Date:Subject:To:From: Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=I/U9rV3KMIaO7+NsupVygPb6Rn/AABT9AeXhpeqlpfs=; b=qRR35wcIYa/wgjPFjdZWOeMLoG 5no/vCrIotdQoi0TDyQUkt9epPORRrXH7jXRbYHjEdFkxzolnirQk5L/6rS2ZlAD5bloau2lNxD25 KmOR7gTlWlmUdtPrvbLx0jdNhsq4DbygLK6fxb0jIFt8Zgy19KiTulQLChID8WAXHKn4=; Original-Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo) by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k3hgl-0002qs-8K for emacs-devel@gnu.org; Thu, 06 Aug 2020 17:20:18 +0200 Received-SPF: pass client-ip=2a01:4f9:2b:f0f::2; envelope-from=larsi@gnus.org; helo=quimby.gnus.org X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:253476 Archived-At: A nice thing about new, shiny machines with a bunch of cores is that it makes saying "make bootstrap" faster when I want to check stuff. However, there's a couple of sections that are single-threaded, which makes the machine mostly idle while building, and I wondered whether anybody has looked into these things... I did a bootstrap (with make -j8) and timestamped the lines to see whether anything in particular stands out. [larsi@stories ~/src/emacs/trunk]$ emake bootstrap 2>&1 | ts -s 00:00:00 rm -f libXMenu11.a ./*.o deps/* ... First deleting a lot of stuff... that's fine... 00:00:03 running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 00:00:03 checking for xcrun... no 00:00:03 checking for GNU Make... make [...] 00:00:22 config.status: executing etc-refcards-emacsver.tex commands And then ./configure takes 19 seconds, and that's fully single-threaded, I believe? And... I'm guessing there's no way to get that to be multi-threaded? But I've also long wondered whether anybody has actually pruned the tests lately? Are all those tests (for things that seem really trivial, mostly) really necessary these days? And then compilation of Emacs and the manual starts, and that bit uses all eight cored: 00:00:22 GEN alloca.h [...] 00:00:22 GEN ../../info/eintr.info 00:00:22 GEN sys/random.h 00:00:22 GEN sys/select.h 00:00:22 GEN ../../info/elisp.info 00:00:38 GEN sys/stat.h 00:00:38 GEN sys/time.h 00:00:38 GEN sys/types.h [...] 00:00:50 CC minibuLoading loadup.el (source)... 00:00:50 dump mode: pbootstrap So the build of the temacs binary just takes 28 seconds. 00:00:50 Loading emacs-lisp/byte-run (source)... 00:00:50 Loading emacs-lisp/backquote (source)... [...] 00:01:02 Dump complete And dumping takes 12 seconds. Then some other bits are compiled and then we start the pbootstrap: 00:01:07 ./temacs --batch -l loadup --temacs=pbootstrap This seems to proceed in parallell until we reach: 00:01:17 INFO Scraping files for loaddefs.el... [...] 00:01:54 INFO Scraping files for loaddefs.el...done Which is single-threaded and takes a whopping 37 seconds! Could that possibly be parallelised in some way? Probably be tricky... 00:01:54 Loading loadup.el (source)... [...] 00:01:58 Dump complete Another dump; takes just four seconds. 00:02:00 ELC ../lisp/language/khmer.elc 00:02:00 ELC ../lisp/language/korean.elc [...] 00:02:46 ELC progmodes/cc-styles.elc 00:02:46 ELC progmodes/cc-mode.elc And finally, all the remaining (undumped) .el files are compiled, and this uses all the cores, so it takes just 46 seconds. Hm... well, I guess if somebody wanted to speed up compilation, then the loaddefs scraping is the biggest single item here to look at. But to sum up, (+ 22 12 37 4) => 75 seconds of the Emacs compilation is single-threaded, while 90 seconds are multi-threaded (and as machines get more and more cores, we'll probably see the single-threaded parts take more than 50% of the time spent). -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no