From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Andrea Corallo Newsgroups: gmane.emacs.bugs Subject: bug#41077: [feature/native-comp] virtual memory exhausted Date: Wed, 06 May 2020 20:12:52 +0000 Message-ID: References: <878si768l2.fsf@gmail.com> <87d07jfm2o.fsf@gmail.com> <87d07hcfpm.fsf_-_@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="34502"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) Cc: 41077-done@debbugs.gnu.org To: =?UTF-8?Q?K=C3=A9vin?= Le Gouguec Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed May 06 22:13:15 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jWQPp-0008oT-6L for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 06 May 2020 22:13:13 +0200 Original-Received: from localhost ([::1]:43590 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWQPo-00057W-1b for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 06 May 2020 16:13:12 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:56338) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWQPg-00057M-3J for bug-gnu-emacs@gnu.org; Wed, 06 May 2020 16:13:04 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:57944) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jWQPd-0002yI-Vs for bug-gnu-emacs@gnu.org; Wed, 06 May 2020 16:13:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jWQPd-0002pH-PA for bug-gnu-emacs@gnu.org; Wed, 06 May 2020 16:13:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Andrea Corallo Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 06 May 2020 20:13:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41077 X-GNU-PR-Package: emacs Original-Received: via spool by 41077-done@debbugs.gnu.org id=D41077.158879597810854 (code D ref 41077); Wed, 06 May 2020 20:13:01 +0000 Original-Received: (at 41077-done) by debbugs.gnu.org; 6 May 2020 20:12:58 +0000 Original-Received: from localhost ([127.0.0.1]:41257 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jWQPa-0002p0-FK for submit@debbugs.gnu.org; Wed, 06 May 2020 16:12:58 -0400 Original-Received: from mx.sdf.org ([205.166.94.20]:60274) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jWQPY-0002op-7B for 41077-done@debbugs.gnu.org; Wed, 06 May 2020 16:12:57 -0400 Original-Received: from sdf.org (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTPS id 046KCq5Z029660 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits) verified NO); Wed, 6 May 2020 20:12:52 GMT Original-Received: (from akrl@localhost) by sdf.org (8.15.2/8.12.8/Submit) id 046KCqNH011495; Wed, 6 May 2020 20:12:52 GMT In-Reply-To: (Andrea Corallo's message of "Wed, 06 May 2020 15:06:55 +0000") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:179844 Archived-At: Andrea Corallo writes: > K=C3=A9vin Le Gouguec writes: > >> Andrea Corallo writes: >> >>> All versions should work but at this point I'd go for releases/gcc-10 or >>> releases/gcc-9.3.0 >> >> Thanks, I went with 9.3.0. Now the "ELC+ELN" steps start smoothly, >> unfortunately I don't think the little buster can finish the job :( >> >> I first ran a -j2 build which crashed after 14 hours while compiling >> char-fold.elc. I didn't get anything more precise than "virtual memory >> exhausted: Cannot allocate memory" on the console. >> >> Thinking it might help to compile only one file at a time, after >> removing the temporary files related to char-fold.el[1], I started a -j1 >> build which picked up where the previous build left off, i.e. with >> char-fold.elc. >> >> Unfortunately that file alone seems to be too much for my system to >> handle; compilation ended with the same "virtual memory exhausted" error >> after less than an hour and a half. >> >> I recorded some information related to memory usage during this second >> run (cf. attached graph[2]). My takeway is that at some point, the >> compilation process's memory usage skyrockets, until the system's memory >> (2GB RAM + 2GB swap) is completely exhausted. >> >> >> I haven't reopened/created a new issue because I'm not sure there's a >> way forward; let me know if you'd like me to perform some more in-depth >> profiling. >> >> >> [1] lisp/char-fold.elc=E2=80=A6 and lisp/eln-i686-pc-linux-gnu-=E2=80=A6= /char-fold=E2=80=A6.eln. >> >> [2] The graph shows: >> - the VSZ of the process using the most virtual memory, >> - the RSS of the process using the most virtual memory, >> - the "available" column shown by free(1) on the "Mem:" line, >> - the "used" column shown by free(1) on the "Swap:" line, >> - the name of the process using the most virtual memory. >> >> Measurements were taken every minute. I can upload the sources for >> this graph (measurement script, measurements, and plotting script) >> if that helps. > > Hi K=C3=A9vin, > > thanks for the great report. > > I suspect memory usage on certain few compilation units goes high > correlated with compile time. > > At the time I've got some measures and these were my results on this. > > 48:32.21 leim/quail/ZIRANMA.el > 19:52.61 leim/quail/ECDICT.el > 19:47.17 leim/quail/ARRAY30.el > 19:16.88 leim/quail/tsang-cns.el > 19:07.88 leim/quail/tsang-b5.el > 18:41.66 org/org.el > 15:25.17 leim/quail/4Corner.el > 14:34.79 leim/quail/PY-b5.el > 14:18.76 char-fold.el > 13:58.44 leim/quail/ZOZY.el > 13:58.04 leim/quail/ETZY.el > 13:10.27 gnus/gnus-sum.el > 11:25.28 leim/quail/quick-cns.el > 11:13.41 leim/quail/quick-b5.el > 8:50.48 leim/quail/TONEPY.el > > I believe both measures suggests we should should black list manually > some of these compilation units, at least the one in "leim/". I'll push > a patch for that. > > I'm not sure was mentioned in this thread but you can work around this > now using by speed 0 or fast boot (or a combination of these). > > make BYTE_COMPILE_EXTRA_FLAGS=3D'--eval "(setq comp-speed 0)"' > > Thanks! > > Andrea > > -- > akrl@sdf.org I must have replied privately to this so I'm quoting my previous mail. That said I've just pushed a commit that exclude from native compilation all the leim db files. Overall compile time looks went down by a factor two! Hope this helps. If you could get a measure of the memory consumption as well that would be great. Thanks Andrea --=20 akrl@sdf.org