* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 @ 2021-01-06 20:48 Édouard Debry 2021-01-06 20:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 13+ messages in thread From: Édouard Debry @ 2021-01-06 20:48 UTC (permalink / raw) To: 45705 Hello, On windows 10, I noticed that emacs could use a lot of memory, up to 6Go and sometimes more. The memory consumption goes up and down from time to time while I am not running any specific program in emacs, apart `lsp-java` As my personal computer has 16Go of RAM, I can afford it. Unfortunately, my work computer has much less and the whole compute completely freezes at one point when using emacs. I did not notice that behavior on linux. I do not know if the master branch has the same problem. What could be the problem ? ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-06 20:48 bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 Édouard Debry @ 2021-01-06 20:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-07 14:25 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-08 14:25 ` Eli Zaretskii 0 siblings, 2 replies; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-06 20:55 UTC (permalink / raw) To: Édouard Debry; +Cc: 45705 Édouard Debry <edouard.debry@gmail.com> writes: > Hello, > > On windows 10, I noticed that emacs could use a lot of memory, up to > 6Go and > sometimes more. The memory consumption goes up and down from time to > time while > I am not running any specific program in emacs, apart `lsp-java` > > As my personal computer has 16Go of RAM, I can afford > it. Unfortunately, my work > computer has much less and the whole compute completely freezes at one > point > when using emacs. > > I did not notice that behavior on linux. I do not know if the master > branch has > the same problem. What could be the problem ? Hi Édouard, AFAIK was never proved recently Emacs garbage collector is failing to recall memory, so I guess this is just some Lisp program that is allocating a lot of memory keeping then those objects referenced. Others may have more detaliled info about. Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-06 20:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-07 14:25 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-08 14:25 ` Eli Zaretskii 1 sibling, 0 replies; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-07 14:25 UTC (permalink / raw) To: 45705; +Cc: edouard.debry Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> writes: > Édouard Debry <edouard.debry@gmail.com> writes: > >> Hello, >> >> On windows 10, I noticed that emacs could use a lot of memory, up to >> 6Go and >> sometimes more. The memory consumption goes up and down from time to >> time while >> I am not running any specific program in emacs, apart `lsp-java` >> >> As my personal computer has 16Go of RAM, I can afford >> it. Unfortunately, my work >> computer has much less and the whole compute completely freezes at one >> point >> when using emacs. >> >> I did not notice that behavior on linux. I do not know if the master >> branch has >> the same problem. What could be the problem ? > > Hi Édouard, > > AFAIK was never proved recently Emacs garbage collector is failing to > recall memory, so I guess this is just some Lisp program that is > allocating a lot of memory keeping then those objects referenced. > > Others may have more detaliled info about. > > Andrea > IOW I believe this is a duplicate of bug#43389. Should we merge these? Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-06 20:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-07 14:25 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-08 14:25 ` Eli Zaretskii 2021-01-08 15:50 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 1 sibling, 1 reply; 13+ messages in thread From: Eli Zaretskii @ 2021-01-08 14:25 UTC (permalink / raw) To: Andrea Corallo; +Cc: edouard.debry, 45705 > Date: Wed, 06 Jan 2021 20:55:35 +0000 > Cc: 45705@debbugs.gnu.org > From: Andrea Corallo via "Bug reports for GNU Emacs, > the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> > > > On windows 10, I noticed that emacs could use a lot of memory, up to > > 6Go and > > sometimes more. The memory consumption goes up and down from time to > > time while > > I am not running any specific program in emacs, apart `lsp-java` > > > > As my personal computer has 16Go of RAM, I can afford > > it. Unfortunately, my work > > computer has much less and the whole compute completely freezes at one > > point > > when using emacs. > > > > I did not notice that behavior on linux. I do not know if the master > > branch has > > the same problem. What could be the problem ? > > Hi Édouard, > > AFAIK was never proved recently Emacs garbage collector is failing to > recall memory, so I guess this is just some Lisp program that is > allocating a lot of memory keeping then those objects referenced. IME, 6 GiB is too much for any Lisp program to explain away. Also, the memory allocator we use on Windows is known to return memory to the OS more than glibc on GNU/Linux. I have never seen my Emacs session get anywhere near 1 GiB, let alone more, and my sessions run for many weeks without restarting. Can you tell what kind of memory footprints you see on your system with the native-comp branch, after running the session for several days or more? ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-08 14:25 ` Eli Zaretskii @ 2021-01-08 15:50 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-08 16:10 ` Eli Zaretskii 0 siblings, 1 reply; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-08 15:50 UTC (permalink / raw) To: Eli Zaretskii; +Cc: edouard.debry, 45705 Eli Zaretskii <eliz@gnu.org> writes: >> Date: Wed, 06 Jan 2021 20:55:35 +0000 >> Cc: 45705@debbugs.gnu.org >> From: Andrea Corallo via "Bug reports for GNU Emacs, >> the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> >> >> > On windows 10, I noticed that emacs could use a lot of memory, up to >> > 6Go and >> > sometimes more. The memory consumption goes up and down from time to >> > time while >> > I am not running any specific program in emacs, apart `lsp-java` >> > >> > As my personal computer has 16Go of RAM, I can afford >> > it. Unfortunately, my work >> > computer has much less and the whole compute completely freezes at one >> > point >> > when using emacs. >> > >> > I did not notice that behavior on linux. I do not know if the master >> > branch has >> > the same problem. What could be the problem ? >> >> Hi Édouard, >> >> AFAIK was never proved recently Emacs garbage collector is failing to >> recall memory, so I guess this is just some Lisp program that is >> allocating a lot of memory keeping then those objects referenced. > > IME, 6 GiB is too much for any Lisp program to explain away. > > Also, the memory allocator we use on Windows is known to return memory > to the OS more than glibc on GNU/Linux. I have never seen my Emacs > session get anywhere near 1 GiB, let alone more, and my sessions run > for many weeks without restarting. > > Can you tell what kind of memory footprints you see on your system > with the native-comp branch, after running the session for several > days or more? Hi Eli, consumptions of few GiB is something I've seen more then once for long standing sessions. You might be right in this being a memory leak, indeed I've no prove of that (I think we have none for the other direction either). Assuming it's a bug I don't see a priori why this should be a different one respect the one reported for master. Regards Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-08 15:50 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-08 16:10 ` Eli Zaretskii 2021-01-08 22:02 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 13+ messages in thread From: Eli Zaretskii @ 2021-01-08 16:10 UTC (permalink / raw) To: Andrea Corallo; +Cc: edouard.debry, 45705 > From: Andrea Corallo <akrl@sdf.org> > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Fri, 08 Jan 2021 15:50:28 +0000 > > consumptions of few GiB is something I've seen more then once for long > standing sessions. You might be right in this being a memory leak, > indeed I've no prove of that (I think we have none for the other > direction either). I'm not yet worried about memory leaks, I'm more worried that there's no memory leaks, and instead using libgccjit indeed requires such large memory amounts. Are you sure this is not the case? ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-08 16:10 ` Eli Zaretskii @ 2021-01-08 22:02 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 7:56 ` Eli Zaretskii 0 siblings, 1 reply; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-08 22:02 UTC (permalink / raw) To: Eli Zaretskii; +Cc: edouard.debry, 45705 Eli Zaretskii <eliz@gnu.org> writes: >> From: Andrea Corallo <akrl@sdf.org> >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Fri, 08 Jan 2021 15:50:28 +0000 >> >> consumptions of few GiB is something I've seen more then once for long >> standing sessions. You might be right in this being a memory leak, >> indeed I've no prove of that (I think we have none for the other >> direction either). > > I'm not yet worried about memory leaks, I'm more worried that there's > no memory leaks, and instead using libgccjit indeed requires such > large memory amounts. Are you sure this is not the case? I see, if we are interested in comparing the memory footprint of using shared libraries for functions vs stock byte-code I think we can "statically" compare two sessions after the startup. I've compiled current native-comp with and without --with-nativecomp repeating the experiment with and without X. These are the data-points: | | --without-x | --without-x --with-nativecomp | | |---------+-------------+-------------------------------+------| | -Q | 49M | 92M | 1.9x | | my-conf | 92M | 179M | 1.9x | | | | --with-nativecomp | | |---------+------+-------------------+------| | -Q | 536M | 756M | 1.4x | | my-conf | 585M | 1453M | 2.4x | So yes shared are using considerably more memory, I think this is expected as also the file footprint suggests native code is less dense that byte-code (actually with a quite similar relative results). Indeed *with use the delta should decay as data are the same and there's no difference in its representation*, so this picture should be more on the worst case side than on the average. Also if we want to see a positive side, multiple Emacs sessions will share the majority the pages allocated for the shared libraries. Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-08 22:02 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 7:56 ` Eli Zaretskii 2021-01-09 10:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 13+ messages in thread From: Eli Zaretskii @ 2021-01-09 7:56 UTC (permalink / raw) To: Andrea Corallo; +Cc: edouard.debry, 45705 > From: Andrea Corallo <akrl@sdf.org> > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Fri, 08 Jan 2021 22:02:38 +0000 > > I've compiled current native-comp with and without --with-nativecomp > repeating the experiment with and without X. These are the data-points: > > | | --without-x | --without-x --with-nativecomp | | > |---------+-------------+-------------------------------+------| > | -Q | 49M | 92M | 1.9x | > | my-conf | 92M | 179M | 1.9x | > > > | | | --with-nativecomp | | > |---------+------+-------------------+------| > | -Q | 536M | 756M | 1.4x | > | my-conf | 585M | 1453M | 2.4x | > > So yes shared are using considerably more memory, I think this is > expected as also the file footprint suggests native code is less dense > that byte-code (actually with a quite similar relative results). Thanks for the data points. What about memory usage when there's a background compilation of Lisp going on? GCC is known to be a memory hog in some cases, so I wonder what happens in this case with libgccjit. (Do we allow multiple async compilations, btw? if so, how many concurrent compilations can be running, and how do we or the user control that?) Also, what are the numbers for a session that has been running for several days? I understand that it would be hard for you to collect such numbers about all the configurations, but could you show the growth of the configuration you are routinely using, which I presume is --with-x --with-nativecomp and with your config? As your numbers above show, it starts at 1.5 GiB, but what is the footprint after a day or a week? > Indeed *with use the delta should decay as data are the same and there's > no difference in its representation*, so this picture should be more on > the worst case side than on the average. That's why I asked to see the memory footprint after the session has run for some time, yes. > Also if we want to see a positive side, multiple Emacs sessions will > share the majority the pages allocated for the shared libraries. Assuming those sessions run the same binary of Emacs, yes. Otherwise, only some of them will be shared, the ones that are in common public directories, and even that only if the running Emacsen are of the same version. Most of the shared code is in the pdumper file, btw, which is outside of the picture for the purposes of this discussion. Thanks. ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-09 7:56 ` Eli Zaretskii @ 2021-01-09 10:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 11:55 ` Eli Zaretskii 0 siblings, 1 reply; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 10:55 UTC (permalink / raw) To: Eli Zaretskii; +Cc: edouard.debry, 45705 Eli Zaretskii <eliz@gnu.org> writes: >> From: Andrea Corallo <akrl@sdf.org> >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Fri, 08 Jan 2021 22:02:38 +0000 >> >> I've compiled current native-comp with and without --with-nativecomp >> repeating the experiment with and without X. These are the data-points: >> >> | | --without-x | --without-x --with-nativecomp | | >> |---------+-------------+-------------------------------+------| >> | -Q | 49M | 92M | 1.9x | >> | my-conf | 92M | 179M | 1.9x | >> >> >> | | | --with-nativecomp | | >> |---------+------+-------------------+------| >> | -Q | 536M | 756M | 1.4x | >> | my-conf | 585M | 1453M | 2.4x | >> >> So yes shared are using considerably more memory, I think this is >> expected as also the file footprint suggests native code is less dense >> that byte-code (actually with a quite similar relative results). > > Thanks for the data points. > > What about memory usage when there's a background compilation of Lisp > going on? GCC is known to be a memory hog in some cases, so I wonder > what happens in this case with libgccjit. In June we changed the way we store immediate objects in the shared and this makes the compilation way lighter on the GCC side (both in time and memory). I've no precise data on this other than the experimental observation that compiling all Elisp files in Emacs on 32bit systems is not anymore an issue. This IIUC implies that the memory footprint for each compilation is always < 2GB. As a note: in all cases except bootstrap the final pass (the one driving libgccjit) is executed as a sub-process, this to protect us from eventual GCC leaks and not to generate unnecessary fragmentation. In async compilation we indeed run all the compilation (also the Lisp computation) in the child process, so compiling should not have impact on the memory footprint of the main Emacs session. > (Do we allow multiple async compilations, btw? if so, how many > concurrent compilations can be running, and how do we or the user > control that?) Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2> > Also, what are the numbers for a session that has been running for > several days? I understand that it would be hard for you to collect > such numbers about all the configurations, but could you show the > growth of the configuration you are routinely using, which I presume > is --with-x --with-nativecomp and with your config? As your numbers > above show, it starts at 1.5 GiB, but what is the footprint after a > day or a week? ATM I can provide this number, this is an Aarch64 daemon compiled with '--without-x' with an up-time of 25 days and is showing a footprint of 765M. >> Indeed *with use the delta should decay as data are the same and there's >> no difference in its representation*, so this picture should be more on >> the worst case side than on the average. > > That's why I asked to see the memory footprint after the session has > run for some time, yes. The hard part is to have a reference to compare against as the memory footprint is strictly connected to the usage. One with very regular working habits should work like one week on vanilla and one week on native-comp to make a comparison. I've no regular working habits so I fear I'm not the best fit for this comparison. Thanks Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-09 10:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 11:55 ` Eli Zaretskii 2021-01-09 12:37 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 13+ messages in thread From: Eli Zaretskii @ 2021-01-09 11:55 UTC (permalink / raw) To: Andrea Corallo; +Cc: edouard.debry, 45705 > From: Andrea Corallo <akrl@sdf.org> > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Sat, 09 Jan 2021 10:55:23 +0000 > > > What about memory usage when there's a background compilation of Lisp > > going on? GCC is known to be a memory hog in some cases, so I wonder > > what happens in this case with libgccjit. > > In June we changed the way we store immediate objects in the shared and > this makes the compilation way lighter on the GCC side (both in time and > memory). I've no precise data on this other than the experimental > observation that compiling all Elisp files in Emacs on 32bit systems is > not anymore an issue. This IIUC implies that the memory footprint for > each compilation is always < 2GB. You assume that the compilations are all done serially? AFAIK, most people build Emacs with "make -jN", so parallel compilation is an important use case. I guess we will have to collect the information about that, if you say we don't have it now. > As a note: in all cases except bootstrap the final pass (the one driving > libgccjit) is executed as a sub-process, this to protect us from > eventual GCC leaks and not to generate unnecessary fragmentation. In > async compilation we indeed run all the compilation (also the Lisp > computation) in the child process, so compiling should not have impact > on the memory footprint of the main Emacs session. That's fine, but the memory footprint of such a subprocess is also of interest, as it could be relevant to the overall memory pressure of the OS, and thus indirectly on the parent Emacs process as well. > > (Do we allow multiple async compilations, btw? if so, how many > > concurrent compilations can be running, and how do we or the user > > control that?) > > Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2> Thanks. This needs further tuning, IMO, both per the FIXME (i.e. provide a primitive to return the number of execution units), and wrt the default value being half of the available units. We should pay attention to the system's load average as well, I think. > > Also, what are the numbers for a session that has been running for > > several days? I understand that it would be hard for you to collect > > such numbers about all the configurations, but could you show the > > growth of the configuration you are routinely using, which I presume > > is --with-x --with-nativecomp and with your config? As your numbers > > above show, it starts at 1.5 GiB, but what is the footprint after a > > day or a week? > > ATM I can provide this number, this is an Aarch64 daemon compiled with > '--without-x' with an up-time of 25 days and is showing a footprint of > 765M. OK, thanks. > The hard part is to have a reference to compare against as the memory > footprint is strictly connected to the usage. One with very regular > working habits should work like one week on vanilla and one week on > native-comp to make a comparison. I've no regular working habits so I > fear I'm not the best fit for this comparison. I agree, these numbers still need to be collected. Maybe we should ask on emacs-devel that people who use the branch report their numbers? ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-09 11:55 ` Eli Zaretskii @ 2021-01-09 12:37 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 17:26 ` Kévin Le Gouguec 0 siblings, 1 reply; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 12:37 UTC (permalink / raw) To: Eli Zaretskii; +Cc: edouard.debry, 45705, Kévin Le Gouguec Eli Zaretskii <eliz@gnu.org> writes: >> From: Andrea Corallo <akrl@sdf.org> >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Sat, 09 Jan 2021 10:55:23 +0000 >> >> > What about memory usage when there's a background compilation of Lisp >> > going on? GCC is known to be a memory hog in some cases, so I wonder >> > what happens in this case with libgccjit. >> >> In June we changed the way we store immediate objects in the shared and >> this makes the compilation way lighter on the GCC side (both in time and >> memory). I've no precise data on this other than the experimental >> observation that compiling all Elisp files in Emacs on 32bit systems is >> not anymore an issue. This IIUC implies that the memory footprint for >> each compilation is always < 2GB. > > You assume that the compilations are all done serially? AFAIK, most > people build Emacs with "make -jN", so parallel compilation is an > important use case. Yeah, we can say this loose information is only a per compilation unit upper-bound. > I guess we will have to collect the information about that, if you say > we don't have it now. I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to produce quite accurate results on memory footprint during the compilation process. Perhaps he has time and he's so kind to gather some data on the current state, that would be extremely helpful. >> As a note: in all cases except bootstrap the final pass (the one driving >> libgccjit) is executed as a sub-process, this to protect us from >> eventual GCC leaks and not to generate unnecessary fragmentation. In >> async compilation we indeed run all the compilation (also the Lisp >> computation) in the child process, so compiling should not have impact >> on the memory footprint of the main Emacs session. > > That's fine, but the memory footprint of such a subprocess is also of > interest, as it could be relevant to the overall memory pressure of > the OS, and thus indirectly on the parent Emacs process as well. Indeed, I thought was relevant to make you aware of this mechanism. >> > (Do we allow multiple async compilations, btw? if so, how many >> > concurrent compilations can be running, and how do we or the user >> > control that?) >> >> Yes see <http://akrl.sdf.org/gccemacs.html#org91858b2> > > Thanks. This needs further tuning, IMO, both per the FIXME > (i.e. provide a primitive to return the number of execution units), > and wrt the default value being half of the available units. We > should pay attention to the system's load average as well, I think. Agree, I guess we'll be able to tune it better when we'll have more data. We could even think of using the memory pressure on the system to make such a decision. >> > Also, what are the numbers for a session that has been running for >> > several days? I understand that it would be hard for you to collect >> > such numbers about all the configurations, but could you show the >> > growth of the configuration you are routinely using, which I presume >> > is --with-x --with-nativecomp and with your config? As your numbers >> > above show, it starts at 1.5 GiB, but what is the footprint after a >> > day or a week? >> >> ATM I can provide this number, this is an Aarch64 daemon compiled with >> '--without-x' with an up-time of 25 days and is showing a footprint of >> 765M. > > OK, thanks. > >> The hard part is to have a reference to compare against as the memory >> footprint is strictly connected to the usage. One with very regular >> working habits should work like one week on vanilla and one week on >> native-comp to make a comparison. I've no regular working habits so I >> fear I'm not the best fit for this comparison. > > I agree, these numbers still need to be collected. Maybe we should > ask on emacs-devel that people who use the branch report their > numbers? Is a good idea, my fear would be only to have very noisy or hard to interpret results. A priori I think would be nice to collect such data also for master too. Thanks Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-09 12:37 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 17:26 ` Kévin Le Gouguec 2021-01-09 19:41 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 13+ messages in thread From: Kévin Le Gouguec @ 2021-01-09 17:26 UTC (permalink / raw) To: Andrea Corallo; +Cc: edouard.debry, 45705 [-- Attachment #1: Type: text/plain, Size: 2626 bytes --] Andrea Corallo <akrl@sdf.org> writes: > Eli Zaretskii <eliz@gnu.org> writes: > >>> From: Andrea Corallo <akrl@sdf.org> >>> >>> In June we changed the way we store immediate objects in the shared and >>> this makes the compilation way lighter on the GCC side (both in time and >>> memory). I've no precise data on this other than the experimental >>> observation that compiling all Elisp files in Emacs on 32bit systems is >>> not anymore an issue. This IIUC implies that the memory footprint for >>> each compilation is always < 2GB. >> >> You assume that the compilations are all done serially? AFAIK, most >> people build Emacs with "make -jN", so parallel compilation is an >> important use case. > >> I guess we will have to collect the information about that, if you say >> we don't have it now. > > I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to > produce quite accurate results on memory footprint during the > compilation process. Perhaps he has time and he's so kind to gather > some data on the current state, that would be extremely helpful. See also bug#41194#20 and bug#41194#28 where I outlined how the improvements reduced compilation time and memory usage. I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's in need of… something (probably exorcism, given the noise). Until I figure that out, here are the (very hacky) scripts I used to measure and plot the RAM usage, in case someone else wants to take some measurements: - ./monitor.sh $PID finds the most RAM-consuming process among $PID and its children, and logs its memory usage (VSZ and RSS) and its command-line. (Logs are collected every 10 seconds; this probably needs to be reduced for faster machines) - ./plot.py uses matplotlib to make graphs out of these measurements; it attempts to replace the command line with the less-verbose diagnostics from "make". - My workflow was to start an emacs session, run M-x compile RET make, then ./monitor.sh $PID_OF_EMACS_SESSION. (PARENT_RE in plot.py should match the command-line of this parent session; its RAM consumption is then labeled as "noise floor" on the graph. This serves no real purpose and should be removed; monitor.sh should be amended to filter the parent session out of monitored PIDs, with some error control to handle the lack of child processes when compilation is finished.) - There are some hardcoded things to tweak at the bottom of plot.py, e.g. how long should a child process last for it to have a label on the graph. [-- Attachment #2: monitor.sh --] [-- Type: application/x-shellscript, Size: 350 bytes --] [-- Attachment #3: plot.py --] [-- Type: text/x-python, Size: 5200 bytes --] #!/usr/bin/env python3 from datetime import datetime, timedelta from pathlib import Path import re import matplotlib from matplotlib import pyplot from matplotlib.dates import DateFormatter, HourLocator, MinuteLocator from matplotlib.ticker import EngFormatter MONITOR_RE = re.compile('\n'.join(( '(?P<time>.+)', r' *(?P<seconds>\d+) +(?P<vsz>\d+) +(?P<rss>\d+) +(?P<args>.+)', ' *(?P<memheader>.+)', 'Mem: *(?P<memvalues>.+)', 'Swap: *(?P<swapvalues>.+)', '' )), flags=re.MULTILINE) def list_snapshots(monitor_log): snapshots = [] for match in MONITOR_RE.finditer(monitor_log): md = match.groupdict() memkeys = md['memheader'].split() memvalues = md['memvalues'].split() swapvalues = md['swapvalues'].split() snapshot = { 'time': datetime.strptime(md['time'], '%Y-%m-%d-%H:%M:%S'), 'uptime': int(md['seconds']), 'vsz': int(md['vsz'])*1024, 'rss': int(md['rss'])*1024, 'process': md['args'], 'mem': {memkeys[i]: int(val)*1024 for i, val in enumerate(memvalues)}, 'swap': {memkeys[i]: int(val)*1024 for i, val in enumerate(swapvalues)} } snapshots.append(snapshot) return snapshots LOADDEFS_RE = re.compile( r'--eval \(setq generated-autoload-file' r' \(expand-file-name \(unmsys--file-name "([^"]+)"\)\)\)' r' -f batch-update-autoloads' ) SEMANTIC_RE = re.compile( r'-l semantic/(?:wisent|bovine)/grammar -f (?:wisent|bovine)-batch-make-parser' r' -o (.+) .+\.[wb]y' ) ELCELN_RE = re.compile( r'\.\./src/(?:bootstrap-)?emacs -batch --no-site-file --no-site-lisp' r' --eval \(setq load-prefer-newer t\) -l comp' r'(?: -f byte-compile-refresh-preloaded)?' r' -f batch-byte-native-compile-for-bootstrap' r' (.+\.el)' ) SHORTENED_NAMES = { LOADDEFS_RE: 'GEN', SEMANTIC_RE: 'GEN', ELCELN_RE: 'ELC+ELN' } QUAIL_TIT_RE = re.compile( r'-l titdic-cnv -f batch-titdic-convert' r' -dir \./\.\./lisp/leim/quail CXTERM-DIC/(.+)\.tit' ) QUAIL_MISC_RE = re.compile( r'-l titdic-cnv -f batch-miscdic-convert' r' -dir \./\.\./lisp/leim/quail MISC-DIC/(.+\.(html|map|cin|cns|b5))' ) QUAIL_JA_RE = re.compile( r'-l ja-dic-cnv -f batch-skkdic-convert' ) PARENT_RE = re.compile( r'$^' # Adjust to match parent process. ) TRANSFORMED_NAMES = { QUAIL_TIT_RE: lambda m: f'GEN ../lisp/leim/quail/{m.group(1)}.el', QUAIL_MISC_RE: lambda m: f'GEN from {m.group(1)}', QUAIL_JA_RE: lambda m: f'GEN ../lisp/leim/ja-dic/ja-dic.el', PARENT_RE: lambda _: '(noise floor)' } def shorten(process): for r, name in SHORTENED_NAMES.items(): match = r.search(process) if match is not None: return f'{name} {match.group(1)}' for r, transform in TRANSFORMED_NAMES.items(): match = r.search(process) if match is not None: return transform(match) if len(process) > 40: return f'{process[:20]}…{process[-20:]}' return process def list_processes(snapshots): t0 = snapshots[0]['time'] current_process = snapshots[0]['process'] current_process_start = t0 processes = [] for s in snapshots[1:]: if s['process'] == current_process: continue s_start = s['time'] processes.append(( current_process, current_process_start, s_start-current_process_start )) current_process = s['process'] current_process_start = s_start processes.append(( current_process, current_process_start, snapshots[-1]['time']-current_process_start )) return processes snapshots = list_snapshots(Path('monitor.log').read_text()) xs = tuple(s['time'] for s in snapshots) vsz = tuple(s['vsz'] for s in snapshots) rss = tuple(s['rss'] for s in snapshots) memavail = tuple(s['mem']['available'] for s in snapshots) swapused = tuple(s['swap']['used'] for s in snapshots) matplotlib.use('TkAgg') fig, axes = pyplot.subplots(figsize=(128, 9.6)) axes.plot(xs, vsz, label='VSZ (process)') axes.plot(xs, rss, label='RSS (process)') axes.plot(xs, memavail, label='available memory (system)', linewidth=0.5) axes.plot(xs, swapused, label='used swap (system)') axes.set_xlim(snapshots[0]['time'], snapshots[-1]['time']) axes.xaxis.set_major_formatter(DateFormatter('%H:%M')) axes.xaxis.set_major_locator(HourLocator()) axes.xaxis.set_minor_locator(MinuteLocator(tuple(5*i for i in range(1, 12)))) axes.xaxis.set_label_text('Hours') axes.set_ylim(0) axes.yaxis.set_major_formatter(EngFormatter(unit='B')) axes.legend() for p, start, duration in list_processes(snapshots): if duration < timedelta(minutes=2): continue pyplot.text(start, 1e9, shorten(p), rotation=45) pyplot.plot( (start, start+duration), (1e9, 1e9), marker='|', linewidth=0.5, linestyle='--', color='black', alpha=0.8 ) pyplot.savefig('monitor.pdf') pyplot.show() ^ permalink raw reply [flat|nested] 13+ messages in thread
* bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 2021-01-09 17:26 ` Kévin Le Gouguec @ 2021-01-09 19:41 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 0 replies; 13+ messages in thread From: Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2021-01-09 19:41 UTC (permalink / raw) To: Kévin Le Gouguec; +Cc: edouard.debry, 45705 Kévin Le Gouguec <kevin.legouguec@gmail.com> writes: > Andrea Corallo <akrl@sdf.org> writes: > >> Eli Zaretskii <eliz@gnu.org> writes: >> >>>> From: Andrea Corallo <akrl@sdf.org> >>>> >>>> In June we changed the way we store immediate objects in the shared and >>>> this makes the compilation way lighter on the GCC side (both in time and >>>> memory). I've no precise data on this other than the experimental >>>> observation that compiling all Elisp files in Emacs on 32bit systems is >>>> not anymore an issue. This IIUC implies that the memory footprint for >>>> each compilation is always < 2GB. >>> >>> You assume that the compilations are all done serially? AFAIK, most >>> people build Emacs with "make -jN", so parallel compilation is an >>> important use case. >> >>> I guess we will have to collect the information about that, if you say >>> we don't have it now. >> >> I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to >> produce quite accurate results on memory footprint during the >> compilation process. Perhaps he has time and he's so kind to gather >> some data on the current state, that would be extremely helpful. > > See also bug#41194#20 and bug#41194#28 where I outlined how the > improvements reduced compilation time and memory usage. > > I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's > in need of… something (probably exorcism, given the noise). At the moment I think we are interested more in the magnitude order and also a measure on 64bit would be very interesting. But you are right I see in bug#41194#20 bug#41194#28 you've already measured the footprint with the new immediate handling! IIRC the worstcase was org.el under 600MB. I think till we get a more recent measure we should assume this as number (I don't expect huge changes tho). Many thanks Andrea ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-01-09 19:41 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-01-06 20:48 bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 Édouard Debry 2021-01-06 20:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-07 14:25 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-08 14:25 ` Eli Zaretskii 2021-01-08 15:50 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-08 16:10 ` Eli Zaretskii 2021-01-08 22:02 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 7:56 ` Eli Zaretskii 2021-01-09 10:55 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 11:55 ` Eli Zaretskii 2021-01-09 12:37 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors 2021-01-09 17:26 ` Kévin Le Gouguec 2021-01-09 19:41 ` Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.