From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Mario Valencia Newsgroups: gmane.emacs.bugs Subject: bug#20220: severe memory leak on emacs 24.4.1 Date: Mon, 30 Mar 2015 20:07:29 -0600 Message-ID: References: <83mw2vzui9.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=089e0158ab9099fcb305128c0fd6 X-Trace: ger.gmane.org 1427767783 1395 80.91.229.3 (31 Mar 2015 02:09:43 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 31 Mar 2015 02:09:43 +0000 (UTC) Cc: 20220@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Mar 31 04:09:29 2015 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1YclbC-0003Fl-0d for geb-bug-gnu-emacs@m.gmane.org; Tue, 31 Mar 2015 04:08:14 +0200 Original-Received: from localhost ([::1]:36556 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YclbA-0002ic-I2 for geb-bug-gnu-emacs@m.gmane.org; Mon, 30 Mar 2015 22:08:12 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:39393) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yclb5-0002iU-Ta for bug-gnu-emacs@gnu.org; Mon, 30 Mar 2015 22:08:09 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Yclb0-0007Xi-NE for bug-gnu-emacs@gnu.org; Mon, 30 Mar 2015 22:08:07 -0400 Original-Received: from debbugs.gnu.org ([140.186.70.43]:51004) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yclb0-0007Xe-Jn for bug-gnu-emacs@gnu.org; Mon, 30 Mar 2015 22:08:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.80) (envelope-from ) id 1Yclb0-0002vn-3K for bug-gnu-emacs@gnu.org; Mon, 30 Mar 2015 22:08:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Mario Valencia Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 31 Mar 2015 02:08:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 20220 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 20220-submit@debbugs.gnu.org id=B20220.142776765911239 (code B ref 20220); Tue, 31 Mar 2015 02:08:01 +0000 Original-Received: (at 20220) by debbugs.gnu.org; 31 Mar 2015 02:07:39 +0000 Original-Received: from localhost ([127.0.0.1]:40780 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Yclac-0002vC-LA for submit@debbugs.gnu.org; Mon, 30 Mar 2015 22:07:39 -0400 Original-Received: from mail-la0-f50.google.com ([209.85.215.50]:33926) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YclaZ-0002uy-Uf for 20220@debbugs.gnu.org; Mon, 30 Mar 2015 22:07:37 -0400 Original-Received: by lagg8 with SMTP id g8so2246078lag.1 for <20220@debbugs.gnu.org>; Mon, 30 Mar 2015 19:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Xxdk4riFkqfu4tPM5u9Cnjm76rgvr3h3mzSwL3XgliY=; b=sC6dwRxcFjkVUOzfiHE671xVH+aKRLox7wAqvUWc94tH49MapCSYla9ziT1u62K4W6 ZXD9OGcK6011a/me3v5nKWj1IUXviWqR/7vjZCSYDP0apm9Mcd5MLEH91FiAxzs8aSCk dvZpCpbzwhOfvrHPO6BseSltLFHTDr/HCkbAXv7ucsU1ACksh33adZ3dVEu0ul/kXvLn SFP1MOZH49PTJQcqMPLLyC66/8YmUPWIwEDCXqeGZzKKT5umkVLbHUKj2nM3lQxOWcFq FqYTSJxuSNMHUhBEkH9Lkd4Z5HIURkHLw7Sr8wnNlkOqyX74jN0a04QuT3JZBSmog1Av 5VKA== X-Received: by 10.152.22.104 with SMTP id c8mr28549149laf.87.1427767650022; Mon, 30 Mar 2015 19:07:30 -0700 (PDT) Original-Received: by 10.112.170.130 with HTTP; Mon, 30 Mar 2015 19:07:29 -0700 (PDT) In-Reply-To: <83mw2vzui9.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 140.186.70.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:101065 Archived-At: --089e0158ab9099fcb305128c0fd6 Content-Type: text/plain; charset=UTF-8 Then the shell execute function is worthless. I had used it for opening the browser and also for opening files with an external program, or to open them in the explorer, i guess i will have to remove all use of that function from my scripts. 2015-03-29 9:35 GMT-06:00 Eli Zaretskii : > > Date: Sat, 28 Mar 2015 17:39:38 -0600 > > From: Mario Valencia > > > > to reproduce it, i create a small emtpy html document. then i evaluate > the > > following expression: > > > > (dotimes (i 100) (browse-url-of-file)) > > > > This causes emacs to start opening the file using google chrome. In the > task > > manager, i can see emacs' memory usage go up by about 5 megabytes each > time a > > tab is opened. When it opens about 30 tabs, emacs is using 138 megabytes > of > > memory, and it gives the error below. > > the translation is something like this: "ShellExecute failed: Storage > space > > insufficient to process this command" > > My harddrive has enough storage space btw. > > (The error is not about disk storage, it's about reserving the address > space in virtual memory.) > > FWIW, I don't see such a large memory increase when I reproduce this, > I see something around 1MB, sometimes 1.5MB. But I didn't try on > Windows 8. > > Anyway, Emacs doesn't allocate any memory when it calls the > ShellExecute function, so I see no way we could leak something here. > > My guess would be that invoking ShellExecute causes Windows to start a > thread in the context of the Emacs process, and reserve 8MB of stack > space for that thread. (On one of 3 systems I tried your recipe, I > actually saw a thread per invocation, each thread was running some > function inside shlwapi.dll, the shared library which implements the > ShellExecute API.) The memory actually used by that thread for its > stack is much smaller than 8MB, of course, but Windows attempts to > reserve 8MB of address space for its stack. > > The 8MB figure comes from the way we link Emacs: we need such a large > stack due to regular expressions, stack-based Lisp objects, and GC > which is deeply recursive. By default, each thread reserves the same > stack space as the program to which it belongs. For threads we launch > in Emacs, we override the default 8MB stack space by a much smaller > value, but we have no such control on threads that Windows itself > starts on behalf of the Emacs process. > > The error message and the failure to launch too many browser tabs are > caused by the fact that Emacs itself reserves about 1.7GB of the > address space for its memory management, leaving only a small portion > of the 2GB address space that 32-bit programs can use on Windows. > Start enough threads with 8MB stack reservation, and you will run out > of address space. > > Emacs 25 uses a different memory allocation scheme for buffers and > strings, which allows to avoid the large memory reservation, so at > least the out-of-memory error should happen there only after many more > ShellExecute invocations (because threads started by Windows on behalf > of ShellExecute will still reserve 8MB of memory for their stack). > > If someone knows how to force threads started by Windows to reserve > less memory, without also lowering the stack space available to Emacs > itself (i.e. its main thread), please tell. Otherwise, I guess we > will have to live with this limitation on Windows. > > --089e0158ab9099fcb305128c0fd6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Then the shell execute function is worthless. I had used i= t for opening the browser and also for opening files with an external progr= am, or to open them in the explorer, i guess i will have to remove all use = of that function from my scripts.

2015-03-29 9:35 GMT-06:00 Eli Zaretskii <eliz@gnu.org= >:
> Date: Sat, 28 Mar 2015 = 17:39:38 -0600
> From: Mario Valencia <mari= ovalspi@gmail.com>
>
> to reproduce it, i create a small emtpy html document= . then i evaluate the
> following expression:
>
> (dotimes (i 100) (browse-url-of-file))
>
> This causes emacs to start opening the file using google chrome. In th= e task
> manager, i can see emacs' memory usage go up by about 5 megabytes = each time a
> tab is opened. When it opens about 30 tabs, emacs is using 138 megabyt= es of
> memory, and it gives the error below.
> the translation is something like this: "ShellExecute failed: Sto= rage space
> insufficient to process this command"
> My harddrive has enough storage space btw.

(The error is not about disk storage, it's about reserving the a= ddress
space in virtual memory.)

FWIW, I don't see such a large memory increase when I reproduce this, I see something around 1MB, sometimes 1.5MB.=C2=A0 But I didn't try on<= br> Windows 8.

Anyway, Emacs doesn't allocate any memory when it calls the
ShellExecute function, so I see no way we could leak something here.

My guess would be that invoking ShellExecute causes Windows to start a
thread in the context of the Emacs process, and reserve 8MB of stack
space for that thread.=C2=A0 (On one of 3 systems I tried your recipe, I actually saw a thread per invocation, each thread was running some
function inside shlwapi.dll, the shared library which implements the
ShellExecute API.)=C2=A0 The memory actually used by that thread for its stack is much smaller than 8MB, of course, but Windows attempts to
reserve 8MB of address space for its stack.

The 8MB figure comes from the way we link Emacs: we need such a large
stack due to regular expressions, stack-based Lisp objects, and GC
which is deeply recursive.=C2=A0 By default, each thread reserves the same<= br> stack space as the program to which it belongs.=C2=A0 For threads we launch=
in Emacs, we override the default 8MB stack space by a much smaller
value, but we have no such control on threads that Windows itself
starts on behalf of the Emacs process.

The error message and the failure to launch too many browser tabs are
caused by the fact that Emacs itself reserves about 1.7GB of the
address space for its memory management, leaving only a small portion
of the 2GB address space that 32-bit programs can use on Windows.
Start enough threads with 8MB stack reservation, and you will run out
of address space.

Emacs 25 uses a different memory allocation scheme for buffers and
strings, which allows to avoid the large memory reservation, so at
least the out-of-memory error should happen there only after many more
ShellExecute invocations (because threads started by Windows on behalf
of ShellExecute will still reserve 8MB of memory for their stack).

If someone knows how to force threads started by Windows to reserve
less memory, without also lowering the stack space available to Emacs
itself (i.e. its main thread), please tell.=C2=A0 Otherwise, I guess we
will have to live with this limitation on Windows.


--089e0158ab9099fcb305128c0fd6--