bug#20220: severe memory leak on emacs 24.4.1

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 5015 bytes --]

to reproduce it, i create a small emtpy html document. then i evaluate the
following expression:

(dotimes (i 100) (browse-url-of-file))

This causes emacs to start opening the file using google chrome. In the
task manager, i can see emacs' memory usage go up by about 5 megabytes each
time a tab is opened. When it opens about 30 tabs, emacs is using 138
megabytes of memory, and it gives the error below.
the translation is something like this: "ShellExecute failed: Storage space
insufficient to process this command"
My harddrive has enough storage space btw.

I was able to reproduce this bug 3 times. I ran emacs with -Q on windows 8.
Im using emacs version:
GNU Emacs 24.4.1 (i686-pc-mingw32)
 of 2014-10-24 on LEG570

Debugger entered--Lisp error: (error "ShellExecute failed: Espacio de
almacenamiento insuficiente para procesar este comando.")
  w32-shell-execute("open" "file:///c:/Users/mario/Desktop/x.html")

browse-url-default-windows-browser("file:///c:/Users/mario/Desktop/x.html"
nil)
  apply(browse-url-default-windows-browser
"file:///c:/Users/mario/Desktop/x.html" nil)
  browse-url-default-browser("file:///c:/Users/mario/Desktop/x.html" nil)
  apply(browse-url-default-browser "file:///c:/Users/mario/Desktop/x.html"
nil)
  browse-url("file:///c:/Users/mario/Desktop/x.html")
  browse-url-of-file("c:/Users/mario/Desktop/x.html")
  browse-url-of-buffer()
  (while (< i --dotimes-limit--) (browse-url-of-buffer) (setq i (1+ i)))
  (let ((--dotimes-limit-- 100) (i 0)) (while (< i --dotimes-limit--)
(browse-url-of-buffer) (setq i (1+ i))))
  (dotimes (i 100) (browse-url-of-buffer))
  eval((dotimes (i 100) (browse-url-of-buffer)) nil)
  eval-expression((dotimes (i 100) (browse-url-of-buffer)) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

In GNU Emacs 24.4.1 (i686-pc-mingw32)
 of 2014-10-24 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr'

Important settings:
  value of $LANG: ESM
  locale-coding-system: cp1252

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> C-x C-f <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> x . h m l <backspace> <backspace>
t m l <return> C-k C-k C-k C-k C-k C-k C-k C-k C-k
C-k C-k C-x C-s M-: <up> ( d o t i m e s - <backspace>
SPC ( i SPC 1 0 0 ) SPC ( b r o w s e - u r l <tab>
<backspace> <backspace> <backspace> b u f <tab> ) )
<return> C-a <S-down> <S-down> <S-down> <S-down> <S-down>
<S-down> <S-down> <S-down> <S-down> <S-down> <S-down>
<S-down> <S-down> <S-down> <S-down> <S-down> <S-down>
<S-down> <S-down> <S-down> <S-down> <S-down> M-w <help-echo>
e i <return> <help-echo> <help-echo> <help-echo> q
C-h C-a <down> <S-down> <S-down> C-c <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> M-w M-w <help-echo>
<help-echo> <help-echo> M-x r e p o r t - e m <tab>
<return>

Recent messages:
Wrote c:/Users/mario/Desktop/x.html
user-error: Beginning of history; no preceding item
Entering debugger...
Mark set
End of buffer [6 times]
Saved text from "Debugger entered--Lisp error: (error "Sh"
30 (#o36, #x1e, ?\C-^)
Back to top level.
Mark set
C-c M-w is undefined

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
help-fns mail-prsvr mail-utils mule-util help-mode easymenu debug
browse-url sgml-mode time-date tooltip electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns
disp-table w32-win w32-vars tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process w32notify w32 multi-tty emacs)

Memory information:
((conses 8 82170 4244)
 (symbols 32 17933 0)
 (miscs 32 88 151)
 (strings 16 12976 3656)
 (string-bytes 1 327227)
 (vectors 8 10094)
 (vector-slots 4 391999 3862)
 (floats 8 61 285)
 (intervals 28 398 48)
 (buffers 508 15))

[-- Attachment #2: Type: text/html, Size: 6959 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On 03/28/2015 04:39 PM, Mario Valencia wrote:
> to reproduce it, i create a small emtpy html document. then i evaluate
> the following expression:
> 
> (dotimes (i 100) (browse-url-of-file))
> 
> This causes emacs to start opening the file using google chrome. In the
> task manager, i can see emacs' memory usage go up by about 5 megabytes
> each time a tab is opened. When it opens about 30 tabs, emacs is using
> 138 megabytes of memory, and it gives the error below.
> the translation is something like this: "ShellExecute failed: Storage
> space insufficient to process this command"
> My harddrive has enough storage space btw.

Do you happen to have any shell extensions installed?  ShellExecute can
run arbitrary third-party code through this extension mechanism.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

I have no idea my friend. How can I check?

2015-03-29 0:01 GMT-06:00 Daniel Colascione <dancol@dancol.org>:

> On 03/28/2015 04:39 PM, Mario Valencia wrote:
> > to reproduce it, i create a small emtpy html document. then i evaluate
> > the following expression:
> >
> > (dotimes (i 100) (browse-url-of-file))
> >
> > This causes emacs to start opening the file using google chrome. In the
> > task manager, i can see emacs' memory usage go up by about 5 megabytes
> > each time a tab is opened. When it opens about 30 tabs, emacs is using
> > 138 megabytes of memory, and it gives the error below.
> > the translation is something like this: "ShellExecute failed: Storage
> > space insufficient to process this command"
> > My harddrive has enough storage space btw.
>
> Do you happen to have any shell extensions installed?  ShellExecute can
> run arbitrary third-party code through this extension mechanism.
>
>

[-- Attachment #2: Type: text/html, Size: 1309 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Sun, 29 Mar 2015 00:04:51 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: 20220@debbugs.gnu.org
> 
> I have no idea my friend. How can I check?

One way is to install 'autoruns' from Sysinternals, it shows the
extensions on the Explorer tab.

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Sat, 28 Mar 2015 17:39:38 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
> 
> to reproduce it, i create a small emtpy html document. then i evaluate the
> following expression:
> 
> (dotimes (i 100) (browse-url-of-file))
> 
> This causes emacs to start opening the file using google chrome. In the task
> manager, i can see emacs' memory usage go up by about 5 megabytes each time a
> tab is opened. When it opens about 30 tabs, emacs is using 138 megabytes of
> memory, and it gives the error below.
> the translation is something like this: "ShellExecute failed: Storage space
> insufficient to process this command"
> My harddrive has enough storage space btw.

(The error is not about disk storage, it's about reserving the address
space in virtual memory.)

FWIW, I don't see such a large memory increase when I reproduce this,
I see something around 1MB, sometimes 1.5MB.  But I didn't try on
Windows 8.

Anyway, Emacs doesn't allocate any memory when it calls the
ShellExecute function, so I see no way we could leak something here.

My guess would be that invoking ShellExecute causes Windows to start a
thread in the context of the Emacs process, and reserve 8MB of stack
space for that thread.  (On one of 3 systems I tried your recipe, I
actually saw a thread per invocation, each thread was running some
function inside shlwapi.dll, the shared library which implements the
ShellExecute API.)  The memory actually used by that thread for its
stack is much smaller than 8MB, of course, but Windows attempts to
reserve 8MB of address space for its stack.

The 8MB figure comes from the way we link Emacs: we need such a large
stack due to regular expressions, stack-based Lisp objects, and GC
which is deeply recursive.  By default, each thread reserves the same
stack space as the program to which it belongs.  For threads we launch
in Emacs, we override the default 8MB stack space by a much smaller
value, but we have no such control on threads that Windows itself
starts on behalf of the Emacs process.

The error message and the failure to launch too many browser tabs are
caused by the fact that Emacs itself reserves about 1.7GB of the
address space for its memory management, leaving only a small portion
of the 2GB address space that 32-bit programs can use on Windows.
Start enough threads with 8MB stack reservation, and you will run out
of address space.

Emacs 25 uses a different memory allocation scheme for buffers and
strings, which allows to avoid the large memory reservation, so at
least the out-of-memory error should happen there only after many more
ShellExecute invocations (because threads started by Windows on behalf
of ShellExecute will still reserve 8MB of memory for their stack).

If someone knows how to force threads started by Windows to reserve
less memory, without also lowering the stack space available to Emacs
itself (i.e. its main thread), please tell.  Otherwise, I guess we
will have to live with this limitation on Windows.

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 3414 bytes --]

Then the shell execute function is worthless. I had used it for opening the
browser and also for opening files with an external program, or to open
them in the explorer, i guess i will have to remove all use of that
function from my scripts.

2015-03-29 9:35 GMT-06:00 Eli Zaretskii <eliz@gnu.org>:

> > Date: Sat, 28 Mar 2015 17:39:38 -0600
> > From: Mario Valencia <mariovalspi@gmail.com>
> >
> > to reproduce it, i create a small emtpy html document. then i evaluate
> the
> > following expression:
> >
> > (dotimes (i 100) (browse-url-of-file))
> >
> > This causes emacs to start opening the file using google chrome. In the
> task
> > manager, i can see emacs' memory usage go up by about 5 megabytes each
> time a
> > tab is opened. When it opens about 30 tabs, emacs is using 138 megabytes
> of
> > memory, and it gives the error below.
> > the translation is something like this: "ShellExecute failed: Storage
> space
> > insufficient to process this command"
> > My harddrive has enough storage space btw.
>
> (The error is not about disk storage, it's about reserving the address
> space in virtual memory.)
>
> FWIW, I don't see such a large memory increase when I reproduce this,
> I see something around 1MB, sometimes 1.5MB.  But I didn't try on
> Windows 8.
>
> Anyway, Emacs doesn't allocate any memory when it calls the
> ShellExecute function, so I see no way we could leak something here.
>
> My guess would be that invoking ShellExecute causes Windows to start a
> thread in the context of the Emacs process, and reserve 8MB of stack
> space for that thread.  (On one of 3 systems I tried your recipe, I
> actually saw a thread per invocation, each thread was running some
> function inside shlwapi.dll, the shared library which implements the
> ShellExecute API.)  The memory actually used by that thread for its
> stack is much smaller than 8MB, of course, but Windows attempts to
> reserve 8MB of address space for its stack.
>
> The 8MB figure comes from the way we link Emacs: we need such a large
> stack due to regular expressions, stack-based Lisp objects, and GC
> which is deeply recursive.  By default, each thread reserves the same
> stack space as the program to which it belongs.  For threads we launch
> in Emacs, we override the default 8MB stack space by a much smaller
> value, but we have no such control on threads that Windows itself
> starts on behalf of the Emacs process.
>
> The error message and the failure to launch too many browser tabs are
> caused by the fact that Emacs itself reserves about 1.7GB of the
> address space for its memory management, leaving only a small portion
> of the 2GB address space that 32-bit programs can use on Windows.
> Start enough threads with 8MB stack reservation, and you will run out
> of address space.
>
> Emacs 25 uses a different memory allocation scheme for buffers and
> strings, which allows to avoid the large memory reservation, so at
> least the out-of-memory error should happen there only after many more
> ShellExecute invocations (because threads started by Windows on behalf
> of ShellExecute will still reserve 8MB of memory for their stack).
>
> If someone knows how to force threads started by Windows to reserve
> less memory, without also lowering the stack space available to Emacs
> itself (i.e. its main thread), please tell.  Otherwise, I guess we
> will have to live with this limitation on Windows.
>
>

[-- Attachment #2: Type: text/html, Size: 4033 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Mon, 30 Mar 2015 20:07:29 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: 20220@debbugs.gnu.org
> 
> Then the shell execute function is worthless. I had used it for opening the
> browser and also for opening files with an external program, or to open them in
> the explorer, i guess i will have to remove all use of that function from my
> scripts.

Well, it's your decision, of course.  However, I think that
"worthless" is too extreme, and not using it at all is too radical,
even if you can do nothing in terms of your system configuration to
bring down the amount of resources consumed by each invocation.  One
thread and 8 MB of memory is not that large, unless you really invoke
that command tens or times in a row.

Moreover, after reproducing your recipe on several machine, I
concluded that the actual consumption of resources does somehow depend
on system configuration.  Out of 4 systems I tried that on, only 1
showed a 8MB increase in the memory footprint and an additional thread
per invocation of w32-shell-execute; all the other systems didn't
start any additional threads and their memory footprint growth was
much smaller.  On Windows 8.1, I actually saw no memory growth at all.

So perhaps reviewing your shell extensions, like Daniel suggested, and
removing those you don't need, will fix this.

It is also possible that if you leave Emacs running for a long enough
time after closing the URL, the thread and its memory will be
released; I didn't try to investigate that.

bug#20220: severe memory leak on emacs 24.4.1

[Stefan Monnier]

> It is also possible that if you leave Emacs running for a long enough
> time after closing the URL, the thread and its memory will be
> released; I didn't try to investigate that.

Right, the question is: if there's only 1 active "ShellExecute" process
at the same time (but repeated in a loop), does the memory still keep growing?
If it does, we definitely have a problem.

[  I suspect this has already been answered, but I haven't read this thread
   with enough care to find out.  ]

        Stefan

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

> However, I think that
>"worthless" is too extreme, and not using it at all is too radical,
>even if you can do nothing in terms of your system configuration to
>bring down the amount of resources consumed by each invocation.  One
>thread and 8 MB of memory is not that large, unless you really invoke
>that command tens or times in a row.

You are wrong. Calling shellexecute tens of times is very common. Just
consider when editing an html file, saving and viewing the file may happen
more than once a minute, so in less than 30 minutes it will hit the bug. Or
when opening files with an external application in dired, if you open more
than 30 files with shellexecute, it will hit the bug, which is not
unlikely. Afterwards, after quitting the debugger, a few more times of
calling the function and emacs will hang permanently.

[-- Attachment #2: Type: text/html, Size: 1581 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Tue, 31 Mar 2015 20:19:27 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 20220@debbugs.gnu.org
> 
> > However, I think that
> >"worthless" is too extreme, and not using it at all is too radical,
> >even if you can do nothing in terms of your system configuration to
> >bring down the amount of resources consumed by each invocation. One
> >thread and 8 MB of memory is not that large, unless you really invoke
> >that command tens or times in a row.
> 
> You are wrong. Calling shellexecute tens of times is very common. Just consider
> when editing an html file, saving and viewing the file may happen more than
> once a minute, so in less than 30 minutes it will hit the bug.

You can use the browser's "refresh" function instead, don't?  That's
what I'd do, regardless of this problem.

> Or when opening files with an external application in dired, if you
> open more than 30 files with shellexecute, it will hit the bug,
> which is not unlikely. Afterwards, after quitting the debugger, a
> few more times of calling the function and emacs will hang
> permanently.

Did you actually see this problem in real life, not with your recipe?

Anyway, I still think you should look at your system's configuration,
since I see nothing similar to this large memory consumption on all
but 1 system where I tried this.  So there are factors here at work
that depend on the system.

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

I've looked a bit more into this problem.

A noteworthy aspect of this is that invoking w32-shell-execute like
this:

  (dotimes (i 100)
    (w32-shell-execute "open" "x:/path/to/empty.html"))

doesn't start any new threads in the context of the Emacs process,
doesn't increase its memory footprint, and therefore doesn't fail.
However, if I do this instead:

  (dotimes (i 100)
    (w32-shell-execute "open" "file:///x:/path/to/empty.html"))

then I do see the same phenomena as with browse-url-of-file.  So the
creation of extra threads is somehow triggered by using the file://
URL (which is what browse-url-of-file does).

Do you see the same on your system?

Once again, I only see all this on a single system out of 4 I tried;
the other 3 don't show extra threads, don't enlarge the Emacs memory
footprint in any significant way, and don't fail with your original
recipe.  So it appears the problem is caused by some relatively rare
conditions.

bug#20220: severe memory leak on emacs 24.4.1

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On 03/31/2015 07:19 PM, Mario Valencia wrote:
>> However, I think that
>>"worthless" is too extreme, and not using it at all is too radical,
>>even if you can do nothing in terms of your system configuration to
>>bring down the amount of resources consumed by each invocation.  One
>>thread and 8 MB of memory is not that large, unless you really invoke
>>that command tens or times in a row.
> 
> You are wrong. Calling shellexecute tens of times is very common. Just
> consider when editing an html file, saving and viewing the file may
> happen more than once a minute, so in less than 30 minutes it will hit
> the bug. Or when opening files with an external application in dired, if
> you open more than 30 files with shellexecute, it will hit the bug,
> which is not unlikely. Afterwards, after quitting the debugger, a few
> more times of calling the function and emacs will hang permanently.

Whatever bit of code is starting a thread (or otherwise permanently
consuming resources) on ShellExecute is broken, not Emacs. I don't think
I've seen that behavior myself. What's the thread start function on that
new thread? What's on its stack?

Anyway, you should be able to work around the problem by using cmd.exe's
"start" builtin, which will run ShellExecute on your behalf.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Wed, 01 Apr 2015 00:53:56 -0700
> From: Daniel Colascione <dancol@dancol.org>
> Cc: 20220@debbugs.gnu.org
> 
> Whatever bit of code is starting a thread (or otherwise permanently
> consuming resources) on ShellExecute is broken, not Emacs. I don't think
> I've seen that behavior myself.

As I wrote, I only see something similar on one particular system out
of 4 I tried this on.  And I'm not yet sure what I see there is what
happens on Mario's system.

> What's the thread start function on that new thread?

SHWLAPI.dll!IUnknown_QueryService+0x87 on my system where I see this.

> What's on its stack?

This:

  wow64win.dll+0x3fe3a
  wow64win.dll+0x1aeac
  wow64.dll!Wow64SystemServiceEx+0xd7
  wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
  wow64.dll!Wow64SystemServiceEx+0x1ce
  wow64.dll!Wow64LdrpInitialize+0x42a
  ntdll.dll!RtlIsDosDeviceName_U+0x23a27
  ntdll.dll!LdrInitializeThunk+0xe
  USER32.dll!DispatchMessageW+0x5c
  SHLWAPI.dll!Ordinal173+0x287b
  ntdll.dll!RtlInitializeExceptionChain+0x63
  ntdll.dll!RtlInitializeExceptionChain+0x36

Mario, can you tell if you see similar things on your system?  I
suggest you verify that you have the same issue, by installing the
Process Explorer and looking in the Properties for the Emacs process,
in the Threads tab.  There you should see a new thread created each
time w32-shell-execute is invoked, and also the symbolic Start Address
of each such thread.

bug#20220: severe memory leak on emacs 24.4.1

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

On 04/01/2015 07:26 AM, Eli Zaretskii wrote:
>> Date: Wed, 01 Apr 2015 00:53:56 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: 20220@debbugs.gnu.org
>>
>> Whatever bit of code is starting a thread (or otherwise permanently
>> consuming resources) on ShellExecute is broken, not Emacs. I don't think
>> I've seen that behavior myself.
> 
> As I wrote, I only see something similar on one particular system out
> of 4 I tried this on.  And I'm not yet sure what I see there is what
> happens on Mario's system.
> 
>> What's the thread start function on that new thread?
> 
> SHWLAPI.dll!IUnknown_QueryService+0x87 on my system where I see this.
> 
>> What's on its stack?
> 
> This:
> 
>   wow64win.dll+0x3fe3a
>   wow64win.dll+0x1aeac
>   wow64.dll!Wow64SystemServiceEx+0xd7
>   wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
>   wow64.dll!Wow64SystemServiceEx+0x1ce
>   wow64.dll!Wow64LdrpInitialize+0x42a
>   ntdll.dll!RtlIsDosDeviceName_U+0x23a27
>   ntdll.dll!LdrInitializeThunk+0xe
>   USER32.dll!DispatchMessageW+0x5c
>   SHLWAPI.dll!Ordinal173+0x287b
>   ntdll.dll!RtlInitializeExceptionChain+0x63
>   ntdll.dll!RtlInitializeExceptionChain+0x36
> 
> Mario, can you tell if you see similar things on your system?  I
> suggest you verify that you have the same issue, by installing the
> Process Explorer and looking in the Properties for the Emacs process,
> in the Threads tab.  There you should see a new thread created each
> time w32-shell-execute is invoked, and also the symbolic Start Address
> of each such thread.

On what OS version is that thread running?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Wed, 01 Apr 2015 09:01:13 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: mariovalspi@gmail.com, monnier@iro.umontreal.ca, 
>  20220@debbugs.gnu.org
> 
> >   wow64win.dll+0x3fe3a
> >   wow64win.dll+0x1aeac
> >   wow64.dll!Wow64SystemServiceEx+0xd7
> >   wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
> >   wow64.dll!Wow64SystemServiceEx+0x1ce
> >   wow64.dll!Wow64LdrpInitialize+0x42a
> >   ntdll.dll!RtlIsDosDeviceName_U+0x23a27
> >   ntdll.dll!LdrInitializeThunk+0xe
> >   USER32.dll!DispatchMessageW+0x5c
> >   SHLWAPI.dll!Ordinal173+0x287b
> >   ntdll.dll!RtlInitializeExceptionChain+0x63
> >   ntdll.dll!RtlInitializeExceptionChain+0x36
> > 
> > Mario, can you tell if you see similar things on your system?  I
> > suggest you verify that you have the same issue, by installing the
> > Process Explorer and looking in the Properties for the Emacs process,
> > in the Threads tab.  There you should see a new thread created each
> > time w32-shell-execute is invoked, and also the symbolic Start Address
> > of each such thread.
> 
> On what OS version is that thread running?

64-bit Windows 7.

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

I think the thread created is this:
SHCORE.DLL!Ordinal254+0x9a0 <- start address

stack:
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x598
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x3e4
wow64.dll!Wow64LdrpInitialize+0x23a
wow64.dll!Wow64LdrpInitialize+0x172
ntdll.dll!LdrInitializeThunk+0x12b
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0xc
KERNELBASE.dll!WaitForSingleObject+0x12
ashShell.dll!Release+0xa2c2
ashShell.dll!Release+0x936b
ashShell.dll!Release+0x5ada2
ashShell.dll!Release+0x448e5
ashShell.dll!Release+0x394b3
ashShell.dll!Release+0x393f5
ntdll.dll!RtlInitializeCriticalSection+0x10e
ntdll.dll!RtlInitializeCriticalSection+0x88
ntdll.dll!EtwEventWrite+0x67e
ntdll.dll!EtwEventWrite+0x4fb
ntdll.dll!RtlInterlockedPushEntrySList+0x192
ntdll.dll!LdrUnloadDll+0x40
KERNELBASE.dll!FreeLibrary+0x19
combase.dll!InternalFillLocalOXIDInfo+0xf1d
combase.dll!InternalFillLocalOXIDInfo+0x1046
combase.dll!InternalFillLocalOXIDInfo+0xffc
combase.dll!CoFreeUnusedLibrariesEx+0x941
combase.dll!CoFreeUnusedLibrariesEx+0x82f
combase.dll!CoFreeUnusedLibrariesEx+0x1280
combase.dll!CoUninitialize+0xb2
SHCORE.DLL!Ordinal140+0x150
ntdll.dll!RtlInitializeExceptionChain+0x8f
ntdll.dll!RtlInitializeExceptionChain+0x5a

also this:
ashShell.dll!Release+0x9d40 <-- start address
stack:
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x598
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x3e4
wow64.dll!Wow64LdrpInitialize+0x23a
wow64.dll!Wow64LdrpInitialize+0x172
ntdll.dll!LdrInitializeThunk+0x12b
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0xc
ntdll.dll!RtlInterlockedPushEntrySList+0x32b
ntdll.dll!RtlInterlockedPushEntrySList+0x355
ntdll.dll!RtlDestroyMemoryBlockLookaside+0x2aca
ntdll.dll!RtlExitUserThread+0x4c
ntdll.dll!RtlInitializeExceptionChain+0x8f
ntdll.dll!RtlInitializeExceptionChain+0x5a

[-- Attachment #2: Type: text/html, Size: 2766 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Sat, 4 Apr 2015 14:24:07 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 20220@debbugs.gnu.org
> 
> I think the thread created is this:
> SHCORE.DLL!Ordinal254+0x9a0 <- start address

SHCORE.dll sounds like a Windows 8 version of shlwapi.dll, or maybe a
part of it.  So this looks very similar to what I saw on that single
Windows 7 system.

Do you also see that the problem happens only if w32-shell-execute is
called with a file:/// URL, and does not happen if it is called with a
normal Windows file name?

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

You are correct; the problem only happens with a file:/// url. The
following executes without problem and emacs memory stays constant:
(dotimes (i 100) (w32-shell-execute "open" "c:/users/mario/desktop/x.html"))
With a file:/// url it triggers the bug:
(dotimes (i 100) (w32-shell-execute "open"
"file:///c:/users/mario/desktop/x.html"))

2015-04-04 15:19 GMT-06:00 Eli Zaretskii <eliz@gnu.org>:

> > Date: Sat, 4 Apr 2015 14:24:07 -0600
> > From: Mario Valencia <mariovalspi@gmail.com>
> > Cc: Daniel Colascione <dancol@dancol.org>, Stefan Monnier <
> monnier@iro.umontreal.ca>, 20220@debbugs.gnu.org
> >
> > I think the thread created is this:
> > SHCORE.DLL!Ordinal254+0x9a0 <- start address
>
> SHCORE.dll sounds like a Windows 8 version of shlwapi.dll, or maybe a
> part of it.  So this looks very similar to what I saw on that single
> Windows 7 system.
>
> Do you also see that the problem happens only if w32-shell-execute is
> called with a file:/// URL, and does not happen if it is called with a
> normal Windows file name?
>

[-- Attachment #2: Type: text/html, Size: 1670 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Fri, 10 Apr 2015 06:16:46 -0500
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 20220@debbugs.gnu.org
> 
> You are correct; the problem only happens with a file:/// url. The following
> executes without problem and emacs memory stays constant:
> (dotimes (i 100) (w32-shell-execute "open" "c:/users/mario/desktop/x.html"))
> With a file:/// url it triggers the bug:
> (dotimes (i 100) (w32-shell-execute "open"
> "file:///c:/users/mario/desktop/x.html"))

Thanks, that's good to know.

So I think the solution is to change w32-shell-execute to convert
file:/// URLs into the normal file-name format internally, and use
that in the call to ShellExecute.

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Fri, 10 Apr 2015 14:24:18 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20220@debbugs.gnu.org
> 
> So I think the solution is to change w32-shell-execute to convert
> file:/// URLs into the normal file-name format internally, and use
> that in the call to ShellExecute.

Btw, until that change is made, you can make an equivalent change in
browse-url.el, e.g. by providing a trivial wrapper for
w32-shell-execute.  Then you will have the problem solved in the same
binary you already have.

bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

Thx, I will. It would also be nice to know what causes the bug, rather than
just fixing it by workaround tho.

2015-04-10 6:29 GMT-05:00 Eli Zaretskii <eliz@gnu.org>:

> > Date: Fri, 10 Apr 2015 14:24:18 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: 20220@debbugs.gnu.org
> >
> > So I think the solution is to change w32-shell-execute to convert
> > file:/// URLs into the normal file-name format internally, and use
> > that in the call to ShellExecute.
>
> Btw, until that change is made, you can make an equivalent change in
> browse-url.el, e.g. by providing a trivial wrapper for
> w32-shell-execute.  Then you will have the problem solved in the same
> binary you already have.
>

[-- Attachment #2: Type: text/html, Size: 1144 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Fri, 10 Apr 2015 06:33:44 -0500
> From: Mario Valencia <mariovalspi@gmail.com>
> Cc: 20220@debbugs.gnu.org
> 
> It would also be nice to know what causes the bug, rather than just
> fixing it by workaround tho.

I agree.  I tried googling for similar issues, but everything that
came up was either not relevant or described real memory leaks in
other people's code.  If you or someone else can find any
documentation on this strange phenomenon, please tell.

bug#20220: severe memory leak on emacs 24.4.1

[Stefan Monnier]

> I agree.  I tried googling for similar issues, but everything that
> came up was either not relevant or described real memory leaks in
> other people's code.

Maybe posting a question to a forum/mailinglist could get us
a better answer.  Not sure what forum/mailinglist would have the right
people, tho.


        Stefan

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Not sure what forum/mailinglist would have the right people, tho.

Neither am I.

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Fri, 10 Apr 2015 14:24:18 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20220@debbugs.gnu.org
> 
> > Date: Fri, 10 Apr 2015 06:16:46 -0500
> > From: Mario Valencia <mariovalspi@gmail.com>
> > Cc: Daniel Colascione <dancol@dancol.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 20220@debbugs.gnu.org
> > 
> > You are correct; the problem only happens with a file:/// url. The following
> > executes without problem and emacs memory stays constant:
> > (dotimes (i 100) (w32-shell-execute "open" "c:/users/mario/desktop/x.html"))
> > With a file:/// url it triggers the bug:
> > (dotimes (i 100) (w32-shell-execute "open"
> > "file:///c:/users/mario/desktop/x.html"))
> 
> Thanks, that's good to know.
> 
> So I think the solution is to change w32-shell-execute to convert
> file:/// URLs into the normal file-name format internally, and use
> that in the call to ShellExecute.

Now done in the development sources for Emacs 25.1 (commit f2e2cd5).

I'm closing this bug.  Feel free to reopen if some new information
emerges about this.

bug#20220: severe memory leak on emacs 24.4.1

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2377 bytes --]

On 03/29/2015 08:35 AM, Eli Zaretskii wrote:
>> Date: Sat, 28 Mar 2015 17:39:38 -0600
>> From: Mario Valencia <mariovalspi@gmail.com>
>>
>> to reproduce it, i create a small emtpy html document. then i evaluate the
>> following expression:
>>
>> (dotimes (i 100) (browse-url-of-file))
>>
>> This causes emacs to start opening the file using google chrome. In the task
>> manager, i can see emacs' memory usage go up by about 5 megabytes each time a
>> tab is opened. When it opens about 30 tabs, emacs is using 138 megabytes of
>> memory, and it gives the error below.
>> the translation is something like this: "ShellExecute failed: Storage space
>> insufficient to process this command"
>> My harddrive has enough storage space btw.
> 
> (The error is not about disk storage, it's about reserving the address
> space in virtual memory.)
> 
> FWIW, I don't see such a large memory increase when I reproduce this,
> I see something around 1MB, sometimes 1.5MB.  But I didn't try on
> Windows 8.
> 
> Anyway, Emacs doesn't allocate any memory when it calls the
> ShellExecute function, so I see no way we could leak something here.
> 
> My guess would be that invoking ShellExecute causes Windows to start a
> thread in the context of the Emacs process, and reserve 8MB of stack
> space for that thread.  (On one of 3 systems I tried your recipe, I
> actually saw a thread per invocation, each thread was running some
> function inside shlwapi.dll, the shared library which implements the
> ShellExecute API.)  The memory actually used by that thread for its
> stack is much smaller than 8MB, of course, but Windows attempts to
> reserve 8MB of address space for its stack.
> 
> The 8MB figure comes from the way we link Emacs: we need such a large
> stack due to regular expressions, stack-based Lisp objects, and GC
> which is deeply recursive.  By default, each thread reserves the same
> stack space as the program to which it belongs.  For threads we launch
> in Emacs, we override the default 8MB stack space by a much smaller
> value, but we have no such control on threads that Windows itself
> starts on behalf of the Emacs process.

Do we need the lisp thread to be the main thread? What about calling
CreateThread very early in initialization with a large dwStackSize,
leaving other threads with default-sized stacks?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#20220: severe memory leak on emacs 24.4.1

[Eli Zaretskii]

> Date: Wed, 01 Apr 2015 00:50:48 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: 20220@debbugs.gnu.org
> 
> Do we need the lisp thread to be the main thread? What about calling
> CreateThread very early in initialization with a large dwStackSize,
> leaving other threads with default-sized stacks?

Might be possible, I don't know.  For example, does any of the code
make any hidden assumptions that the Lisp thread is the main thread?

And in any case, if threads are created for each invocation, and never
die as long as Emacs runs, then sooner or later Emacs will run out of
address space and/or handles (I see on that single system where I can
reproduce the problem that each such thread consumes 2 handles, and I
don't see it exit).

So it sounds like trying to find and fix the reason(s) for this rare
behavior, or find a workaround (other than not using the API) is a
better alternative, at least at this point.