"resource temporarily unavailable" errors on windows 7

"resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

I keep encountering the following error each time I try to start a
subprocess under emacs:

   File error: Spawning child process, resource temporarily unavailable

This happens on Windows 7, 32 bit with the prebuilt binaries versions
23.4, 24.0.93 and 24.0.94 but *not* with Emacs 23.3.  It happens when
launching all kinds of sub-processes (compilation commands, version
control commands, diff, aspell, bash, etc).  It also happens on two
other colleagues computers, also running Windows 7, 32 bit.  If I set
the Windows XP compatibility flag on emacs.exe, the problem goes away,
but makes Emacs unusable for me, since the programs I need to run do
not work when the compatibility mode is set.

I run a diff between Emacs 23.3 and 23.4 and identified that a new
allocate_heap() function was added in 23.4.  If I remove the function,
falling back on the one used in 23.3 than I no longer get the above
error.  (I attached a patch to show which function I talk about, not
to suggest removing the function).

Should this version of allocate_heap() be used on a Windows 7, 32 bit
platform?

Alex.



diff -up --tabsize=4 c\:/emacs-24.0.94/src/w32heap.c\~
c\:/emacs-24.0.94/src/w32heap.c
--- c:/emacs-24.0.94/src/w32heap.c~	2012-02-14 00:13:25.000000000 +0800
+++ c:/emacs-24.0.94/src/w32heap.c	2012-03-02 10:38:51.591246600 +0800
@@ -114,7 +114,6 @@ get_data_end (void)
   return data_region_end;
 }

-#if !defined (USE_LISP_UNION_TYPE) && !defined (USE_LSB_TAG)
 static char *
 allocate_heap (void)
 {
@@ -141,27 +140,6 @@ allocate_heap (void)

   return ptr;
 }
-#else  /* USE_LISP_UNION_TYPE || USE_LSB_TAG */
-static char *
-allocate_heap (void)
-{
-  unsigned long size = 0x80000000; /* start by asking for 2GB */
-  void *ptr = NULL;
-
-  while (!ptr && size > 0x00100000)
-    {
-      reserved_heap_size = size;
-      ptr = VirtualAlloc (NULL,
-			  get_reserved_heap_size (),
-			  MEM_RESERVE,
-			  PAGE_NOACCESS);
-      size -= 0x00800000; /* if failed, decrease request by 8MB */
-    }
-
-  return ptr;
-}
-#endif /* USE_LISP_UNION_TYPE || USE_LSB_TAG */
-

 /* Emulate Unix sbrk.  Note that ralloc.c expects the return value to
    be the address of the _start_ (not end) of the new block in case of

Diff finished.  Mon Mar 12 07:47:19 2012

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Mon, 12 Mar 2012 12:56:13 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> 
> I keep encountering the following error each time I try to start a
> subprocess under emacs:
> 
>    File error: Spawning child process, resource temporarily unavailable
> 
> This happens on Windows 7, 32 bit with the prebuilt binaries versions
> 23.4, 24.0.93 and 24.0.94 but *not* with Emacs 23.3.  It happens when
> launching all kinds of sub-processes (compilation commands, version
> control commands, diff, aspell, bash, etc).  It also happens on two
> other colleagues computers, also running Windows 7, 32 bit.  If I set
> the Windows XP compatibility flag on emacs.exe, the problem goes away,
> but makes Emacs unusable for me, since the programs I need to run do
> not work when the compatibility mode is set.
> 
> I run a diff between Emacs 23.3 and 23.4 and identified that a new
> allocate_heap() function was added in 23.4.  If I remove the function,
> falling back on the one used in 23.3 than I no longer get the above
> error.  (I attached a patch to show which function I talk about, not
> to suggest removing the function).
> 
> Should this version of allocate_heap() be used on a Windows 7, 32 bit
> platform?

I don't see why not; we need more specific information to make a
decision.

This function is responsible for reserving memory for the Emacs
process at startup.

The question is, why reserving the memory the way allocate_heap does
causes trouble for you.  What that function does is start by asking
for 2GB or reserved memory, and if that fails, asks for less and less
in steps of 8MB.  Can you step through that function with a debugger
and see how much memory it eventually allocates?

Using the version that assumes no USE_LSB_TAG gives you only 256MB,
which is too low a number.  This could be a fallback if we find no
better way, but I'd like first to understand why the current code
fails and how.

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/13 Eli Zaretskii <eliz@gnu.org>:
>> Date: Mon, 12 Mar 2012 12:56:13 +0800
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>>
>>
>> Should this version of allocate_heap() be used on a Windows 7, 32 bit
>> platform?
>
> I don't see why not; we need more specific information to make a
> decision.
>
> This function is responsible for reserving memory for the Emacs
> process at startup.
>
> The question is, why reserving the memory the way allocate_heap does
> causes trouble for you.  What that function does is start by asking
> for 2GB or reserved memory, and if that fails, asks for less and less
> in steps of 8MB.  Can you step through that function with a debugger
> and see how much memory it eventually allocates?

The new alocate_heap function allocates 1'811'939'328 bytes of memory
(approx 1.8Gb).  If I run emacs.exe in Windows XP compatibility mode,
the function allocates only 1'468'006'400 (1.4Gb), in this case I also
get the following warning in GDB:

   warning: FTH: (6936): *** Fault tolerant heap shim applied to
current process. This is usually due to previous crashes. ***

My understanding is that in a 32bit windows process, only 2Gb of
address space is usable by the application.  Could this mean that
after alocate_heap() is called, the entire address space of the emacs
process is allocated and CreateProcess() cannot allocate memory for
its internal needs?

Since only 1.4 Gb are allocated in Windows XP compatibility mode, this
would leave 0.4 Gb free to be used by CreateProcess(), so subprocesses
can be started without problems.

Alex.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Tue, 13 Mar 2012 08:18:10 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> My understanding is that in a 32bit windows process, only 2Gb of
> address space is usable by the application.

That's true.

> Could this mean that after alocate_heap() is called, the entire
> address space of the emacs process is allocated and CreateProcess()
> cannot allocate memory for its internal needs?

The function we are talking about does not _allocate_ memory, it just
_reserves_ it.  Memory is only allocated (i.e. committed) when Emacs
actually needs it (and decommitted when it's no longer needed).

> Since only 1.4 Gb are allocated in Windows XP compatibility mode, this
> would leave 0.4 Gb free to be used by CreateProcess(), so subprocesses
> can be started without problems.

How much memory is "left" after allocate_heap does its job depends on
how much _virtual_ memory do you have.  IOW, the 2GB number is
_per_application_, not for _all_ applications.  See my other mail for
more practical questions.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

[Please keep the list on the CC.]

> Date: Tue, 13 Mar 2012 09:31:18 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> 
> Just a quick update: I modified allocate_heap() to start asking for
> 1.5GB initially and that also seemed to fix the subprocess creation
> problem.

What value exactly did you use?

Also, how much virtual memory do you have on that machine?  Is it
possible that enlarging the amount of virtual memory lets Emacs run OK
even with the original 1.8GB allocation size?

Finally, what happens if you start an Emacs session (with the 1.5GB
allocation that you say work for you), and then start another Emacs
session without exiting the first one?  Does the second session start
successfully and is able to start subprocesses normally?

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/13 Eli Zaretskii <eliz@gnu.org>:
> [Please keep the list on the CC.]
>
>> Date: Tue, 13 Mar 2012 09:31:18 +0800
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>>
>> Just a quick update: I modified allocate_heap() to start asking for
>> 1.5GB initially and that also seemed to fix the subprocess creation
>> problem.
>
> What value exactly did you use?

  unsigned long size = 0x60000000; /* start by asking for 1.5GB */

>
> Also, how much virtual memory do you have on that machine?

The machine has 6GB of physical memory, 3GB usable (800Mb free), total
virtual memory is 6Gb (3GB free).  I don't think I'm running out of
memory.

> Is it
> possible that enlarging the amount of virtual memory lets Emacs run OK
> even with the original 1.8GB allocation size?

Since I have more memory available than Emacs requests, I don't think
enlarging the amount of virtual memory would help.  Starting a new
Emacs session does not seem to affect the amount of free memory
available significantly: my Emacs session uses 25Mb of physical memory
after start-up.  My main Emacs session (where I do my work) currently
uses 63Mb, not a very large value.  The 1.5GB "reservation" by
allocate_heap() does not decrease the amount of available virtual
memory.

>
> Finally, what happens if you start an Emacs session (with the 1.5GB
> allocation that you say work for you), and then start another Emacs
> session without exiting the first one?  Does the second session start
> successfully and is able to start subprocesses normally?

Yes, I can start a second Emacs session and subprocess creation works
fine.  In fact, I started 5 simultaneous sessions and tested that for
good measure :-)


Perhaps I was a bit imprecise with the terminology, but I did realise
that allocate_heap() only reserves the memory and that the 2GB address
space limit is per process, not per system.

There seems to be some documentation on the net that ANSI
CreateProcess will allocate memory to convert the command line to
UNICODE before calling the UNICODE version of CreateProcess, this is
what I meant by "CreateProcess fails to allocate memory". Could this
memory  allocation fail if all the memory is already reserved for
something
else?

Alex.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Tue, 13 Mar 2012 12:50:09 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> 2012/3/13 Eli Zaretskii <eliz@gnu.org>:
> > [Please keep the list on the CC.]
> >
> >> Date: Tue, 13 Mar 2012 09:31:18 +0800
> >> From: Alex Harsanyi <alexharsanyi@gmail.com>
> >>
> >> Just a quick update: I modified allocate_heap() to start asking for
> >> 1.5GB initially and that also seemed to fix the subprocess creation
> >> problem.
> >
> > What value exactly did you use?
> 
>   unsigned long size = 0x60000000; /* start by asking for 1.5GB */

Is this the largest value that gives you a fully functional Emacs?  Or
can you enlarge it some more, before subprocesses start failing?  I'm
not happy reducing the maximum memory possible by 0.5GB, especially
since no one else complained about this yet, including people who run
Emacs on Windows 7.  (But I still hope that we can find a better
solution than just reducing the memory request; see below.  So this is
for now a fallback.)

> Since I have more memory available than Emacs requests, I don't think
> enlarging the amount of virtual memory would help.  Starting a new
> Emacs session does not seem to affect the amount of free memory
> available significantly: my Emacs session uses 25Mb of physical memory
> after start-up.  My main Emacs session (where I do my work) currently
> uses 63Mb, not a very large value.  The 1.5GB "reservation" by
> allocate_heap() does not decrease the amount of available virtual
> memory.

OK, good.  This seems to indicate that the problem is with the memory
of the Emacs process itself, not with the memory for other processes,
such as the subprocess you are trying to launch.  Most probably,
something we do during spawning the subprocess tries to reserve
memory, similarly to what allocate_heap does, and fails because the
sum of what's already reserved and this new reservation exceeds 2GB.

> Perhaps I was a bit imprecise with the terminology, but I did realise
> that allocate_heap() only reserves the memory and that the 2GB address
> space limit is per process, not per system.

Sorry, I couldn't be certain that you understand all the details about
that function.

> There seems to be some documentation on the net that ANSI
> CreateProcess will allocate memory to convert the command line to
> UNICODE before calling the UNICODE version of CreateProcess, this is
> what I meant by "CreateProcess fails to allocate memory". Could this
> memory  allocation fail if all the memory is already reserved for
> something else?

It could in theory, but I very much doubt that conversion of any
reasonable command line to UTF-16 could ever need so much memory
(0.2GB, the difference between the 1.8GB allocation you reported and
the 2GB limit).

Anyway, we have no evidence that the error comes from the
CreateProcess API.  On the contrary (see below), I think we have
evidence that it comes from elsewhere.

I thought about this some more, read some stuff on MSDN and elsewhere,
and I have the following questions and observations.

First, the error message.  It is triggered by this code in
w32proc.c:sys_spawnve:

  cp = new_child ();
  if (cp == NULL)
    {
      errno = EAGAIN;
      return -1;
    }

Now, in new_child we see this:

    cp->char_avail = CreateEvent (NULL, TRUE, FALSE, NULL);
    if (cp->char_avail)
      {
	cp->char_consumed = CreateEvent (NULL, FALSE, FALSE, NULL);
	if (cp->char_consumed)
	  {
	    cp->thrd = CreateThread (NULL, 1024, reader_thread, cp, 0, &id);
	    if (cp->thrd)
	      return cp;
	  }
      }
    delete_child (cp);
    return NULL;
  }

So this can happen due to error either in one of the two calls to
CreateEvent or in the call to CreateThread.  My crystal ball bets on
the latter, but could you please verify that this is the problem?  If
it is, please add a call to GetLastError there and see which error
code is returned.

Assuming it's CreateThread that fails, the obvious suspicion would be
that it fails because creating a thread reserves some memory for its
stack.  The call above is careful to override the amount of stack
memory that is actually _committed_ and sets it at a mere 1KB, but the
memory _reserved_ for the stack is still 1MB, according to MSDN.

Question: does the problem happen right when you invoke the first
subprocess, or only after several subprocesses are already running,
and you try to invoke another one?

Also, can you use the VMMap utility (from Sysinternals) and tell what
it shows in the "Private Data", "Thread Stack", and "Free" categories,
when you run Emacs with the original code in allocate_heap, after you
get the error message?  Can you spot, in particular, how much memory
is reserved  and committed for Emacs and for the additional thread it
starts in new_child?

TIA

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

[-- Attachment #1: Type: text/plain, Size: 2880 bytes --]

2012/3/14 Eli Zaretskii <eliz@gnu.org>:

> So this can happen due to error either in one of the two calls to
> CreateEvent or in the call to CreateThread.  My crystal ball bets on
> the latter, but could you please verify that this is the problem?  If
> it is, please add a call to GetLastError there and see which error
> code is returned.

Indeed it fails on the CreateThread call, GetLastError returns 8: "Not
enough storage is available to process this command. "

>
> Assuming it's CreateThread that fails, the obvious suspicion would be
> that it fails because creating a thread reserves some memory for its
> stack.  The call above is careful to override the amount of stack
> memory that is actually _committed_ and sets it at a mere 1KB, but the
> memory _reserved_ for the stack is still 1MB, according to MSDN.
>
> Question: does the problem happen right when you invoke the first
> subprocess, or only after several subprocesses are already running,
> and you try to invoke another one?

It does not happen all the time, I'd say about 1 out of 4 tries for a
`M-x diff-backup RET' call.  In terms of running processes,
(process-list) returns:

    (#<process ielm> #<process cleartool> #<process edit-server>
#<process server>)

BTW, it also happens when I run "emacs -Q", but less often, although
when I tested the CreateThread failure, I run emacs -Q under gdb and
it failed on the third try.

>
> Also, can you use the VMMap utility (from Sysinternals) and tell what
> it shows in the "Private Data", "Thread Stack", and "Free" categories,
> when you run Emacs with the original code in allocate_heap, after you
> get the error message?  Can you spot, in particular, how much memory
> is reserved  and committed for Emacs and for the additional thread it
> starts in new_child?

I attached the VMMap for the Emacs processs, right after a failure.

On an "Emacs -Q" process, I have:

  Stack                49'152K  (224K commited)
  Private Data   1'796'128K  (2'136K commited)
  Free                142'528K

Each time I start a shell process "M-x shell" (I rename the buffers,
so I have multiple shells running), the stack reserved size grows by
about 8Mb, the commited size by about 12K and the free memory
decreases by 8Mb.  The Private Data commited size also changes by a
few KB.

When I start emacs normally (reading the init files), the Free section
has 47'224K and decreases by 8Mb for each shell process I start (Stack
increases accordingly).

I also noticed that spontaneously, the Stack reserved size grows by
about 10-12Mb (Free section decreases accordingly), than the memory is
released a few seconds later.  By spontaneous, I mean that I don't
interact with the Emacs session, I just hit refresh on the VMMap
application.  This happens also when I start emacs with "emacs -Q".

Alex.

[-- Attachment #2: emacs-vmmap.zip --]
[-- Type: application/zip, Size: 64079 bytes --]

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/14 Alex Harsanyi <alexharsanyi@gmail.com>:

> Each time I start a shell process "M-x shell" (I rename the buffers,
> so I have multiple shells running), the stack reserved size grows by
> about 8Mb, the commited size by about 12K and the free memory
> decreases by 8Mb.  The Private Data commited size also changes by a
> few KB.

I checked the executable headers, and the default thread stack size is
set to 8Mb. "objdump -p emacs.exe" lists (among other things):

    SizeOfStackReserve      00800000
    SizeOfStackCommit      00001000

I also read the MSDN documentation for CreateThread and it looks like
the dwStackSize parameter represents the commit size unless the
STACK_SIZE_PARAM_IS_A_RESERVATION flag is passed to CreateThread.   I
tried adding this flag, but it was not recognized by mingw so I
replaced it with the value (0x00010000).

When I recompiled, the resulting emacs would reserve 1Mb of stack for
each new thread.  I than tried a few values for the dwStackSize
parameter and the smallest stack reservation I could obtain was 64K,
Here is the call:

   cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
0x00010000, &id);

It also seems that the reserved stack size reverts to 1Mb if
dwStackSize is too small: the folowing values for dwStackSize will
result in 1Mb being reserved:  4, 1024, 4*1024 and the following
values will cause 64K to be reserved: 8*1024, 32*1024, 64*1024.

Alex.





Alex.


>
> When I start emacs normally (reading the init files), the Free section
> has 47'224K and decreases by 8Mb for each shell process I start (Stack
> increases accordingly).
>
> I also noticed that spontaneously, the Stack reserved size grows by
> about 10-12Mb (Free section decreases accordingly), than the memory is
> released a few seconds later.  By spontaneous, I mean that I don't
> interact with the Emacs session, I just hit refresh on the VMMap
> application.  This happens also when I start emacs with "emacs -Q".
>
> Alex.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Wed, 14 Mar 2012 13:34:42 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> I also read the MSDN documentation for CreateThread and it looks like
> the dwStackSize parameter represents the commit size unless the
> STACK_SIZE_PARAM_IS_A_RESERVATION flag is passed to CreateThread.   I
> tried adding this flag, but it was not recognized by mingw so I
> replaced it with the value (0x00010000).
> 
> When I recompiled, the resulting emacs would reserve 1Mb of stack for
> each new thread.  I than tried a few values for the dwStackSize
> parameter and the smallest stack reservation I could obtain was 64K,
> Here is the call:
> 
>    cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
> 0x00010000, &id);

And does this smaller stack reservation improve the situation, if you
return the initial memory request in allocate_heap to its original
value.

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/15 Eli Zaretskii <eliz@gnu.org>:
>> Date: Wed, 14 Mar 2012 13:34:42 +0800
>> From: Alex Harsanyi <alexharsanyi@gmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> I also read the MSDN documentation for CreateThread and it looks like
>> the dwStackSize parameter represents the commit size unless the
>> STACK_SIZE_PARAM_IS_A_RESERVATION flag is passed to CreateThread.   I
>> tried adding this flag, but it was not recognized by mingw so I
>> replaced it with the value (0x00010000).
>>
>> When I recompiled, the resulting emacs would reserve 1Mb of stack for
>> each new thread.  I than tried a few values for the dwStackSize
>> parameter and the smallest stack reservation I could obtain was 64K,
>> Here is the call:
>>
>>    cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
>> 0x00010000, &id);
>
> And does this smaller stack reservation improve the situation, if you
> return the initial memory request in allocate_heap to its original
> value.
>

Yes it does.  I'm using a build with allocate_heap starting at 2GB
(the original value) and this updated CreateThread call as my working
Emacs.   I will continue to use it and I will report if I see any more
errors.

Also, the strange dwStackSize  interpretation by CreateThread was
tested on Windows 7 32 bit only.   I'm not sure how this behaves on
other windows versions.

Perhaps at minimum, the mingw build should be updated to use 1Mb stack
reservation in the program header?

Alex.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Thu, 15 Mar 2012 08:02:42 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> >>    cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
> >> 0x00010000, &id);
> >
> > And does this smaller stack reservation improve the situation, if you
> > return the initial memory request in allocate_heap to its original
> > value.
> >
> 
> Yes it does.  I'm using a build with allocate_heap starting at 2GB
> (the original value) and this updated CreateThread call as my working
> Emacs.   I will continue to use it and I will report if I see any more
> errors.

OK, thanks.  If you don't see any problems with this, we could think
about making this fix permanent.

> Also, the strange dwStackSize  interpretation by CreateThread was
> tested on Windows 7 32 bit only.   I'm not sure how this behaves on
> other windows versions.

Since we are in pretest, the prudent thing would be to condition this
fix on that combination only.

> Perhaps at minimum, the mingw build should be updated to use 1Mb stack
> reservation in the program header?

I don't want to do that in general, since Emacs on Windows uses
threads for features other than interaction with subprocesses, and
those other threads might need larger stack.  Also, doesn't the value
in the header determine the stack size of the main thread as well?  If
it does, then 8MB is there for a reason: Emacs devours a lot of stack
space when it does GC, because GC is an extremely recursive process.

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Thu, 15 Mar 2012 06:04:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Thu, 15 Mar 2012 08:02:42 +0800
> > From: Alex Harsanyi <alexharsanyi@gmail.com>
> > Cc: emacs-devel@gnu.org
> > 
> > >>    cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
> > >> 0x00010000, &id);
> > >
> > > And does this smaller stack reservation improve the situation, if you
> > > return the initial memory request in allocate_heap to its original
> > > value.
> > >
> > 
> > Yes it does.  I'm using a build with allocate_heap starting at 2GB
> > (the original value) and this updated CreateThread call as my working
> > Emacs.   I will continue to use it and I will report if I see any more
> > errors.
> 
> OK, thanks.  If you don't see any problems with this, we could think
> about making this fix permanent.

As part of your testing, please try the code below, which should allow
you to see whether Emacs can launch many simultaneous subprocesses (it
can support up to 31 on Windows).

Thanks.


 (progn
   (dotimes (i 25)
     (start-process "hi" nil "ls")
     (message "%d" i)))

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/16 Eli Zaretskii <eliz@gnu.org>:
>> Date: Thu, 15 Mar 2012 06:04:00 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>>
>> > Date: Thu, 15 Mar 2012 08:02:42 +0800
>> > From: Alex Harsanyi <alexharsanyi@gmail.com>
>> > Cc: emacs-devel@gnu.org
>> >
>> > >>    cp->thrd = CreateThread (NULL, 64 * 1024, reader_thread, cp,
>> > >> 0x00010000, &id);
>> > >
>> > > And does this smaller stack reservation improve the situation, if you
>> > > return the initial memory request in allocate_heap to its original
>> > > value.
>> > >
>> >
>> > Yes it does.  I'm using a build with allocate_heap starting at 2GB
>> > (the original value) and this updated CreateThread call as my working
>> > Emacs.   I will continue to use it and I will report if I see any more
>> > errors.
>>
>> OK, thanks.  If you don't see any problems with this, we could think
>> about making this fix permanent.
>
> As part of your testing, please try the code below, which should allow
> you to see whether Emacs can launch many simultaneous subprocesses (it
> can support up to 31 on Windows).
>
> Thanks.
>
>
>  (progn
>   (dotimes (i 25)
>     (start-process "hi" nil "ls")
>     (message "%d" i)))

The above code worked fine on my fixed Emacs.
Alex

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Fri, 16 Mar 2012 09:12:50 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> >  (progn
> >   (dotimes (i 25)
> >     (start-process "hi" nil "ls")
> >     (message "%d" i)))
> 
> The above code worked fine on my fixed Emacs.

Great, thanks.

Re: "resource temporarily unavailable" errors on windows 7

[Jason Rumney]

Alex Harsanyi <alexharsanyi@gmail.com> wrote:

>Also, the strange dwStackSize  interpretation by CreateThread was
>tested on Windows 7 32 bit only.   I'm not sure how this behaves on
>other windows versions.

I saw this error on Windows 7 64 bit too just recently (using 32 bit Emacs)

Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Wed, 14 Mar 2012 12:27:31 +0800
> From: Alex Harsanyi <alexharsanyi@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> > Question: does the problem happen right when you invoke the first
> > subprocess, or only after several subprocesses are already running,
> > and you try to invoke another one?
> 
> It does not happen all the time, I'd say about 1 out of 4 tries for a
> `M-x diff-backup RET' call.  In terms of running processes,
> (process-list) returns:
> 
>     (#<process ielm> #<process cleartool> #<process edit-server>
> #<process server>)
> 
> BTW, it also happens when I run "emacs -Q", but less often, although
> when I tested the CreateThread failure, I run emacs -Q under gdb and
> it failed on the third try.

"On the third try" meaning that you have 2 other subprocesses running,
or did you invoke a process, then terminated it, then invoked another,
etc. 3 times?

> I also noticed that spontaneously, the Stack reserved size grows by
> about 10-12Mb (Free section decreases accordingly), than the memory is
> released a few seconds later.

Is that for the Emacs process?  If so, I would suspect that GC kicked
in.  You can see if that's so by setting garbage-collection-messages
non-nil.

Did you try increasing the size of the original request in
allocate_heap, and if you did, what was the maximum value that still
gives you a good Emacs session?

Thanks for working on this.

Re: "resource temporarily unavailable" errors on windows 7

[Alex Harsanyi]

2012/3/15 Eli Zaretskii <eliz@gnu.org>:
>
> "On the third try" meaning that you have 2 other subprocesses running,
> or did you invoke a process, then terminated it, then invoked another,
> etc. 3 times?

The sub-process was diff, I was running `M-x diff-backup RET' on
w32proc.c and waiting for it to finish, than running it again.  I just
checked the source for diff-backup and it seems that it runs diff in
async mode.  Since I just looked at the *vc-diff* buffer, it is
possible that one run of diff overlapped over the next one.  So
probably I had three parallel processes running, or at least
reader_thread did not terminate before the next process started.

>
>> I also noticed that spontaneously, the Stack reserved size grows by
>> about 10-12Mb (Free section decreases accordingly), than the memory is
>> released a few seconds later.
>
> Is that for the Emacs process?  If so, I would suspect that GC kicked
> in.  You can see if that's so by setting garbage-collection-messages
> non-nil.

This is a good explanation

>
> Did you try increasing the size of the original request in
> allocate_heap, and if you did, what was the maximum value that still
> gives you a good Emacs session?

All this was done with allocate_heap starting the request at 2GB.
As I mentioned in a previous email, this ended up with approx 1.8Gb
being reserved for the heap.

> Thanks for working on this.

Thanks for helping me out.

Alex