bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Jack Holloway]

[-- Attachment #1: Type: text/html, Size: 5694 bytes --]

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Stefan Monnier]

Thank you for your report.  But I find it difficult to read it because
it is in HTML format.  Can you please reply with a plain text version?
Thank you,


        Stefan

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Jack Holloway]

Sorry for the gaffe -- this should be plain text.  Let me know if it's not.
---------------------
Saving a 32MB file to a CIFS path fails.  Examining Procmon
events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
with length 32MB.

Perhaps write-region should break down the write into a series
of blksize writes?

Here's the relevant backtrace:
Debugger entered--Lisp error: (error "IO error writing 
z:/Temp/test/tmp2092CxD: Invalid argument")
   write-region(nil nil "z:/Temp/test/tmp2092CxD" nil 
"z:/Temp/test/temp-write-once-032768" 
"z:/Temp/test/temp-write-once-032768" excl)
   byte-code("..." [dir tempname realname buffer-file-truename 
make-temp-name expand-file-name "tmp" write-region nil excl] 8)
   basic-save-buffer-2()
   basic-save-buffer-1()
   basic-save-buffer()
   save-buffer()
   write-file("z:/Temp/test/temp-write-once-032768")

This was running on Windows XP64 Professional SP2.  The drive "z:" is
mapped to a NAS running Samba.  This error doesn't occur
when writing to the host's internal NTFS, it does occur with other
CIFS targets.  copy-file doesn't have this problem.

In GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790)
  of 2010-05-08 on G41R2F1
Windowing system distributor `Microsoft Corp.', version 5.2.3790
configured using `configure --with-gcc (3.4) --no-opt --cflags 
-Ic:/xpm/include'


On 5/21/2010 6:51 PM, Stefan Monnier wrote:
> Thank you for your report.  But I find it difficult to read it because
> it is in HTML format.  Can you please reply with a plain text version?
> Thank you,
>
>
>          Stefan
>
>    

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Eli Zaretskii]

> Date: Fri, 21 May 2010 19:12:41 -0700
> From: Jack Holloway <forum2@sprucehead.com>
> Cc: 6237@debbugs.gnu.org
> 
> Saving a 32MB file to a CIFS path fails.  Examining Procmon
> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
> with length 32MB.
> 
> Perhaps write-region should break down the write into a series
> of blksize writes?

Can you see which size of a single write succeeds with that volume?

Also, could you see how many bytes did it succeed to write before the
failure?

> copy-file doesn't have this problem.

The MS-Windows version of copy-file doesn't use WriteFile.  It uses
CopyFile.

Anyway, how did you use copy-file?  Was the source file on the same
CIFS volume, or was it on a local disk?

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Jack Holloway]

On 5/21/2010 11:31 PM, Eli Zaretskii wrote:
>> Date: Fri, 21 May 2010 19:12:41 -0700
>> From: Jack Holloway<forum2@sprucehead.com>
>> Cc: 6237@debbugs.gnu.org
>>
>> Saving a 32MB file to a CIFS path fails.  Examining Procmon
>> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
>> with length 32MB.
>>
>> Perhaps write-region should break down the write into a series
>> of blksize writes?
>>      
> Can you see which size of a single write succeeds with that volume?
>
> Also, could you see how many bytes did it succeed to write before the
> failure?
>    
I tried successively larger buffers starting at 32KB, viz.
Written 32K
Written 48K
Written 64K
Written 96K
Written 128K
Written 192K
Written 256K
Written 384K
Written 512K
Written 768K
Written 1024K
Written 1536K
Written 2048K
Written 3072K
Written 4096K
Written 6144K
Written 8192K
Written 12288K
Written 16384K
Written 24576K
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-032768: Invalid argument)
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-049152: Invalid argument)
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-065536: Invalid argument)
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-098304: Invalid argument)
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-131072: Invalid argument)
!!! Unable to append - (error IO error writing 
z:/Temp/test/temp-write-once-196608: Invalid argument)

The would-be 32MB file exists, with zero length.
>> copy-file doesn't have this problem.
>>      
> The MS-Windows version of copy-file doesn't use WriteFile.  It uses
> CopyFile.
>
> Anyway, how did you use copy-file?  Was the source file on the same
> CIFS volume, or was it on a local disk?
>
>    
I've tried all 4 combinations of {internal, CIFS}*{src,dest} with 
transfer sizes up to
196MB without a problem,

I was referring to DEFUN ("copy-file",...) in fileio.c which writes in 
16K blocks,
...
while ((n = emacs_read (ifd, buf, sizeof buf)) > 0)
     if (emacs_write (ofd, buf, n) != n)
       report_file_error ("I/O error", Fcons (newname, Qnil));
...

as opposed to DEFUN ("write-region",...) which, in this case, writes the 
entire buffer
...
else if (XINT (start) != XINT (end))
     {
       failure = 0 > a_write (desc, Qnil,
                  XINT (start), XINT (end) - XINT (start),
&annotations, &coding);
       save_errno = errno;
     }
...

I know nothing about Windows internals, but it would seem that some 
downstream
WinNT I/O routine can't handle 32MB or larger writes, so perhaps 
write-region could segment
the write?  The annotations hair makes it hard for me to see if this is 
easy.

     Jack

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Andreas Schwab]

Jack Holloway <forum2@sprucehead.com> writes:

> Sorry for the gaffe -- this should be plain text.  Let me know if it's not.
> ---------------------
> Saving a 32MB file to a CIFS path fails.  Examining Procmon
> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
> with length 32MB.
>
> Perhaps write-region should break down the write into a series
> of blksize writes?

See also this thread on the git mailing list:
<http://thread.gmane.org/gmane.comp.version-control.git/145294>.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Eli Zaretskii]

> Date: Sat, 22 May 2010 00:49:56 -0700
> From: Jack Holloway <forum2@sprucehead.com>
> CC: 6237@debbugs.gnu.org
> 
> Written 16384K
> Written 24576K
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-032768: Invalid argument)
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-049152: Invalid argument)
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-065536: Invalid argument)
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-098304: Invalid argument)
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-131072: Invalid argument)
> !!! Unable to append - (error IO error writing 
> z:/Temp/test/temp-write-once-196608: Invalid argument)
> 
> The would-be 32MB file exists, with zero length.

So 24MB succeeds, but 32MB and all sizes above it fail?

But why does the error message say "Unable to append"?  Why "append"?

> I was referring to DEFUN ("copy-file",...) in fileio.c which writes in 
> 16K blocks,
> ...
> while ((n = emacs_read (ifd, buf, sizeof buf)) > 0)
>      if (emacs_write (ofd, buf, n) != n)
>        report_file_error ("I/O error", Fcons (newname, Qnil));

This is not the code that runs on Windows.  That one is a little ways
earlier in the body of copy-file:

  #ifdef WINDOWSNT
    if (!CopyFile (SDATA (encoded_file),
		   SDATA (encoded_newname),
		   FALSE))
      report_file_error ("Copying file", Fcons (file, Fcons (newname, Qnil)));
    /* CopyFile retains the timestamp by default.  */
    else if (NILP (keep_time))
      {
	EMACS_TIME now;
	DWORD attributes;
	char * filename;

	EMACS_GET_TIME (now);
	filename = SDATA (encoded_newname);

	/* Ensure file is writable while its modified time is set.  */
	attributes = GetFileAttributes (filename);
	SetFileAttributes (filename, attributes & ~FILE_ATTRIBUTE_READONLY);
	if (set_file_times (filename, now, now))
	  {
	    /* Restore original attributes.  */
	    SetFileAttributes (filename, attributes);
	    xsignal2 (Qfile_date_error,
		      build_string ("Cannot set file date"), newname);
	  }
	/* Restore original attributes.  */
	SetFileAttributes (filename, attributes);
      }
  #else /* not WINDOWSNT */

> I know nothing about Windows internals, but it would seem that some
> downstream WinNT I/O routine can't handle 32MB or larger writes, so
> perhaps write-region could segment the write?  The annotations hair
> makes it hard for me to see if this is easy.

The breaking of writes into smaller chunks should be in Windows
specific code, i.e. in w32.c:sys_write.  And that is very easy.

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Eli Zaretskii]

> Date: Sat, 22 May 2010 11:38:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 6237@debbugs.gnu.org
> 
> > I know nothing about Windows internals, but it would seem that some
> > downstream WinNT I/O routine can't handle 32MB or larger writes, so
> > perhaps write-region could segment the write?  The annotations hair
> > makes it hard for me to see if this is easy.
> 
> The breaking of writes into smaller chunks should be in Windows
> specific code, i.e. in w32.c:sys_write.  And that is very easy.

If you can build Emacs yourself from sources, could you please try the
following patch?  It incorporates the info I found in the git mailing
list thread pointed out by Andreas (thanks!).


=== modified file 'src/w32.c'
--- src/w32.c	2010-05-01 18:28:00 +0000
+++ src/w32.c	2010-05-22 09:33:53 +0000
@@ -5700,7 +5700,34 @@ sys_write (int fd, const void * buffer, 
     }
   else
 #endif
-    nchars = _write (fd, buffer, count);
+    {
+      /* Some networked filesystems don't like too large writes, so
+	 break them into smaller chunks.  See the Comments section of
+	 the MSDN documentation of WriteFile for details behind the
+	 choice of the value of CHUNK below.  See also the thread
+	 http://thread.gmane.org/gmane.comp.version-control.git/145294
+	 in the git mailing list.  */
+      const unsigned char *p = buffer;
+      const unsigned chunk = 30 * 1024 * 1024;
+
+      nchars = 0;
+      while (count > 0)
+	{
+	  unsigned this_chunk = count < chunk ? count : chunk;
+	  int n = _write (fd, p, this_chunk);
+
+	  nchars += n;
+	  if (n < 0)
+	    {
+	      nchars = n;
+	      break;
+	    }
+	  else if (n < this_chunk)
+	    break;
+	  count -= n;
+	  p += n;
+	}
+    }
 
   return nchars;
 }

bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS

[Eli Zaretskii]

> Date: Sat, 22 May 2010 12:39:41 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 6237@debbugs.gnu.org
> 
> > Date: Sat, 22 May 2010 11:38:35 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: 6237@debbugs.gnu.org
> > 
> > > I know nothing about Windows internals, but it would seem that some
> > > downstream WinNT I/O routine can't handle 32MB or larger writes, so
> > > perhaps write-region could segment the write?  The annotations hair
> > > makes it hard for me to see if this is easy.
> > 
> > The breaking of writes into smaller chunks should be in Windows
> > specific code, i.e. in w32.c:sys_write.  And that is very easy.
> 
> If you can build Emacs yourself from sources, could you please try the
> following patch?  It incorporates the info I found in the git mailing
> list thread pointed out by Andreas (thanks!).

Now installed in the repository (revno 100416).