* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
@ 2010-05-21 2:57 Jack Holloway
2010-05-22 1:51 ` Stefan Monnier
0 siblings, 1 reply; 9+ messages in thread
From: Jack Holloway @ 2010-05-21 2:57 UTC (permalink / raw
To: 6237
[-- Attachment #1: Type: text/html, Size: 5694 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-21 2:57 bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS Jack Holloway
@ 2010-05-22 1:51 ` Stefan Monnier
2010-05-22 2:12 ` Jack Holloway
0 siblings, 1 reply; 9+ messages in thread
From: Stefan Monnier @ 2010-05-22 1:51 UTC (permalink / raw
To: Jack Holloway; +Cc: 6237
Thank you for your report. But I find it difficult to read it because
it is in HTML format. Can you please reply with a plain text version?
Thank you,
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 1:51 ` Stefan Monnier
@ 2010-05-22 2:12 ` Jack Holloway
2010-05-22 6:31 ` Eli Zaretskii
2010-05-22 8:08 ` Andreas Schwab
0 siblings, 2 replies; 9+ messages in thread
From: Jack Holloway @ 2010-05-22 2:12 UTC (permalink / raw
To: Stefan Monnier; +Cc: 6237
Sorry for the gaffe -- this should be plain text. Let me know if it's not.
---------------------
Saving a 32MB file to a CIFS path fails. Examining Procmon
events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
with length 32MB.
Perhaps write-region should break down the write into a series
of blksize writes?
Here's the relevant backtrace:
Debugger entered--Lisp error: (error "IO error writing
z:/Temp/test/tmp2092CxD: Invalid argument")
write-region(nil nil "z:/Temp/test/tmp2092CxD" nil
"z:/Temp/test/temp-write-once-032768"
"z:/Temp/test/temp-write-once-032768" excl)
byte-code("..." [dir tempname realname buffer-file-truename
make-temp-name expand-file-name "tmp" write-region nil excl] 8)
basic-save-buffer-2()
basic-save-buffer-1()
basic-save-buffer()
save-buffer()
write-file("z:/Temp/test/temp-write-once-032768")
This was running on Windows XP64 Professional SP2. The drive "z:" is
mapped to a NAS running Samba. This error doesn't occur
when writing to the host's internal NTFS, it does occur with other
CIFS targets. copy-file doesn't have this problem.
In GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790)
of 2010-05-08 on G41R2F1
Windowing system distributor `Microsoft Corp.', version 5.2.3790
configured using `configure --with-gcc (3.4) --no-opt --cflags
-Ic:/xpm/include'
On 5/21/2010 6:51 PM, Stefan Monnier wrote:
> Thank you for your report. But I find it difficult to read it because
> it is in HTML format. Can you please reply with a plain text version?
> Thank you,
>
>
> Stefan
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 2:12 ` Jack Holloway
@ 2010-05-22 6:31 ` Eli Zaretskii
2010-05-22 7:49 ` Jack Holloway
2010-05-22 8:08 ` Andreas Schwab
1 sibling, 1 reply; 9+ messages in thread
From: Eli Zaretskii @ 2010-05-22 6:31 UTC (permalink / raw
To: Jack Holloway; +Cc: 6237
> Date: Fri, 21 May 2010 19:12:41 -0700
> From: Jack Holloway <forum2@sprucehead.com>
> Cc: 6237@debbugs.gnu.org
>
> Saving a 32MB file to a CIFS path fails. Examining Procmon
> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
> with length 32MB.
>
> Perhaps write-region should break down the write into a series
> of blksize writes?
Can you see which size of a single write succeeds with that volume?
Also, could you see how many bytes did it succeed to write before the
failure?
> copy-file doesn't have this problem.
The MS-Windows version of copy-file doesn't use WriteFile. It uses
CopyFile.
Anyway, how did you use copy-file? Was the source file on the same
CIFS volume, or was it on a local disk?
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 6:31 ` Eli Zaretskii
@ 2010-05-22 7:49 ` Jack Holloway
2010-05-22 8:38 ` Eli Zaretskii
0 siblings, 1 reply; 9+ messages in thread
From: Jack Holloway @ 2010-05-22 7:49 UTC (permalink / raw
To: Eli Zaretskii; +Cc: 6237
On 5/21/2010 11:31 PM, Eli Zaretskii wrote:
>> Date: Fri, 21 May 2010 19:12:41 -0700
>> From: Jack Holloway<forum2@sprucehead.com>
>> Cc: 6237@debbugs.gnu.org
>>
>> Saving a 32MB file to a CIFS path fails. Examining Procmon
>> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
>> with length 32MB.
>>
>> Perhaps write-region should break down the write into a series
>> of blksize writes?
>>
> Can you see which size of a single write succeeds with that volume?
>
> Also, could you see how many bytes did it succeed to write before the
> failure?
>
I tried successively larger buffers starting at 32KB, viz.
Written 32K
Written 48K
Written 64K
Written 96K
Written 128K
Written 192K
Written 256K
Written 384K
Written 512K
Written 768K
Written 1024K
Written 1536K
Written 2048K
Written 3072K
Written 4096K
Written 6144K
Written 8192K
Written 12288K
Written 16384K
Written 24576K
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-032768: Invalid argument)
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-049152: Invalid argument)
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-065536: Invalid argument)
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-098304: Invalid argument)
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-131072: Invalid argument)
!!! Unable to append - (error IO error writing
z:/Temp/test/temp-write-once-196608: Invalid argument)
The would-be 32MB file exists, with zero length.
>> copy-file doesn't have this problem.
>>
> The MS-Windows version of copy-file doesn't use WriteFile. It uses
> CopyFile.
>
> Anyway, how did you use copy-file? Was the source file on the same
> CIFS volume, or was it on a local disk?
>
>
I've tried all 4 combinations of {internal, CIFS}*{src,dest} with
transfer sizes up to
196MB without a problem,
I was referring to DEFUN ("copy-file",...) in fileio.c which writes in
16K blocks,
...
while ((n = emacs_read (ifd, buf, sizeof buf)) > 0)
if (emacs_write (ofd, buf, n) != n)
report_file_error ("I/O error", Fcons (newname, Qnil));
...
as opposed to DEFUN ("write-region",...) which, in this case, writes the
entire buffer
...
else if (XINT (start) != XINT (end))
{
failure = 0 > a_write (desc, Qnil,
XINT (start), XINT (end) - XINT (start),
&annotations, &coding);
save_errno = errno;
}
...
I know nothing about Windows internals, but it would seem that some
downstream
WinNT I/O routine can't handle 32MB or larger writes, so perhaps
write-region could segment
the write? The annotations hair makes it hard for me to see if this is
easy.
Jack
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 2:12 ` Jack Holloway
2010-05-22 6:31 ` Eli Zaretskii
@ 2010-05-22 8:08 ` Andreas Schwab
1 sibling, 0 replies; 9+ messages in thread
From: Andreas Schwab @ 2010-05-22 8:08 UTC (permalink / raw
To: Jack Holloway; +Cc: 6237
Jack Holloway <forum2@sprucehead.com> writes:
> Sorry for the gaffe -- this should be plain text. Let me know if it's not.
> ---------------------
> Saving a 32MB file to a CIFS path fails. Examining Procmon
> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile
> with length 32MB.
>
> Perhaps write-region should break down the write into a series
> of blksize writes?
See also this thread on the git mailing list:
<http://thread.gmane.org/gmane.comp.version-control.git/145294>.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 7:49 ` Jack Holloway
@ 2010-05-22 8:38 ` Eli Zaretskii
2010-05-22 9:39 ` Eli Zaretskii
0 siblings, 1 reply; 9+ messages in thread
From: Eli Zaretskii @ 2010-05-22 8:38 UTC (permalink / raw
To: Jack Holloway; +Cc: 6237
> Date: Sat, 22 May 2010 00:49:56 -0700
> From: Jack Holloway <forum2@sprucehead.com>
> CC: 6237@debbugs.gnu.org
>
> Written 16384K
> Written 24576K
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-032768: Invalid argument)
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-049152: Invalid argument)
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-065536: Invalid argument)
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-098304: Invalid argument)
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-131072: Invalid argument)
> !!! Unable to append - (error IO error writing
> z:/Temp/test/temp-write-once-196608: Invalid argument)
>
> The would-be 32MB file exists, with zero length.
So 24MB succeeds, but 32MB and all sizes above it fail?
But why does the error message say "Unable to append"? Why "append"?
> I was referring to DEFUN ("copy-file",...) in fileio.c which writes in
> 16K blocks,
> ...
> while ((n = emacs_read (ifd, buf, sizeof buf)) > 0)
> if (emacs_write (ofd, buf, n) != n)
> report_file_error ("I/O error", Fcons (newname, Qnil));
This is not the code that runs on Windows. That one is a little ways
earlier in the body of copy-file:
#ifdef WINDOWSNT
if (!CopyFile (SDATA (encoded_file),
SDATA (encoded_newname),
FALSE))
report_file_error ("Copying file", Fcons (file, Fcons (newname, Qnil)));
/* CopyFile retains the timestamp by default. */
else if (NILP (keep_time))
{
EMACS_TIME now;
DWORD attributes;
char * filename;
EMACS_GET_TIME (now);
filename = SDATA (encoded_newname);
/* Ensure file is writable while its modified time is set. */
attributes = GetFileAttributes (filename);
SetFileAttributes (filename, attributes & ~FILE_ATTRIBUTE_READONLY);
if (set_file_times (filename, now, now))
{
/* Restore original attributes. */
SetFileAttributes (filename, attributes);
xsignal2 (Qfile_date_error,
build_string ("Cannot set file date"), newname);
}
/* Restore original attributes. */
SetFileAttributes (filename, attributes);
}
#else /* not WINDOWSNT */
> I know nothing about Windows internals, but it would seem that some
> downstream WinNT I/O routine can't handle 32MB or larger writes, so
> perhaps write-region could segment the write? The annotations hair
> makes it hard for me to see if this is easy.
The breaking of writes into smaller chunks should be in Windows
specific code, i.e. in w32.c:sys_write. And that is very easy.
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 8:38 ` Eli Zaretskii
@ 2010-05-22 9:39 ` Eli Zaretskii
2010-05-22 19:13 ` Eli Zaretskii
0 siblings, 1 reply; 9+ messages in thread
From: Eli Zaretskii @ 2010-05-22 9:39 UTC (permalink / raw
To: forum2; +Cc: 6237
> Date: Sat, 22 May 2010 11:38:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 6237@debbugs.gnu.org
>
> > I know nothing about Windows internals, but it would seem that some
> > downstream WinNT I/O routine can't handle 32MB or larger writes, so
> > perhaps write-region could segment the write? The annotations hair
> > makes it hard for me to see if this is easy.
>
> The breaking of writes into smaller chunks should be in Windows
> specific code, i.e. in w32.c:sys_write. And that is very easy.
If you can build Emacs yourself from sources, could you please try the
following patch? It incorporates the info I found in the git mailing
list thread pointed out by Andreas (thanks!).
=== modified file 'src/w32.c'
--- src/w32.c 2010-05-01 18:28:00 +0000
+++ src/w32.c 2010-05-22 09:33:53 +0000
@@ -5700,7 +5700,34 @@ sys_write (int fd, const void * buffer,
}
else
#endif
- nchars = _write (fd, buffer, count);
+ {
+ /* Some networked filesystems don't like too large writes, so
+ break them into smaller chunks. See the Comments section of
+ the MSDN documentation of WriteFile for details behind the
+ choice of the value of CHUNK below. See also the thread
+ http://thread.gmane.org/gmane.comp.version-control.git/145294
+ in the git mailing list. */
+ const unsigned char *p = buffer;
+ const unsigned chunk = 30 * 1024 * 1024;
+
+ nchars = 0;
+ while (count > 0)
+ {
+ unsigned this_chunk = count < chunk ? count : chunk;
+ int n = _write (fd, p, this_chunk);
+
+ nchars += n;
+ if (n < 0)
+ {
+ nchars = n;
+ break;
+ }
+ else if (n < this_chunk)
+ break;
+ count -= n;
+ p += n;
+ }
+ }
return nchars;
}
^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS
2010-05-22 9:39 ` Eli Zaretskii
@ 2010-05-22 19:13 ` Eli Zaretskii
0 siblings, 0 replies; 9+ messages in thread
From: Eli Zaretskii @ 2010-05-22 19:13 UTC (permalink / raw
To: forum2; +Cc: 6237-done
> Date: Sat, 22 May 2010 12:39:41 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 6237@debbugs.gnu.org
>
> > Date: Sat, 22 May 2010 11:38:35 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: 6237@debbugs.gnu.org
> >
> > > I know nothing about Windows internals, but it would seem that some
> > > downstream WinNT I/O routine can't handle 32MB or larger writes, so
> > > perhaps write-region could segment the write? The annotations hair
> > > makes it hard for me to see if this is easy.
> >
> > The breaking of writes into smaller chunks should be in Windows
> > specific code, i.e. in w32.c:sys_write. And that is very easy.
>
> If you can build Emacs yourself from sources, could you please try the
> following patch? It incorporates the info I found in the git mailing
> list thread pointed out by Andreas (thanks!).
Now installed in the repository (revno 100416).
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2010-05-22 19:13 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-05-21 2:57 bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS Jack Holloway
2010-05-22 1:51 ` Stefan Monnier
2010-05-22 2:12 ` Jack Holloway
2010-05-22 6:31 ` Eli Zaretskii
2010-05-22 7:49 ` Jack Holloway
2010-05-22 8:38 ` Eli Zaretskii
2010-05-22 9:39 ` Eli Zaretskii
2010-05-22 19:13 ` Eli Zaretskii
2010-05-22 8:08 ` Andreas Schwab
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.