* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS @ 2010-05-21 2:57 Jack Holloway 2010-05-22 1:51 ` Stefan Monnier 0 siblings, 1 reply; 9+ messages in thread From: Jack Holloway @ 2010-05-21 2:57 UTC (permalink / raw) To: 6237 [-- Attachment #1: Type: text/html, Size: 5694 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-21 2:57 bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS Jack Holloway @ 2010-05-22 1:51 ` Stefan Monnier 2010-05-22 2:12 ` Jack Holloway 0 siblings, 1 reply; 9+ messages in thread From: Stefan Monnier @ 2010-05-22 1:51 UTC (permalink / raw) To: Jack Holloway; +Cc: 6237 Thank you for your report. But I find it difficult to read it because it is in HTML format. Can you please reply with a plain text version? Thank you, Stefan ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 1:51 ` Stefan Monnier @ 2010-05-22 2:12 ` Jack Holloway 2010-05-22 6:31 ` Eli Zaretskii 2010-05-22 8:08 ` Andreas Schwab 0 siblings, 2 replies; 9+ messages in thread From: Jack Holloway @ 2010-05-22 2:12 UTC (permalink / raw) To: Stefan Monnier; +Cc: 6237 Sorry for the gaffe -- this should be plain text. Let me know if it's not. --------------------- Saving a 32MB file to a CIFS path fails. Examining Procmon events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile with length 32MB. Perhaps write-region should break down the write into a series of blksize writes? Here's the relevant backtrace: Debugger entered--Lisp error: (error "IO error writing z:/Temp/test/tmp2092CxD: Invalid argument") write-region(nil nil "z:/Temp/test/tmp2092CxD" nil "z:/Temp/test/temp-write-once-032768" "z:/Temp/test/temp-write-once-032768" excl) byte-code("..." [dir tempname realname buffer-file-truename make-temp-name expand-file-name "tmp" write-region nil excl] 8) basic-save-buffer-2() basic-save-buffer-1() basic-save-buffer() save-buffer() write-file("z:/Temp/test/temp-write-once-032768") This was running on Windows XP64 Professional SP2. The drive "z:" is mapped to a NAS running Samba. This error doesn't occur when writing to the host's internal NTFS, it does occur with other CIFS targets. copy-file doesn't have this problem. In GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790) of 2010-05-08 on G41R2F1 Windowing system distributor `Microsoft Corp.', version 5.2.3790 configured using `configure --with-gcc (3.4) --no-opt --cflags -Ic:/xpm/include' On 5/21/2010 6:51 PM, Stefan Monnier wrote: > Thank you for your report. But I find it difficult to read it because > it is in HTML format. Can you please reply with a plain text version? > Thank you, > > > Stefan > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 2:12 ` Jack Holloway @ 2010-05-22 6:31 ` Eli Zaretskii 2010-05-22 7:49 ` Jack Holloway 2010-05-22 8:08 ` Andreas Schwab 1 sibling, 1 reply; 9+ messages in thread From: Eli Zaretskii @ 2010-05-22 6:31 UTC (permalink / raw) To: Jack Holloway; +Cc: 6237 > Date: Fri, 21 May 2010 19:12:41 -0700 > From: Jack Holloway <forum2@sprucehead.com> > Cc: 6237@debbugs.gnu.org > > Saving a 32MB file to a CIFS path fails. Examining Procmon > events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile > with length 32MB. > > Perhaps write-region should break down the write into a series > of blksize writes? Can you see which size of a single write succeeds with that volume? Also, could you see how many bytes did it succeed to write before the failure? > copy-file doesn't have this problem. The MS-Windows version of copy-file doesn't use WriteFile. It uses CopyFile. Anyway, how did you use copy-file? Was the source file on the same CIFS volume, or was it on a local disk? ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 6:31 ` Eli Zaretskii @ 2010-05-22 7:49 ` Jack Holloway 2010-05-22 8:38 ` Eli Zaretskii 0 siblings, 1 reply; 9+ messages in thread From: Jack Holloway @ 2010-05-22 7:49 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 6237 On 5/21/2010 11:31 PM, Eli Zaretskii wrote: >> Date: Fri, 21 May 2010 19:12:41 -0700 >> From: Jack Holloway<forum2@sprucehead.com> >> Cc: 6237@debbugs.gnu.org >> >> Saving a 32MB file to a CIFS path fails. Examining Procmon >> events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile >> with length 32MB. >> >> Perhaps write-region should break down the write into a series >> of blksize writes? >> > Can you see which size of a single write succeeds with that volume? > > Also, could you see how many bytes did it succeed to write before the > failure? > I tried successively larger buffers starting at 32KB, viz. Written 32K Written 48K Written 64K Written 96K Written 128K Written 192K Written 256K Written 384K Written 512K Written 768K Written 1024K Written 1536K Written 2048K Written 3072K Written 4096K Written 6144K Written 8192K Written 12288K Written 16384K Written 24576K !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-032768: Invalid argument) !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-049152: Invalid argument) !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-065536: Invalid argument) !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-098304: Invalid argument) !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-131072: Invalid argument) !!! Unable to append - (error IO error writing z:/Temp/test/temp-write-once-196608: Invalid argument) The would-be 32MB file exists, with zero length. >> copy-file doesn't have this problem. >> > The MS-Windows version of copy-file doesn't use WriteFile. It uses > CopyFile. > > Anyway, how did you use copy-file? Was the source file on the same > CIFS volume, or was it on a local disk? > > I've tried all 4 combinations of {internal, CIFS}*{src,dest} with transfer sizes up to 196MB without a problem, I was referring to DEFUN ("copy-file",...) in fileio.c which writes in 16K blocks, ... while ((n = emacs_read (ifd, buf, sizeof buf)) > 0) if (emacs_write (ofd, buf, n) != n) report_file_error ("I/O error", Fcons (newname, Qnil)); ... as opposed to DEFUN ("write-region",...) which, in this case, writes the entire buffer ... else if (XINT (start) != XINT (end)) { failure = 0 > a_write (desc, Qnil, XINT (start), XINT (end) - XINT (start), &annotations, &coding); save_errno = errno; } ... I know nothing about Windows internals, but it would seem that some downstream WinNT I/O routine can't handle 32MB or larger writes, so perhaps write-region could segment the write? The annotations hair makes it hard for me to see if this is easy. Jack ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 7:49 ` Jack Holloway @ 2010-05-22 8:38 ` Eli Zaretskii 2010-05-22 9:39 ` Eli Zaretskii 0 siblings, 1 reply; 9+ messages in thread From: Eli Zaretskii @ 2010-05-22 8:38 UTC (permalink / raw) To: Jack Holloway; +Cc: 6237 > Date: Sat, 22 May 2010 00:49:56 -0700 > From: Jack Holloway <forum2@sprucehead.com> > CC: 6237@debbugs.gnu.org > > Written 16384K > Written 24576K > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-032768: Invalid argument) > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-049152: Invalid argument) > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-065536: Invalid argument) > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-098304: Invalid argument) > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-131072: Invalid argument) > !!! Unable to append - (error IO error writing > z:/Temp/test/temp-write-once-196608: Invalid argument) > > The would-be 32MB file exists, with zero length. So 24MB succeeds, but 32MB and all sizes above it fail? But why does the error message say "Unable to append"? Why "append"? > I was referring to DEFUN ("copy-file",...) in fileio.c which writes in > 16K blocks, > ... > while ((n = emacs_read (ifd, buf, sizeof buf)) > 0) > if (emacs_write (ofd, buf, n) != n) > report_file_error ("I/O error", Fcons (newname, Qnil)); This is not the code that runs on Windows. That one is a little ways earlier in the body of copy-file: #ifdef WINDOWSNT if (!CopyFile (SDATA (encoded_file), SDATA (encoded_newname), FALSE)) report_file_error ("Copying file", Fcons (file, Fcons (newname, Qnil))); /* CopyFile retains the timestamp by default. */ else if (NILP (keep_time)) { EMACS_TIME now; DWORD attributes; char * filename; EMACS_GET_TIME (now); filename = SDATA (encoded_newname); /* Ensure file is writable while its modified time is set. */ attributes = GetFileAttributes (filename); SetFileAttributes (filename, attributes & ~FILE_ATTRIBUTE_READONLY); if (set_file_times (filename, now, now)) { /* Restore original attributes. */ SetFileAttributes (filename, attributes); xsignal2 (Qfile_date_error, build_string ("Cannot set file date"), newname); } /* Restore original attributes. */ SetFileAttributes (filename, attributes); } #else /* not WINDOWSNT */ > I know nothing about Windows internals, but it would seem that some > downstream WinNT I/O routine can't handle 32MB or larger writes, so > perhaps write-region could segment the write? The annotations hair > makes it hard for me to see if this is easy. The breaking of writes into smaller chunks should be in Windows specific code, i.e. in w32.c:sys_write. And that is very easy. ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 8:38 ` Eli Zaretskii @ 2010-05-22 9:39 ` Eli Zaretskii 2010-05-22 19:13 ` Eli Zaretskii 0 siblings, 1 reply; 9+ messages in thread From: Eli Zaretskii @ 2010-05-22 9:39 UTC (permalink / raw) To: forum2; +Cc: 6237 > Date: Sat, 22 May 2010 11:38:35 +0300 > From: Eli Zaretskii <eliz@gnu.org> > Cc: 6237@debbugs.gnu.org > > > I know nothing about Windows internals, but it would seem that some > > downstream WinNT I/O routine can't handle 32MB or larger writes, so > > perhaps write-region could segment the write? The annotations hair > > makes it hard for me to see if this is easy. > > The breaking of writes into smaller chunks should be in Windows > specific code, i.e. in w32.c:sys_write. And that is very easy. If you can build Emacs yourself from sources, could you please try the following patch? It incorporates the info I found in the git mailing list thread pointed out by Andreas (thanks!). === modified file 'src/w32.c' --- src/w32.c 2010-05-01 18:28:00 +0000 +++ src/w32.c 2010-05-22 09:33:53 +0000 @@ -5700,7 +5700,34 @@ sys_write (int fd, const void * buffer, } else #endif - nchars = _write (fd, buffer, count); + { + /* Some networked filesystems don't like too large writes, so + break them into smaller chunks. See the Comments section of + the MSDN documentation of WriteFile for details behind the + choice of the value of CHUNK below. See also the thread + http://thread.gmane.org/gmane.comp.version-control.git/145294 + in the git mailing list. */ + const unsigned char *p = buffer; + const unsigned chunk = 30 * 1024 * 1024; + + nchars = 0; + while (count > 0) + { + unsigned this_chunk = count < chunk ? count : chunk; + int n = _write (fd, p, this_chunk); + + nchars += n; + if (n < 0) + { + nchars = n; + break; + } + else if (n < this_chunk) + break; + count -= n; + p += n; + } + } return nchars; } ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 9:39 ` Eli Zaretskii @ 2010-05-22 19:13 ` Eli Zaretskii 0 siblings, 0 replies; 9+ messages in thread From: Eli Zaretskii @ 2010-05-22 19:13 UTC (permalink / raw) To: forum2; +Cc: 6237-done > Date: Sat, 22 May 2010 12:39:41 +0300 > From: Eli Zaretskii <eliz@gnu.org> > Cc: 6237@debbugs.gnu.org > > > Date: Sat, 22 May 2010 11:38:35 +0300 > > From: Eli Zaretskii <eliz@gnu.org> > > Cc: 6237@debbugs.gnu.org > > > > > I know nothing about Windows internals, but it would seem that some > > > downstream WinNT I/O routine can't handle 32MB or larger writes, so > > > perhaps write-region could segment the write? The annotations hair > > > makes it hard for me to see if this is easy. > > > > The breaking of writes into smaller chunks should be in Windows > > specific code, i.e. in w32.c:sys_write. And that is very easy. > > If you can build Emacs yourself from sources, could you please try the > following patch? It incorporates the info I found in the git mailing > list thread pointed out by Andreas (thanks!). Now installed in the repository (revno 100416). ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS 2010-05-22 2:12 ` Jack Holloway 2010-05-22 6:31 ` Eli Zaretskii @ 2010-05-22 8:08 ` Andreas Schwab 1 sibling, 0 replies; 9+ messages in thread From: Andreas Schwab @ 2010-05-22 8:08 UTC (permalink / raw) To: Jack Holloway; +Cc: 6237 Jack Holloway <forum2@sprucehead.com> writes: > Sorry for the gaffe -- this should be plain text. Let me know if it's not. > --------------------- > Saving a 32MB file to a CIFS path fails. Examining Procmon > events shows a return status "INSUFFICIENT RESOURCES" on a WriteFile > with length 32MB. > > Perhaps write-region should break down the write into a series > of blksize writes? See also this thread on the git mailing list: <http://thread.gmane.org/gmane.comp.version-control.git/145294>. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2010-05-22 19:13 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-05-21 2:57 bug#6237: 23.2; INSUFFICIENT RESOURCES writing file via CIFS Jack Holloway 2010-05-22 1:51 ` Stefan Monnier 2010-05-22 2:12 ` Jack Holloway 2010-05-22 6:31 ` Eli Zaretskii 2010-05-22 7:49 ` Jack Holloway 2010-05-22 8:38 ` Eli Zaretskii 2010-05-22 9:39 ` Eli Zaretskii 2010-05-22 19:13 ` Eli Zaretskii 2010-05-22 8:08 ` Andreas Schwab
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).