problem cloning large file (32MB) on Windows/Network Shares
Matt Mackall
mpm at selenic.com
Thu Jan 13 15:15:06 CST 2011
On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
> Hi,
> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
> errors when cloning a repository to a network share where there is a
> file in the repository ~40MB (it failed using tortoisehg or hg command
> line). In checking the hg mail list there are several different
> threads but one was similar and particularly helpful:
> Re: TortoiseHG - Large File Commit (only 40Meg is size)
> In the above thread, the original problem was reported on a Windows
> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
>
> In the last message of this thread from Matt Mackall, he suspected a
> problem with the I/O limit and provided a python script to test for an
> i/o limit error. This was quite helpful. Even though I'm not a
> python programmer, I ran the script (with a couple of mods to enable
> catching the error a bit better) on several systems.
>
> Here's a summary of my testing using the python script or hg (they yield the same result):
> o.s. platform cloning 33MB file python script 33MB file
> network share to network share
> Windows 2003 64-bit Error Error
> Windows 2008 64-bit ok ok
> Windows XP 32-bit ok ok
Thanks for narrowing it down by OS version. Did you test against local
disk?
> I checked the web and saw that there are some issues when doing
> unbuffered i/o to network shares on Windows 2003. Evidently, the
> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> locked to the user address space). So attempting to write > 32MB will
> error out with insufficient resources. Since I have c++ apps on many
> 2003 servers that write large files without problems I was curious
> what the difference is; I'm guessing the the python script (from
> above) and hg is attempting to write 32MB of unbuffered I/O. Just a
> wag. How does hg use python to do file copy? Is it with buffered or
> unbuffered I/O? Is it configurable?
It's not the size of the file that matters, it's the size of the _single
write()_ call to the Windows kernel. It's probably this:
http://support.microsoft.com/kb/899149
Essentially, a lazy kernel programmer at Microsoft left out a "for" loop
when working on the network share code.
That article suggests that adding a "+" to the access mode in my Python
test script might work around the issue. Please test.
(This is in no way related to so-called "unbuffered I/O" which is a very
weird feature that allows device drivers to directly DMA to/from
userspace buffers. This is similar to O_DIRECT in Linux.)
--
Mathematics is the supreme nostalgia of our time.
More information about the Mercurial
mailing list