problem cloning large file (32MB) on Windows/Network Shares

Matt Mackall mpm at selenic.com
Thu Jan 13 15:15:06 CST 2011


On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
> Hi,
> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
> errors when cloning a repository to a network share where there is a
> file in the repository ~40MB (it failed using tortoisehg or hg command
> line).  In checking the hg mail list there are several different
> threads but one was similar and particularly helpful: 
> Re: TortoiseHG - Large File Commit (only 40Meg is size)
> In the above thread, the original problem was reported on a Windows
> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
> 
> In the last message of this thread from Matt Mackall, he suspected a
> problem with the I/O limit and provided a python script to test for an
> i/o limit error.  This was quite helpful.  Even though I'm not a
> python programmer, I ran the script (with a couple of mods to enable
> catching the error a bit better) on several systems. 
> 
> Here's a summary of my testing using the python script or hg (they yield the same result):
>      o.s. platform                     cloning 33MB file     python script 33MB file
>                                                network share          to network share
>      Windows 2003 64-bit            Error                         Error
>      Windows 2008 64-bit              ok                              ok
>      Windows XP 32-bit                 ok                              ok

Thanks for narrowing it down by OS version. Did you test against local
disk?

> I checked the web and saw that there are some issues when doing
> unbuffered i/o to network shares on Windows 2003.  Evidently, the
> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> locked to the user address space).  So attempting to write > 32MB will
> error out with insufficient resources.  Since I have c++ apps on many
> 2003 servers that write large files without problems I was curious
> what the difference is; I'm guessing the the python script (from
> above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
> wag.  How does hg use python to do file copy?  Is it with buffered or
> unbuffered I/O?  Is it configurable?

It's not the size of the file that matters, it's the size of the _single
write()_ call to the Windows kernel. It's probably this:

http://support.microsoft.com/kb/899149

Essentially, a lazy kernel programmer at Microsoft left out a "for" loop
when working on the network share code.

That article suggests that adding a "+" to the access mode in my Python
test script might work around the issue. Please test.

(This is in no way related to so-called "unbuffered I/O" which is a very
weird feature that allows device drivers to directly DMA to/from
userspace buffers. This is similar to O_DIRECT in Linux.)

-- 
Mathematics is the supreme nostalgia of our time.




More information about the Mercurial mailing list