problem cloning large file (32MB) on Windows/Network Shares

[Adrian Buehlmann]

On 2011-01-13 22:15, Matt Mackall wrote:
> On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
>> Hi,
>> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
>> errors when cloning a repository to a network share where there is a
>> file in the repository ~40MB (it failed using tortoisehg or hg command
>> line).  In checking the hg mail list there are several different
>> threads but one was similar and particularly helpful: 
>> Re: TortoiseHG - Large File Commit (only 40Meg is size)
>> In the above thread, the original problem was reported on a Windows
>> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
>>
>> In the last message of this thread from Matt Mackall, he suspected a
>> problem with the I/O limit and provided a python script to test for an
>> i/o limit error.  This was quite helpful.  Even though I'm not a
>> python programmer, I ran the script (with a couple of mods to enable
>> catching the error a bit better) on several systems. 
>>
>> Here's a summary of my testing using the python script or hg (they yield the same result):
>>      o.s. platform                     cloning 33MB file     python script 33MB file
>>                                                network share          to network share
>>      Windows 2003 64-bit            Error                         Error
>>      Windows 2008 64-bit              ok                              ok
>>      Windows XP 32-bit                 ok                              ok
> 
> Thanks for narrowing it down by OS version. Did you test against local
> disk?
> 
>> I checked the web and saw that there are some issues when doing
>> unbuffered i/o to network shares on Windows 2003.  Evidently, the
>> kernel's i/o manager has a hard limit of 32MB (which is mapped and
>> locked to the user address space).  So attempting to write > 32MB will
>> error out with insufficient resources.  Since I have c++ apps on many
>> 2003 servers that write large files without problems I was curious
>> what the difference is; I'm guessing the the python script (from
>> above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
>> wag.  How does hg use python to do file copy?  Is it with buffered or
>> unbuffered I/O?  Is it configurable?
> 
> It's not the size of the file that matters, it's the size of the _single
> write()_ call to the Windows kernel. It's probably this:
> 
> http://support.microsoft.com/kb/899149
> 
> Essentially, a lazy kernel programmer at Microsoft left out a "for" loop
> when working on the network share code.
> 
> That article suggests that adding a "+" to the access mode in my Python
> test script might work around the issue. Please test.
> 
> (This is in no way related to so-called "unbuffered I/O" which is a very
> weird feature that allows device drivers to directly DMA to/from
> userspace buffers. This is similar to O_DIRECT in Linux.)
> 

But Mercurial on Windows doesn't use fopen for most file accesses,
bypassing the C run-time libraries with util.posixfile, as I've laid out
in my other email in this thread.

Does kb/899149 apply to the CreateFile API too?