problem cloning large file (32MB) on Windows/Network Shares

[rollin]

Hi Adrian,
After calling the win32 CreateFile, do you then call the win32 WriteFile?
I'll create a test program and try.

cheers, roy

On Thu, Jan 13, 2011 at 4:23 PM, Adrian Buehlmann <adrian at cadifra.com>wrote:

> On 2011-01-13 22:15, Matt Mackall wrote:
> > On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
> >> Hi,
> >> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
> >> errors when cloning a repository to a network share where there is a
> >> file in the repository ~40MB (it failed using tortoisehg or hg command
> >> line).  In checking the hg mail list there are several different
> >> threads but one was similar and particularly helpful:
> >> Re: TortoiseHG - Large File Commit (only 40Meg is size)
> >> In the above thread, the original problem was reported on a Windows
> >> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
> >>
> >> In the last message of this thread from Matt Mackall, he suspected a
> >> problem with the I/O limit and provided a python script to test for an
> >> i/o limit error.  This was quite helpful.  Even though I'm not a
> >> python programmer, I ran the script (with a couple of mods to enable
> >> catching the error a bit better) on several systems.
> >>
> >> Here's a summary of my testing using the python script or hg (they yield
> the same result):
> >>      o.s. platform                     cloning 33MB file     python
> script 33MB file
> >>                                                network share          to
> network share
> >>      Windows 2003 64-bit            Error                         Error
> >>      Windows 2008 64-bit              ok                              ok
> >>      Windows XP 32-bit                 ok
>  ok
> >
> > Thanks for narrowing it down by OS version. Did you test against local
> > disk?
> >
> >> I checked the web and saw that there are some issues when doing
> >> unbuffered i/o to network shares on Windows 2003.  Evidently, the
> >> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> >> locked to the user address space).  So attempting to write > 32MB will
> >> error out with insufficient resources.  Since I have c++ apps on many
> >> 2003 servers that write large files without problems I was curious
> >> what the difference is; I'm guessing the the python script (from
> >> above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
> >> wag.  How does hg use python to do file copy?  Is it with buffered or
> >> unbuffered I/O?  Is it configurable?
> >
> > It's not the size of the file that matters, it's the size of the _single
> > write()_ call to the Windows kernel. It's probably this:
> >
> > http://support.microsoft.com/kb/899149
> >
> > Essentially, a lazy kernel programmer at Microsoft left out a "for" loop
> > when working on the network share code.
> >
> > That article suggests that adding a "+" to the access mode in my Python
> > test script might work around the issue. Please test.
> >
> > (This is in no way related to so-called "unbuffered I/O" which is a very
> > weird feature that allows device drivers to directly DMA to/from
> > userspace buffers. This is similar to O_DIRECT in Linux.)
> >
>
> But Mercurial on Windows doesn't use fopen for most file accesses,
> bypassing the C run-time libraries with util.posixfile, as I've laid out
> in my other email in this thread.
>
> Does kb/899149 apply to the CreateFile API too?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial/attachments/20110113/fe1ad670/attachment.htm>