problem cloning large file (32MB) on Windows/Network Shares

[rollin]

> Does kb/899149 apply to the CreateFile API too? (from Adrian)

There are several web sites stating the following:
ERROR_NO_SYSTEM_RESOURCES(1450) when using data buffers larger than 32MB or
64MB. The reason for this error is that WriteFile is used with a
no-buffering flag, and the maximum buffer size (for mapping virtual
addresses to physical addresses with a memory descriptor list) has been
exceeded by the file size. On x86 (32-bit) or IA64 (Itanium 64-bit) systems,
the maximum buffer size is just under 64MB. For x64 systems, the maximum
buffer size is just under 32MB. The maximum unbuffered read and write size
limits are imposed by the design of the IO manager inside the Windows
executive. The same problem could occur on x86 or IA64 machines, but not
until the file size is exceeding 64MB.
(This one came from http://support.microsoft.com/kb/2484430; but it's not
exactly the same thing because it's really describiing an issue with mt.exe
creating a 0-byte dll when using CreateFile/WriteFile.)



On Thu, Jan 13, 2011 at 5:00 PM, Matt Mackall <mpm at selenic.com> wrote:

> On Thu, 2011-01-13 at 22:23 +0100, Adrian Buehlmann wrote:
> > On 2011-01-13 22:15, Matt Mackall wrote:
> > > On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
> > >> Hi,
> > >> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
> > >> errors when cloning a repository to a network share where there is a
> > >> file in the repository ~40MB (it failed using tortoisehg or hg command
> > >> line).  In checking the hg mail list there are several different
> > >> threads but one was similar and particularly helpful:
> > >> Re: TortoiseHG - Large File Commit (only 40Meg is size)
> > >> In the above thread, the original problem was reported on a Windows
> > >> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
> > >>
> > >> In the last message of this thread from Matt Mackall, he suspected a
> > >> problem with the I/O limit and provided a python script to test for an
> > >> i/o limit error.  This was quite helpful.  Even though I'm not a
> > >> python programmer, I ran the script (with a couple of mods to enable
> > >> catching the error a bit better) on several systems.
> > >>
> > >> Here's a summary of my testing using the python script or hg (they
> yield the same result):
> > >>      o.s. platform                     cloning 33MB file     python
> script 33MB file
> > >>                                                network share
>  to network share
> > >>      Windows 2003 64-bit            Error
> Error
> > >>      Windows 2008 64-bit              ok
>  ok
> > >>      Windows XP 32-bit                 ok
>  ok
> > >
> > > Thanks for narrowing it down by OS version. Did you test against local
> > > disk?
> > >
> > >> I checked the web and saw that there are some issues when doing
> > >> unbuffered i/o to network shares on Windows 2003.  Evidently, the
> > >> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> > >> locked to the user address space).  So attempting to write > 32MB will
> > >> error out with insufficient resources.  Since I have c++ apps on many
> > >> 2003 servers that write large files without problems I was curious
> > >> what the difference is; I'm guessing the the python script (from
> > >> above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
> > >> wag.  How does hg use python to do file copy?  Is it with buffered or
> > >> unbuffered I/O?  Is it configurable?
> > >
> > > It's not the size of the file that matters, it's the size of the
> _single
> > > write()_ call to the Windows kernel. It's probably this:
> > >
> > > http://support.microsoft.com/kb/899149
> > >
> > > Essentially, a lazy kernel programmer at Microsoft left out a "for"
> loop
> > > when working on the network share code.
> > >
> > > That article suggests that adding a "+" to the access mode in my Python
> > > test script might work around the issue. Please test.
> > >
> > > (This is in no way related to so-called "unbuffered I/O" which is a
> very
> > > weird feature that allows device drivers to directly DMA to/from
> > > userspace buffers. This is similar to O_DIRECT in Linux.)
> > >
> >
> > But Mercurial on Windows doesn't use fopen for most file accesses,
> > bypassing the C run-time libraries with util.posixfile, as I've laid out
> > in my other email in this thread.
>
> And "rollin" has already confirmed that it's not Mercurial-specific by
> running my Python test script.
>
> > Does kb/899149 apply to the CreateFile API too?
>
> Who knows? The description of the problem is more or less useless as it
> talks about N different irrelevant IDEs when the real issue "occurs
> because of a limitation of the operating system", and not the IDEs or
> even their runtime libraries.
>
> But it's hard to imagine how it wouldn't affect CreateFile users as
> well. It's not as if there's a separate implementation of all the I/O
> functions for shares vs local drives. No, they instead will all funnel
> through a generic kernel-internal API that translates the userspace
> requests to filesystem-specific code. If a bug is correlated with a
> filesystem type, it's almost certainly due to a bug in the filesystem
> driver and not at the higher generic levels.
>
> --
> Mathematics is the supreme nostalgia of our time.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial/attachments/20110113/958374bb/attachment.htm>