problem cloning large file (32MB) on Windows/Network Shares

[rollin]

When I modified my test program to use GENERIC_READ | GENERIC_WRITE, it
worked on both the local drive and the shared drive.

cheers, roy

On Thu, Jan 13, 2011 at 6:45 PM, Adrian Buehlmann <adrian at cadifra.com>wrote:

> On 2011-01-13 23:00, Matt Mackall wrote:
> > On Thu, 2011-01-13 at 22:23 +0100, Adrian Buehlmann wrote:
> >> On 2011-01-13 22:15, Matt Mackall wrote:
> >>> On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
> >>>> Hi,
> >>>> I'm new to hg and migrating micro$oft vss to hg; I'm encountering
> >>>> errors when cloning a repository to a network share where there is a
> >>>> file in the repository ~40MB (it failed using tortoisehg or hg command
> >>>> line).  In checking the hg mail list there are several different
> >>>> threads but one was similar and particularly helpful:
> >>>> Re: TortoiseHG - Large File Commit (only 40Meg is size)
> >>>> In the above thread, the original problem was reported on a Windows
> >>>> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
> >>>>
> >>>> In the last message of this thread from Matt Mackall, he suspected a
> >>>> problem with the I/O limit and provided a python script to test for an
> >>>> i/o limit error.  This was quite helpful.  Even though I'm not a
> >>>> python programmer, I ran the script (with a couple of mods to enable
> >>>> catching the error a bit better) on several systems.
> >>>>
> >>>> Here's a summary of my testing using the python script or hg (they
> yield the same result):
> >>>>      o.s. platform                     cloning 33MB file     python
> script 33MB file
> >>>>                                                network share
>  to network share
> >>>>      Windows 2003 64-bit            Error
> Error
> >>>>      Windows 2008 64-bit              ok
>  ok
> >>>>      Windows XP 32-bit                 ok
>  ok
> >>>
> >>> Thanks for narrowing it down by OS version. Did you test against local
> >>> disk?
> >>>
> >>>> I checked the web and saw that there are some issues when doing
> >>>> unbuffered i/o to network shares on Windows 2003.  Evidently, the
> >>>> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> >>>> locked to the user address space).  So attempting to write > 32MB will
> >>>> error out with insufficient resources.  Since I have c++ apps on many
> >>>> 2003 servers that write large files without problems I was curious
> >>>> what the difference is; I'm guessing the the python script (from
> >>>> above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
> >>>> wag.  How does hg use python to do file copy?  Is it with buffered or
> >>>> unbuffered I/O?  Is it configurable?
> >>>
> >>> It's not the size of the file that matters, it's the size of the
> _single
> >>> write()_ call to the Windows kernel. It's probably this:
> >>>
> >>> http://support.microsoft.com/kb/899149
> >>>
> >>> Essentially, a lazy kernel programmer at Microsoft left out a "for"
> loop
> >>> when working on the network share code.
> >>>
> >>> That article suggests that adding a "+" to the access mode in my Python
> >>> test script might work around the issue. Please test.
> >>>
> >>> (This is in no way related to so-called "unbuffered I/O" which is a
> very
> >>> weird feature that allows device drivers to directly DMA to/from
> >>> userspace buffers. This is similar to O_DIRECT in Linux.)
> >>>
> >>
> >> But Mercurial on Windows doesn't use fopen for most file accesses,
> >> bypassing the C run-time libraries with util.posixfile, as I've laid out
> >> in my other email in this thread.
> >
> > And "rollin" has already confirmed that it's not Mercurial-specific by
> > running my Python test script.
> >
> >> Does kb/899149 apply to the CreateFile API too?
> >
> > Who knows? The description of the problem is more or less useless as it
> > talks about N different irrelevant IDEs when the real issue "occurs
> > because of a limitation of the operating system", and not the IDEs or
> > even their runtime libraries.
> >
> > But it's hard to imagine how it wouldn't affect CreateFile users as
> > well. It's not as if there's a separate implementation of all the I/O
> > functions for shares vs local drives. No, they instead will all funnel
> > through a generic kernel-internal API that translates the userspace
> > requests to filesystem-specific code. If a bug is correlated with a
> > filesystem type, it's almost certainly due to a bug in the filesystem
> > driver and not at the higher generic levels.
> >
>
> On a possibly unrelated note, I found this amazing snippet of text in
> the description of CreateFile [1]
>
> '''
> When an application creates a file across a network, it is better to use
> GENERIC_READ | GENERIC_WRITE for dwDesiredAccess than to use
> GENERIC_WRITE alone. The resulting code is faster, because the
> redirector can use the cache manager and send fewer SMBs with more data.
> This combination also avoids an issue where writing to a file across a
> network can occasionally return ERROR_ACCESS_DENIED.
> '''
>
> I really have no idea what exactly that "occasional ERROR_ACCESS_DENIED
> across a network" could cause :(
>
> 'w+' for mode leads to (GENERIC_READ | GENERIC_WRITE) for
> dwDesiredAccess in osutil.c
>
> [1] http://msdn.microsoft.com/en-us/library/aa363858(v=vs.85).aspx<http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial/attachments/20110113/fdd55542/attachment.htm>