problem cloning large file (32MB) on Windows/Network Shares

[rollin]

Hi Matt,
>> That article suggests that adding a "+" to the access mode in my Python
test script might work around the issue. Please test. (from Matt)

> I'll test but I'm a bit skeptical :(. (from roy)
Despite being skeptical, adding the "+" to the access mode did fix the
problem for your Python script.  Being skeptical, I tested again without the
"+" and it failed.  So, using python to open and write a large dataset to a
file one should be aware of this quirk.

Adrian says that Mercurial is using win32 to create the file (and I assume
to also write the file); so I would think there's an issue with using the
createfile and writefile (if that's what hg is using) on windows 2003 and
network shares.

cheers, roy

On Thu, Jan 13, 2011 at 4:34 PM, rollin <rollinforall at gmail.com> wrote:

> Hi Matt,
>
> >Thanks for narrowing it down by OS version. Did you test against local
> disk?
>
> No errors were encountered using local disks on any of the o.s. tested.
>
>
> > That article suggests that adding a "+" to the access mode in my Python
> test script might work around the issue. Please test.
>
> I'll test but I'm a bit skeptical :(.
>
> cheers, roy
>
>
>
> On Thu, Jan 13, 2011 at 4:15 PM, Matt Mackall <mpm at selenic.com> wrote:
>
>> On Wed, 2011-01-12 at 22:52 -0500, rollin wrote:
>> > Hi,
>> > I'm new to hg and migrating micro$oft vss to hg; I'm encountering
>> > errors when cloning a repository to a network share where there is a
>> > file in the repository ~40MB (it failed using tortoisehg or hg command
>> > line).  In checking the hg mail list there are several different
>> > threads but one was similar and particularly helpful:
>> > Re: TortoiseHG - Large File Commit (only 40Meg is size)
>> > In the above thread, the original problem was reported on a Windows
>> > 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
>> >
>> > In the last message of this thread from Matt Mackall, he suspected a
>> > problem with the I/O limit and provided a python script to test for an
>> > i/o limit error.  This was quite helpful.  Even though I'm not a
>> > python programmer, I ran the script (with a couple of mods to enable
>> > catching the error a bit better) on several systems.
>> >
>> > Here's a summary of my testing using the python script or hg (they yield
>> the same result):
>> >      o.s. platform                     cloning 33MB file     python
>> script 33MB file
>> >                                                network share          to
>> network share
>> >      Windows 2003 64-bit            Error                         Error
>> >      Windows 2008 64-bit              ok                              ok
>> >      Windows XP 32-bit                 ok
>>  ok
>>
>> Thanks for narrowing it down by OS version. Did you test against local
>> disk?
>>
>> > I checked the web and saw that there are some issues when doing
>> > unbuffered i/o to network shares on Windows 2003.  Evidently, the
>> > kernel's i/o manager has a hard limit of 32MB (which is mapped and
>> > locked to the user address space).  So attempting to write > 32MB will
>> > error out with insufficient resources.  Since I have c++ apps on many
>> > 2003 servers that write large files without problems I was curious
>> > what the difference is; I'm guessing the the python script (from
>> > above) and hg is attempting to write 32MB of unbuffered I/O.   Just a
>> > wag.  How does hg use python to do file copy?  Is it with buffered or
>> > unbuffered I/O?  Is it configurable?
>>
>> It's not the size of the file that matters, it's the size of the _single
>> write()_ call to the Windows kernel. It's probably this:
>>
>> http://support.microsoft.com/kb/899149
>>
>> Essentially, a lazy kernel programmer at Microsoft left out a "for" loop
>> when working on the network share code.
>>
>> That article suggests that adding a "+" to the access mode in my Python
>> test script might work around the issue. Please test.
>>
>> (This is in no way related to so-called "unbuffered I/O" which is a very
>> weird feature that allows device drivers to directly DMA to/from
>> userspace buffers. This is similar to O_DIRECT in Linux.)
>>
>> --
>> Mathematics is the supreme nostalgia of our time.
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial/attachments/20110113/128c110b/attachment.htm>