problem cloning large file (32MB) on Windows/Network Shares

[Adrian Buehlmann]

On 2011-01-13 04:52, rollin wrote:
> Hi,
> I'm new to hg and migrating micro$oft vss to hg; I'm encountering errors
> when cloning a repository to a network share where there is a file in
> the repository ~40MB (it failed using tortoisehg or hg command line). 
> In checking the hg mail list there are several different threads but one
> was similar and particularly helpful:
> 
> ****Re: TortoiseHG - Large File Commit (only 40Meg is size) <http://marc.info/?t=129319327100001&r=1&w=2>****
> 
> In the above thread, the original problem was reported on a Windows
> 32-bit XP computer; where my problem is on Windows 64-bit 2003 server.
> 
> In the last message of this thread from Matt Mackall, he suspected a
> problem with the I/O limit and provided a python script to test for an
> i/o limit error.  This was quite helpful.  Even though I'm not a python
> programmer, I ran the script (with a couple of mods to enable catching
> the error a bit better) on several systems.
> 
> Here's a summary of my testing using the python script or hg (they yield
> the same result):
>      o.s. platform                     cloning 33MB file     python
> script 33MB file
>                                                network share          to
> network share
>      Windows 2003 64-bit            Error                         Error
>      Windows 2008 64-bit              ok                              ok
>      Windows XP 32-bit                 ok                              ok
> 
> I checked the web and saw that there are some issues when doing
> unbuffered i/o to network shares on Windows 2003.  Evidently, the
> kernel's i/o manager has a hard limit of 32MB (which is mapped and
> locked to the user address space).  So attempting to write > 32MB will
> error out with insufficient resources.  Since I have c++ apps on many
> 2003 servers that write large files without problems I was curious what
> the difference is; I'm guessing the the python script (from above) and
> hg is attempting to write 32MB of unbuffered I/O.   Just a wag.  How
> does hg use python to do file copy?  Is it with buffered or unbuffered
> I/O?  Is it configurable?

Mercurial does most file access using util.posixfile, which for
Mercurial running on the Windows platform is the implementation in

  http://selenic.com/repo/hg/file/20a54bdf2328/mercurial/osutil.c#l405

which implements a Python class in C.

This in turn calls the windows API function CreateFile [1]

  http://selenic.com/repo/hg/file/20a54bdf2328/mercurial/osutil.c#l476

to open/create files, setting the dwFlagsAndAttributes parameter to
FILE_ATTRIBUTE_NORMAL, which means: default buffering ("If none of these
flags is specified, the system uses a default general-purpose caching
scheme" -- quoting [1]).

There are some file accesses that are done using Python's built-in open
function [2], which I haven't looked at how it is implemented (yet). But
writing and reading tracked files in the store (inside .hg/store) and
the working directory is done using util.posixfile.

(As a side note, see [3] for some interesting nasty differences between
these two file access methods. We're still fighting with these...)

[1] http://msdn.microsoft.com/en-us/library/aa363858(v=vs.85).aspx
[2] http://docs.python.org/library/functions.html#open
[3] http://mercurial.selenic.com/wiki/UnlinkingFilesOnWindows