[PATCH 5 of 5] exchange: use file closer when applying stream clones

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc <gregory.szorc at gmail.com>
# Date 1443553088 25200
#      Tue Sep 29 11:58:08 2015 -0700
# Node ID cea1c79d6ab088caedb93b6576f9293f307355cc
# Parent  a8d8f0d70593aaa6fb2b86dc77d29c021d83277d
exchange: use file closer when applying stream clones

Stream clones essentially stream raw files to disk. They have the
potential to write thousands of files. Have them use the file closer
so they close files as efficiently as possible.

On my Windows machine, this patch has the following impacting on stream
clone performance for the mozilla-central repository (tested with the
"bundleclone" extension so there is no variance due to networks):

Before: ~435s (3.56 MB/s)
After:  ~339s (4.55 MB/s)

For comparison, I created a .tar (no compression) archive of the .hg
directory so I could compare performance against other tools (stream
bundles are essentially tar archives - file contents are copied with
no stream modification). The popular 7-zip archiving program took 414s
to unpack this archive. 7-zip is a compiled/native program.
Considering the marginal time difference between 7-zip and Mercurial
before this patch, I reckon 7-zip is using a single thread for I/O.
I won't report the time of the `tar` program on Windows because it is
abysmal. Mercurial was already significantly faster than it. Anyway,
after this patch we can say that Mercurial is faster than 7-zip on
Windows! Honestly, that doesn't say much: my SSD can easily copy
files at 100-200 MB/s on Windows.

As with the previous patch, I insrumented time spent waiting on the
background file closing thread to dip below its high water mark. It
reported a staggering 238s! Worthwhile followup work would be to
investigate whether multiple file close threads decreases wall time
further. Although I have no idea how multiple I/O threads will scale on
Windows. Another option is to "leak" file descriptors. When you stick to
Win32 APIs like CreateFile() and CloseHandle() (you don't involve file
descriptors), you can have 2^24 open handles in a Windows process. These
will automatically close on process exit. Python, however, operates on
file descriptors. While Mercurial calls CreateFile(), it passes the
returned handle into _open_osfhandle() to obtain a file descriptor and
gives that to Python to turn into a Python file object. File handles are
subject to a default limit of 512 (which can be raised to 2048). With a
lot of work (you'd have to reimplement the Python file object API), you
could maintain pure Win32 function calls everywhere, leak handles, and
then maybe, just maybe process close will be more efficient than
CloseFile() on thousands of file handles. (This would likely require
spawning a child process to hold the handles so command servers wouldn't
leak forever.) It's an interesting (although large effort idea).

Windows I/O and Python on Windows is very capable at writing at very
fast speeds: manifest writing finished in just a second or two - and it
is a ~455M file! There has been some chatter around building a repo
class that operates on "indexed stream bundles" - reading revlogs from a
single "packed" file and writing out individual revlogs on disk only as
new data is written. If this existed, I suspect stream bundle
application would be measured in seconds, not minutes. This would also
make excellent follow-up work. And it would likely benefit Linux and OS
X as well.

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -1533,11 +1533,13 @@ def consumestreamclone(repo, fp):
         handled_bytes = 0
         repo.ui.progress(_('clone'), 0, total=total_bytes)
         start = time.time()
 
+        closer = util.filecloser(repo.ui)
         tr = repo.transaction(_('clone'))
         try:
             for i in xrange(total_files):
+                closer.verifystate()
                 # XXX doesn't support '\n' or '\r' in filenames
                 l = fp.readline()
                 try:
                     name, size = l.split('\0', 1)
@@ -1554,12 +1556,13 @@ def consumestreamclone(repo, fp):
                     handled_bytes += len(chunk)
                     repo.ui.progress(_('clone'), handled_bytes,
                                      total=total_bytes)
                     ofp.write(chunk)
-                ofp.close()
+                closer.close(ofp)
             tr.close()
         finally:
             tr.release()
+            closer.cleanup()
 
         # Writing straight to files circumvented the inmemory caches
         repo.invalidate()