[PATCH] util: increase the copy buffer size in shutil's copyfile

Augie Fackler raf at durin42.com
Fri May 6 16:50:30 EDT 2016


On Fri, May 06, 2016 at 01:36:41PM -0700, Tony Tung wrote:
> # HG changeset patch
> # User Tony Tung <ttung at fb.com>
> # Date 1462566734 25200
> #      Fri May 06 13:32:14 2016 -0700
> # Branch stable
> # Node ID edee9b1c3a2849fcb11900959ad785b202608c4f
> # Parent  97811ff7964710d32cae951df1da8019b46151a2
> util: increase the copy buffer size in shutil's copyfile
>
> The default copy buffer size in shutil is 16KB.  This is far too
> conservative for modern systems.  Use a 4MB buffer instead to copy files
> more efficiently.
>
> This manifests prominently with journaling large dirstates, which can be
> multi-second operations.
>
> diff --git a/mercurial/util.py b/mercurial/util.py
> --- a/mercurial/util.py
> +++ b/mercurial/util.py
> @@ -20,6 +20,7 @@
>  import collections
>  import datetime
>  import errno
> +import functools
>  import gc
>  import hashlib
>  import imp
> @@ -1010,6 +1011,23 @@
>
>      return check
>
> +# The default copy buffer in python's shutil is 16KB.  This is far too
> +# conservative in modern systems.  Copy more at a time so we don't incur so much
> +# system call overhead.
> +def wrap(container, funcname, newfunc):
> +    origfunc = getattr(container, funcname)
> +    wrap = functools.partial(newfunc, origfunc)
> +    functools.update_wrapper(wrap, origfunc)
> +    setattr(container, funcname, wrap)
> +
> +def copyfileobj_lgbuffer(orig, *args, **kwargs):
> +    if len(args) == 2 and 'length' not in kwargs:
> +        args = (args[0], args[1], (4 * 1024 * 1024))
> +
> +    orig(*args, **kwargs)
> +
> +wrap(shutil, 'copyfileobj', copyfileobj_lgbuffer)

Why are you monkeypatching shutil's implementation here? I'm happy to
have a helper method in util, but I'm deeply skeptical of
monkeypatching the stdlib, doubly so when it's a method we don't
currently call outside of a test.

> +
>  def copyfile(src, dest, hardlink=False, copystat=False):
>      '''copy a file, preserving mode and optionally other stat info like
>      atime/mtime'''
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel


More information about the Mercurial-devel mailing list