[PATCH] worker: use multiprocessing to find cpu count

[Gregory Szorc]

On Mon, Aug 24, 2015 at 7:02 AM, Augie Fackler <raf at durin42.com> wrote:

> On Sat, Aug 22, 2015 at 09:52:07AM -0700, Gregory Szorc wrote:
> > # HG changeset patch
> > # User Gregory Szorc <gregory.szorc at gmail.com>
> > # Date 1432584638 25200
> > #      Mon May 25 13:10:38 2015 -0700
> > # Node ID a3f28ff761510272136887b739fb3a81d0d5a9af
> > # Parent  d9d3d49c4cf77049d12920980e91bf8e4a4ecda2
> > worker: use multiprocessing to find cpu count
>
> Nice. Queued.
>
> I wonder if we should explore using multiprocessing for some worker
> pool type stuff on Windows, since it has vaguely working fork() emulation.
>
>
My experience is multiprocessing is generally a PITA, especially on
Windows. There are also problems with multiprocessing not working on BSDs.
Although I think the latter has been fixed with newer versions of Python
2.7. Not sure about 2.6.

I find the concurrent.futures interface more tolerable. Although it may
suffer from many of the same limitations.



> >
> > The multiprocessing package was added in Python 2.6.
> > The implementation of worker.countcpus() is very similar to
> > multiprocessing.cpu_count(). Ditch our one-off code.
> >
> > multiprocessing does result in a number of imports. However,
> > the lazy importer ensures that we don't import anything until
> > cpu_count() is called. Furthermore, if we are doing something
> > with multiple cores, chances are the time of that operation
> > will dwarf the import time, so module bloat isn't a concern
> > here.
> >
> > diff --git a/mercurial/worker.py b/mercurial/worker.py
> > --- a/mercurial/worker.py
> > +++ b/mercurial/worker.py
> > @@ -7,8 +7,9 @@
> >
> >  from __future__ import absolute_import
> >
> >  import errno
> > +import multiprocessing
> >  import os
> >  import signal
> >  import sys
> >  import threading
> > @@ -17,26 +18,12 @@ from .i18n import _
> >  from . import util
> >
> >  def countcpus():
> >      '''try to count the number of CPUs on the system'''
> > -
> > -    # posix
> >      try:
> > -        n = int(os.sysconf('SC_NPROCESSORS_ONLN'))
> > -        if n > 0:
> > -            return n
> > -    except (AttributeError, ValueError):
> > -        pass
> > -
> > -    # windows
> > -    try:
> > -        n = int(os.environ['NUMBER_OF_PROCESSORS'])
> > -        if n > 0:
> > -            return n
> > -    except (KeyError, ValueError):
> > -        pass
> > -
> > -    return 1
> > +        return multiprocessing.cpu_count()
> > +    except NotImplementedError:
> > +        return 1
> >
> >  def _numworkers(ui):
> >      s = ui.config('worker', 'numcpus')
> >      if s:
> > _______________________________________________
> > Mercurial-devel mailing list
> > Mercurial-devel at selenic.com
> > https://selenic.com/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20150824/65341770/attachment.html>