Python Multiprocess using Pool fails on AWS Ubuntu
I have a simple string matching script that tests just fine for
multiprocessing with up to 8 Pool workers on my local mac with 4 cores.
However, the same script on an AWS c1.xlarge with 8 cores generally kills
all but 2 workers, the CPU only works at 25%, and after a few rounds stops
with MemoryError.
I'm not too familiar with server configuration, so I'm wondering if there
are any settings to tweak?
The pool implementation looks as follows, but doesn't seem to be the issue
as it works locally. There would be several thousand targets per worker,
and it doesn't run past the first five or so. Happy to share more of the
code if necessary.
pool = Pool(processes = numProcesses)
totalTargets = len(getTargets('all'))
targetsPerBatch = totalTargets / numProcesses
pool.map_async(runMatch, itertools.izip(itertools.repeat(targetsPerBatch),
xrange(0, totalTargets, targetsPerBatch))).get(99999999)
pool.close()
pool.join()
No comments:
Post a Comment