Sunday, 16 November 2014

More on logging and multiprocessing

My original effort turned out to be too simple. It works OK on Linux but there are problems with Windows because the logger properties are not inherited by child processes. The solution was to use some ideas from Vinay Sajip's excellent post.
I had to modify it a little to cope with my mixed single/multiprocessing modules and ended up defining just the one handler, which passed every log message to the queue, regardless of whether it came from the main process or the child processes.
The clever bit was combining the logger with a multiprocessing.pool object that is re-used to save the overhead of launching and killing the child processes. For Linux this happens automatically, but with Windows I had to send an initializing task to each member of the pool to set up the logger.

The queue is set up like this:

    m = mp.Manager()
    q = m.Queue()
    l = mp.Process(target=ml.listener_process,
                   args=(q, ml.listener_configurer))
    l.start()
    ml.worker_configurer(q, delay=False)
    logger = logging.getLogger('cupcake')

...do stuff...

    logger.info('Ending listener process')
    q.put_nowait(None)
    l.join()


and the initializer sent to child processes in Windows is set up like this:

            if os.name == 'nt':
                _ = pool.map_async(ml.worker_configurer,
                                   [self._q for _ in range(mp.cpu_count())])


I had to add a short time delay in the worker_configurer method otherwise it would exit so quickly it would sometimes be executed multiple times on one process and not at all on others.

No comments:

Post a Comment