I found a better way to set up the logger in child processes, thanks to StackOverflow! I now call worker-configurer at the beginning of each pool process, but the configurer checks for existing logger handlers:

`def worker_configurer(queue):
    root = logging.getLogger()
    if not root.handlers:
        h = QueueHandler(queue)
        root.addHandler(h)
        root.setLevel(logging.DEBUG)
    return
`
No more need for os checking and artificial time delays!

More on logging and multiprocessing

My original effort turned out to be too simple. It works OK on Linux but there are problems with Windows because the logger properties are not inherited by child processes. The solution was to use some ideas from Vinay Sajip's excellent post.
I had to modify it a little to cope with my mixed single/multiprocessing modules and ended up defining just the one handler, which passed every log message to the queue, regardless of whether it came from the main process or the child processes.
The clever bit was combining the logger with a multiprocessing.pool object that is re-used to save the overhead of launching and killing the child processes. For Linux this happens automatically, but with Windows I had to send an initializing task to each member of the pool to set up the logger.

The queue is set up like this:

    m = mp.Manager()
    q = m.Queue()
    l = mp.Process(target=ml.listener_process,
                   args=(q, ml.listener_configurer))
    l.start()
    ml.worker_configurer(q, delay=False)
    logger = logging.getLogger('cupcake')

...do stuff...

    logger.info('Ending listener process')
    q.put_nowait(None)
    l.join()

and the initializer sent to child processes in Windows is set up like this:

            if os.name == 'nt':
                _ = pool.map_async(ml.worker_configurer,
                                   [self._q for _ in range(mp.cpu_count())])

I had to add a short time delay in the worker_configurer method otherwise it would exit so quickly it would sometimes be executed multiple times on one process and not at all on others.

Logging and warnings

Now that some of my code is nearing release quality, I'm replacing all the diagnostic 'print' statements with logging commands. There are a few articles on StackOverflow describing how to make logging process-safe I don't really need that since the number of messages I expect to log are so few that there is no real chance of two processes trying to write to the file at the same time.

One tricky part was catching warnings that are thrown by the underlying scipy library. It turned out to be quite easy to fix.

The module containing main() has to import logging and then at the start of the main() function I add the following

logging.basicConfig(filename='example.log',
                    filemode='w',
                    format='%(levelname)s:%(message)s',
                    level=logging.DEBUG)
 
Then in the sub-modules, import logging and warnings and I used this structure to catch the warning, convert it to an exception that I could then pass to the logger. Finally I ran the function again without catching the warning to regenerate the result (which was still valid; the warning was just for not achieving an accuracy limit)

def function_that_gets_scipy_warning(self, args):
    with warnings.catch_warnings():
        warnings.filterwarnings('error')
        try:
            result = scipy.somefunction(args)
        except Warning, warn:
            logging.warning(warn)
            warnings.filterwarnings('ignore')
            result = scipy.somefunction(args)
    return result 
 
This writes the warning out to the example.log file and not to the terminal.

This blog is intended to record some of the interesting stuff I discover and learn while developing a program to design aircraft wing and engine ice protection systems.
It will cover both the physical problem and the coded solutions. There will be no particular order, it will just be as it occurs to me.
My program is written mostly in python with heavy use of the numpy and scipy libraries and a fair bit of Fortran using numpy's f2py tool and my first post is related to using dictionary keys in python.
To capture the properties of each individual physical element, be it a slat, an intake lip or an engine spinner for example, I have a class called 'body'. In all the main code I can save information relating to each body in a dictionary using the body object itself as a key. This makes it easy to call the methods of the body object when iterating through the dictionary, for example:

for body in bodies_list:
....body.some_method()

However, I also make use of the multiprocessing module in python to make use of multiple cores, when I use a new process to calculate something about a body I pass the body object to the new process, but what happens is a new copy of the body object is created in the new process so trying to write back to the dictionary would give me a key error. To get around this I give the body object a public property 'name', which gets copied to the new process as a string and this can be used as a key when writing back to the dictionary.
References in python is one of the behaviours that is more alien to languages such as C++ where it is clearer what is an object and what is a pointer.