Safe use of unix signals with multiprocessing module in python
""
Dear reader, since you are interested in this blog post, I am assuming
you are familiar with signal module, which lets you catch unix signals
from within your python script. I am also assuming you are an enthusiast
parallel processing fan boy like me and try to make the best use of python’s
parallel processsing frame work - multiprocessing in day to day
programming. :)
""
The conflict
signal is a great module! multiprocessing is too. But together they
behave aginst each other if you are not careful about their interaction.
If you ever tried to use both of them in a single script, you know what
I mean.
In a long running multiprocessing application, safe and clean exit
of the child processes is a must. There are a lot of ways to achieve this.
One common way is to send an Event object to the child process. The
child will occassionally check whether that exit event is set or not and
then exit if necessary. Here’s a demo script implementing this pattern,
where the parent process feeds some data to a child worker process -
#!/usr/bin/env python3
import time
import queue
import signal
from multiprocessing import Process, Event, Queue
# The worker is intentionally too much lazy!
def lazy_ass_worker(exit_event, work_queue):
while not exit_event.is_set():
try: work = work_queue.get(timeout=1.0)
except queue.Empty: continue
print("I did job {} already! :)".format(work))
print("A small nap won't hurt anyone!")
time.sleep(1.0)
print("Doing cleanup before leaving ...")
exit_event = Event()
work_queue = Queue()
# Spawn the worker process.
cp= Process(target=lazy_ass_worker, args=(exit_event, work_queue),)
cp.start()
# Send some integers to the worker process.
for x in range(100):
work_queue.put(x)
# We wait for CTRL+C from the user.
try: signal.pause()
except KeyboardInterrupt:
# Since our worker is too delicate, we should notify it with the
# exit event and then wait for it's safe arraival / joining.
exit_event.set()
cp.join()At the end of the script, I naively tried to catch KeyboardInterrupt
exception(which is generated by default as a response to SIGINT signal)
in the parent process and then tried to notify the child about exit condition
so that it can do all the clean up before exiting. But if you actually run
the above script, and press CTRL+C during the execution, the line
Doing cleanup before leaving … is never printed. Here’s what happens
in my computer -
oscar@notebook ~ % python3 demo.py
I did job 0 already! :)
A small nap won't hurt anyone!
I did job 1 already! :)
A small nap won't hurt anyone!
^CProcess Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
self.run()
File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "demo.py", line 16, in lazy_ass_worker
time.sleep(1.0)
KeyboardInterrupt
oscar@notebook ~ %What just happened?
Signals are propagated down the process tree. - that is what has
happened! Even if you catch a signal in the parent process, child processes
still receive and handle that signal. This comes in conflict with the kind of
pattern we used in the above demo, where synchronization primitives from
multiprocessing module is used for safe cleanup.
The workaround
Every child process spawned by the multiprocessing module inherits signal
handlers from the parent process. If we set the signal handlers to SIG_IGN
for our target signals before spawning new processes, the child processes
will ignore the signals. With this strategy, our demo needs some minor
modifications -
# Save a reference to the original signal handler for SIGINT.
default_handler = signal.getsignal(signal.SIGINT)
# Set signal handling of SIGINT to ignore mode.
signal.signal(signal.SIGINT, signal.SIG_IGN)
exit_event = Event()
work_queue = Queue()
# Spawn the worker process.
cp= Process(target=lazy_ass_worker, args=(exit_event, work_queue),)
cp.start()
# Since we spawned all the necessary processes already,
# restore default signal handling for the parent process.
signal.signal(signal.SIGINT, default_handler)In the above code, after all the necessary process spawnings are done, the
default signal handler is restored. If you use custom signal handlers, they
should be defined at this stage. Note that using some facilities of
multiprocessing module, such as Queue, Manager etc implicitly spawns
additional processes. They should be be taken care of in a similar manner.
Caveats
Blocking important termination signals like SIGINT, SIGTERM etc. in the
child process is problematic at early stages of development.
Programming errors or runtime errors in the code can leave your
development system dirty with lots of zombie processes. In that case,
just kill them with a SIGKILL, as it can not be ignored.