From 2c089ef8c64cdc4f38769005ee79ecb965194d1a Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 29 Mar 2018 11:15:29 -0700 Subject: Document fork support --- doc/fork_support.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 doc/fork_support.md (limited to 'doc/fork_support.md') diff --git a/doc/fork_support.md b/doc/fork_support.md new file mode 100644 index 0000000000..141f87a435 --- /dev/null +++ b/doc/fork_support.md @@ -0,0 +1,46 @@ +# Background # + +In gRPC python, multithreading is not usable due to GIL +(global interpreter lock).Users are using multiprocessing and +concurrent.futures module to accomplish multiprocessing. These modules fork +processes underneath. Various issues have been reported when using these +modules. Historically, we didn't support forking in gRPC, but some users seem +to be doing fine until their code started to break 1.6. This was +likely caused by the addition of background c-threads and a background +Python thread. + +# Current Status # +## 1.7 ## +A pthread_atfork() handler was added in 1.7 to automatically shut down +the background c-threads when fork was called. This does not shut down the +background Python thread, so users could not have any open channels when +forking(). + +## 1.9 ## +A regression was noted in cases where users are doing fork/exec. This +was due to pthread_atfork() handler that was added in 1.7 to partially +support forking in gRPC. A deadlock can happen around GIL when pthread_atfork +handler is holding the lock while another thread is blocked on it and the +handler is waiting for that thread to terminate. We have provided a workaround +for this issue by allowing users to turn off the handler using env flag +```GRPC_ENABLE_FORK_SUPPORT=False```. This should be set whenever a user expects +to always call exec immediately following fork. It will disable the fork +handlers. + +# Future Work # +## 1.11 ## +The background Python thread was removed entirely. This allows forking +after creating a channel. However, the channel cannot be used by both the +parent and child process after the fork. + +Additionally, the fork/exec workaround of setting +```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Now the fork +handlers are automatically not run when multiple threads are calling +into gRPC. + + +## 1.1x ## +We would like to support forking() and using the channel from both the parent +and child process. Additionally, we would like to support servers that +use a prefork model, where the child processes accept the connections +and handle requests. -- cgit v1.2.3