From 2c089ef8c64cdc4f38769005ee79ecb965194d1a Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 29 Mar 2018 11:15:29 -0700 Subject: Document fork support --- doc/fork_support.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 doc/fork_support.md diff --git a/doc/fork_support.md b/doc/fork_support.md new file mode 100644 index 0000000000..141f87a435 --- /dev/null +++ b/doc/fork_support.md @@ -0,0 +1,46 @@ +# Background # + +In gRPC python, multithreading is not usable due to GIL +(global interpreter lock).Users are using multiprocessing and +concurrent.futures module to accomplish multiprocessing. These modules fork +processes underneath. Various issues have been reported when using these +modules. Historically, we didn't support forking in gRPC, but some users seem +to be doing fine until their code started to break 1.6. This was +likely caused by the addition of background c-threads and a background +Python thread. + +# Current Status # +## 1.7 ## +A pthread_atfork() handler was added in 1.7 to automatically shut down +the background c-threads when fork was called. This does not shut down the +background Python thread, so users could not have any open channels when +forking(). + +## 1.9 ## +A regression was noted in cases where users are doing fork/exec. This +was due to pthread_atfork() handler that was added in 1.7 to partially +support forking in gRPC. A deadlock can happen around GIL when pthread_atfork +handler is holding the lock while another thread is blocked on it and the +handler is waiting for that thread to terminate. We have provided a workaround +for this issue by allowing users to turn off the handler using env flag +```GRPC_ENABLE_FORK_SUPPORT=False```. This should be set whenever a user expects +to always call exec immediately following fork. It will disable the fork +handlers. + +# Future Work # +## 1.11 ## +The background Python thread was removed entirely. This allows forking +after creating a channel. However, the channel cannot be used by both the +parent and child process after the fork. + +Additionally, the fork/exec workaround of setting +```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Now the fork +handlers are automatically not run when multiple threads are calling +into gRPC. + + +## 1.1x ## +We would like to support forking() and using the channel from both the parent +and child process. Additionally, we would like to support servers that +use a prefork model, where the child processes accept the connections +and handle requests. -- cgit v1.2.3 From 34daf66ddcf3024a0fd4f534936b152c1007dd83 Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 10 May 2018 11:41:02 -0700 Subject: Clean up doc --- doc/fork_support.md | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/doc/fork_support.md b/doc/fork_support.md index 141f87a435..45466434ac 100644 --- a/doc/fork_support.md +++ b/doc/fork_support.md @@ -1,11 +1,13 @@ # Background # -In gRPC python, multithreading is not usable due to GIL -(global interpreter lock).Users are using multiprocessing and -concurrent.futures module to accomplish multiprocessing. These modules fork -processes underneath. Various issues have been reported when using these -modules. Historically, we didn't support forking in gRPC, but some users seem -to be doing fine until their code started to break 1.6. This was +In Python multithreading is ineffective at concurrency for CPU bound tasks +due to the GIL. +(global interpreter lock). Users use multiprocessing, subprocess, +and concurrent.futures.ProcessPoolExecutor, to work around the GIL. +These modules call fork() underneath the hood. Various issues have +been reported when using these modules with gRPC Python. +Historically, we didn't support forking in gRPC, but some users seem +to be doing fine until their code started to break on version 1.6. This was likely caused by the addition of background c-threads and a background Python thread. @@ -19,28 +21,29 @@ forking(). ## 1.9 ## A regression was noted in cases where users are doing fork/exec. This was due to pthread_atfork() handler that was added in 1.7 to partially -support forking in gRPC. A deadlock can happen around GIL when pthread_atfork -handler is holding the lock while another thread is blocked on it and the -handler is waiting for that thread to terminate. We have provided a workaround -for this issue by allowing users to turn off the handler using env flag -```GRPC_ENABLE_FORK_SUPPORT=False```. This should be set whenever a user expects -to always call exec immediately following fork. It will disable the fork -handlers. +support forking in gRPC. A deadlock can happen when pthread_atfork +handler is running, and an application thread is calling into gRPC. +We have provided a workaround for this issue by allowing users to turn +off the handler using env flag ```GRPC_ENABLE_FORK_SUPPORT=False```. +This should be set whenever a user expects to always call exec +immediately following fork. It will disable the fork handlers. -# Future Work # ## 1.11 ## The background Python thread was removed entirely. This allows forking after creating a channel. However, the channel cannot be used by both the -parent and child process after the fork. +parent and child process after the fork. Additionaly, the process should +not fork if there are any active RPCs on the channel. -Additionally, the fork/exec workaround of setting +# Future Work # +## 1.13 ## +The workaround when using fork/exec by setting ```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Now the fork handlers are automatically not run when multiple threads are calling into gRPC. ## 1.1x ## -We would like to support forking() and using the channel from both the parent +We would like to support forking and using the channel from both the parent and child process. Additionally, we would like to support servers that use a prefork model, where the child processes accept the connections and handle requests. -- cgit v1.2.3 From 93f49753accc92d1b64b4c45df318a1bcf990613 Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 10 May 2018 13:17:30 -0700 Subject: More PR feedback --- doc/fork_support.md | 49 +++++++++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/doc/fork_support.md b/doc/fork_support.md index 45466434ac..ba8483815d 100644 --- a/doc/fork_support.md +++ b/doc/fork_support.md @@ -1,22 +1,24 @@ # Background # -In Python multithreading is ineffective at concurrency for CPU bound tasks -due to the GIL. -(global interpreter lock). Users use multiprocessing, subprocess, -and concurrent.futures.ProcessPoolExecutor, to work around the GIL. -These modules call fork() underneath the hood. Various issues have -been reported when using these modules with gRPC Python. -Historically, we didn't support forking in gRPC, but some users seem +In Python, multithreading is ineffective at concurrency for CPU bound tasks +due to the GIL (global interpreter lock). Extension modules can release +the GIL in CPU bound tasks, but that isn't an option in pure Python. +Users use libraries such as multiprocessing, subprocess, concurrent.futures.ProcessPoolExecutor, +etc, to work around the GIL. These modules call fork() underneath the hood. Various issues have +been reported when using these modules with gRPC Python. gRPC Python wraps +gRPC core, which uses multithreading for performance, and hence doesn't support fork(). +Historically, we didn't support forking in gRPC, but some users seemed to be doing fine until their code started to break on version 1.6. This was likely caused by the addition of background c-threads and a background Python thread. # Current Status # -## 1.7 ## -A pthread_atfork() handler was added in 1.7 to automatically shut down -the background c-threads when fork was called. This does not shut down the -background Python thread, so users could not have any open channels when -forking(). + +## 1.11 ## +The background Python thread was removed entirely. This allows forking +after creating a channel. However, the channel must not have issued any +RPCs prior to the fork. Attempting to fork with an active channel that +has been used can result in deadlocks/corrupted wire data. ## 1.9 ## A regression was noted in cases where users are doing fork/exec. This @@ -28,22 +30,17 @@ off the handler using env flag ```GRPC_ENABLE_FORK_SUPPORT=False```. This should be set whenever a user expects to always call exec immediately following fork. It will disable the fork handlers. -## 1.11 ## -The background Python thread was removed entirely. This allows forking -after creating a channel. However, the channel cannot be used by both the -parent and child process after the fork. Additionaly, the process should -not fork if there are any active RPCs on the channel. +## 1.7 ## +A pthread_atfork() handler was added in 1.7 to automatically shut down +the background c-threads when fork was called. This does not shut down the +background Python thread, so users could not have any open channels when +forking(). # Future Work # + ## 1.13 ## The workaround when using fork/exec by setting -```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Now the fork -handlers are automatically not run when multiple threads are calling +```GRPC_ENABLE_FORK_SUPPORT=False``` should no longer be needed. Following +[this PR](https://github.com/grpc/grpc/pull/14647), fork +handlers will not automatically run when multiple threads are calling into gRPC. - - -## 1.1x ## -We would like to support forking and using the channel from both the parent -and child process. Additionally, we would like to support servers that -use a prefork model, where the child processes accept the connections -and handle requests. -- cgit v1.2.3 From 3323bafef8ef90fbe745d74f58b56ea67a13e5c5 Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 10 May 2018 14:14:37 -0700 Subject: Generate Projects --- tools/doxygen/Doxyfile.c++ | 1 + tools/doxygen/Doxyfile.c++.internal | 1 + tools/doxygen/Doxyfile.core | 1 + tools/doxygen/Doxyfile.core.internal | 1 + 4 files changed, 4 insertions(+) diff --git a/tools/doxygen/Doxyfile.c++ b/tools/doxygen/Doxyfile.c++ index 884eabb956..450ff95362 100644 --- a/tools/doxygen/Doxyfile.c++ +++ b/tools/doxygen/Doxyfile.c++ @@ -777,6 +777,7 @@ doc/cpp/perf_notes.md \ doc/environment_variables.md \ doc/epoll-polling-engine.md \ doc/fail_fast.md \ +doc/fork_support.md \ doc/g_stands_for.md \ doc/health-checking.md \ doc/http-grpc-status-mapping.md \ diff --git a/tools/doxygen/Doxyfile.c++.internal b/tools/doxygen/Doxyfile.c++.internal index 66796bae57..7c6b75d8ed 100644 --- a/tools/doxygen/Doxyfile.c++.internal +++ b/tools/doxygen/Doxyfile.c++.internal @@ -777,6 +777,7 @@ doc/cpp/perf_notes.md \ doc/environment_variables.md \ doc/epoll-polling-engine.md \ doc/fail_fast.md \ +doc/fork_support.md \ doc/g_stands_for.md \ doc/health-checking.md \ doc/http-grpc-status-mapping.md \ diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index 04f9d78850..c47d36ce85 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -779,6 +779,7 @@ doc/cpp-style-guide.md \ doc/environment_variables.md \ doc/epoll-polling-engine.md \ doc/fail_fast.md \ +doc/fork_support.md \ doc/g_stands_for.md \ doc/health-checking.md \ doc/http-grpc-status-mapping.md \ diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 82b4769544..ba7603f957 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -779,6 +779,7 @@ doc/cpp-style-guide.md \ doc/environment_variables.md \ doc/epoll-polling-engine.md \ doc/fail_fast.md \ +doc/fork_support.md \ doc/g_stands_for.md \ doc/health-checking.md \ doc/http-grpc-status-mapping.md \ -- cgit v1.2.3 From 35babffd0eff1e0b04092e9d5f9257b477441033 Mon Sep 17 00:00:00 2001 From: kpayson64 Date: Thu, 10 May 2018 15:13:59 -0700 Subject: Changes --- doc/fork_support.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/fork_support.md b/doc/fork_support.md index ba8483815d..d0f59f25da 100644 --- a/doc/fork_support.md +++ b/doc/fork_support.md @@ -4,9 +4,9 @@ In Python, multithreading is ineffective at concurrency for CPU bound tasks due to the GIL (global interpreter lock). Extension modules can release the GIL in CPU bound tasks, but that isn't an option in pure Python. Users use libraries such as multiprocessing, subprocess, concurrent.futures.ProcessPoolExecutor, -etc, to work around the GIL. These modules call fork() underneath the hood. Various issues have +etc, to work around the GIL. These modules call ```fork()``` underneath the hood. Various issues have been reported when using these modules with gRPC Python. gRPC Python wraps -gRPC core, which uses multithreading for performance, and hence doesn't support fork(). +gRPC core, which uses multithreading for performance, and hence doesn't support ```fork()```. Historically, we didn't support forking in gRPC, but some users seemed to be doing fine until their code started to break on version 1.6. This was likely caused by the addition of background c-threads and a background @@ -22,7 +22,7 @@ has been used can result in deadlocks/corrupted wire data. ## 1.9 ## A regression was noted in cases where users are doing fork/exec. This -was due to pthread_atfork() handler that was added in 1.7 to partially +was due to ```pthread_atfork()``` handler that was added in 1.7 to partially support forking in gRPC. A deadlock can happen when pthread_atfork handler is running, and an application thread is calling into gRPC. We have provided a workaround for this issue by allowing users to turn @@ -31,10 +31,10 @@ This should be set whenever a user expects to always call exec immediately following fork. It will disable the fork handlers. ## 1.7 ## -A pthread_atfork() handler was added in 1.7 to automatically shut down +A ```pthread_atfork()``` handler was added in 1.7 to automatically shut down the background c-threads when fork was called. This does not shut down the background Python thread, so users could not have any open channels when -forking(). +forking. # Future Work # -- cgit v1.2.3