From 31d3134d5b525e2483d023d5e8efe461b7eab171 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Wed, 26 Sep 2018 12:36:11 -0700 Subject: Cq documentation --- doc/core/grpc-cq.md | 64 +++++++++++++++++++++++++++++++++++++++++++++++++ doc/images/grpc-cq.png | Bin 0 -> 41659 bytes 2 files changed, 64 insertions(+) create mode 100644 doc/core/grpc-cq.md create mode 100644 doc/images/grpc-cq.png (limited to 'doc') diff --git a/doc/core/grpc-cq.md b/doc/core/grpc-cq.md new file mode 100644 index 0000000000..b485c35456 --- /dev/null +++ b/doc/core/grpc-cq.md @@ -0,0 +1,64 @@ +# gRPC Completion Queue + +_Author: Sree Kuchibhotla (@sreecha) - Sep 2018_ + +Code: [completion_queue.cc](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/surface/completion_queue.cc) + +This document gives an overview of completion queue architecture and focuses mainly on the interaction between completion queue and the Polling engine layer. + +## Completion queue attributes +Completion queue has two attributes + + - Completion_type: + - GRPC_CQ_NEXT: grpc_completion_queue_next() can be called (but not grpc_completion_queue_pluck()) + - GRPC_CQ_PLUCK: grpc_completion_queue_pluck() can be called (but not grpc_completion_queue_next()) + - GRPC_CQ_CALLBACK: The tags in the queue are function pointers to callbacks. Also, neither next() nor pluck() can be called on this + + - Polling_type: + - GRPC_CQ_NON_POLLING: Threads calling completion_queue_next/pluck do not do any polling + - GRPC_CQ_DEFAULT_POLLING: Threads calling completion_queue_next/pluck do polling + - GRPC_CQ_NON_LISTENING: Functionally similar to default polling except for a boolean attribute that states that the cq is non-listening. This is used by the grpc-server code to not associate any listening sockets with this completion-queue’s pollset + + +## Details + +![image](../images/grpc-cq.png) + + +### **grpc\_completion\_queue\_next()** & **grpc_completion_queue_pluck()** APIS + + +``` C++ +grpc_completion_queue_next(cq, deadline)/pluck(cq, deadline, tag) { + while(true) { + \\ 1. If an event is queued in the completion queue, dequeue and return + \\ (in case of pluck() dequeue only if the tag is the one we are interested in) + + \\ 2. If completion queue shutdown return + + \\ 3. In case of pluck, add (tag, worker) pair to the tag<->worker map on the cq + + \\ 4. Call grpc_pollset_work(cq’s-pollset, deadline) to do polling + \\ Note that if this function found some fds to be readable/writable/error, + \\ it would have scheduled those closures (which may queue completion events + \\ on SOME completion queue - not necessarily this one) + } +} +``` + +### Queuing a completion event (i.e., "tag") + +``` C++ +grpc_cq_end_op(cq, tag) { + \\ 1. Queue the tag in the event queue + + \\ 2. Find the pollset corresponding to the completion queue + \\ (i) If the cq is of type GRPC_CQ_NEXT, then KICK ANY worker + \\ i.e., call grpc_pollset_kick(pollset, nullptr) + \\ (ii) If the cq is of type GRPC_CQ_PLUCK, then search the tag<->worker + \\ map on the completion queue to find the worker. Then specifically + \\ kick that worker i.e call grpc_pollset_kick(pollset, worker) +} + +``` + diff --git a/doc/images/grpc-cq.png b/doc/images/grpc-cq.png new file mode 100644 index 0000000000..2d9e095862 Binary files /dev/null and b/doc/images/grpc-cq.png differ -- cgit v1.2.3 From a74492e8a41a086f82d73f640d0e69d187c38be6 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 27 Sep 2018 11:11:01 -0700 Subject: Polling engine usage in client server --- .../grpc-client-server-polling-engine-usage.md | 32 +++++++++++++++++++++ doc/images/grpc-call-channel-cq.png | Bin 0 -> 46078 bytes doc/images/grpc-client-lb-pss.png | Bin 0 -> 56397 bytes doc/images/grpc-server-cq-fds.png | Bin 0 -> 42096 bytes tools/doxygen/Doxyfile.core | 1 + tools/doxygen/Doxyfile.core.internal | 1 + 6 files changed, 34 insertions(+) create mode 100644 doc/core/grpc-client-server-polling-engine-usage.md create mode 100644 doc/images/grpc-call-channel-cq.png create mode 100644 doc/images/grpc-client-lb-pss.png create mode 100644 doc/images/grpc-server-cq-fds.png (limited to 'doc') diff --git a/doc/core/grpc-client-server-polling-engine-usage.md b/doc/core/grpc-client-server-polling-engine-usage.md new file mode 100644 index 0000000000..8de3979a56 --- /dev/null +++ b/doc/core/grpc-client-server-polling-engine-usage.md @@ -0,0 +1,32 @@ +# Polling Engine Usage on gRPC client and Server + +_Author: Sree Kuchibhotla (@sreecha) - Sep 2018_ + + +This document talks about how polling engine is used in gRPC core (both on client and server code paths). + +## gRPC client + +### Relation between Call, Channel (sub-channels), Completion queue, `grpc_pollset` +- A gRPC Call is tied to a channel (more specifically a sub-channel) and a completion queue for the lifetime of the call. +- Once a _sub-channel_ is picked for the call, the file-descriptor (socket fd in case of TCP channels) is added to the pollset corresponding to call's completion queue. (Recall that as per [grpc-cq](grpc-cq.md), a completion queue has a pollset by default) + +![image](../images/grpc-call-channel-cq.png) + + +### Making progress on Async `connect()` on sub-channels (`grpc_pollset_set` usecase) +- A gRPC channel is created between a client and a 'target'. The 'target' may resolve in to one or more backend servers. +- A sub-channel is the 'connection' from a client to the backend server +- While establishing sub-cannels (i.e connections) to the backends, gRPC issues async [`connect()`](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_client_posix.cc#L296) calls which may not complete right away. When the `connect()` eventually succeeds, the socket fd is makde 'writable' + - This means that the polling engine must be monitoring all these sub-channel `fd`s for writable events and we need to make sure there is a polling thread that monitors all these fds + - To accomplish this, the `grpc_pollset_set` is used the following way (see picture below) + +![image](../images/grpc-client-lb-pss.png) + +## gRPC server + +- The listening fd (i.e., the socket fd corresponding to the server listening port) is added to each of the server completion queues. Note that in gRPC we use SO_REUSEPORT option and create multiple listening fds but all of them map to the same listening port +- A new incoming channel is randomly assigned to some server completion queue (see picture below) + +![image](../images/grpc-server-cq-fds.png) + diff --git a/doc/images/grpc-call-channel-cq.png b/doc/images/grpc-call-channel-cq.png new file mode 100644 index 0000000000..d73b987ee9 Binary files /dev/null and b/doc/images/grpc-call-channel-cq.png differ diff --git a/doc/images/grpc-client-lb-pss.png b/doc/images/grpc-client-lb-pss.png new file mode 100644 index 0000000000..188e3654ec Binary files /dev/null and b/doc/images/grpc-client-lb-pss.png differ diff --git a/doc/images/grpc-server-cq-fds.png b/doc/images/grpc-server-cq-fds.png new file mode 100644 index 0000000000..e52bfac81c Binary files /dev/null and b/doc/images/grpc-server-cq-fds.png differ diff --git a/tools/doxygen/Doxyfile.core b/tools/doxygen/Doxyfile.core index aa75bc6828..43a9b846a2 100644 --- a/tools/doxygen/Doxyfile.core +++ b/tools/doxygen/Doxyfile.core @@ -771,6 +771,7 @@ doc/compression_cookbook.md \ doc/connection-backoff-interop-test-description.md \ doc/connection-backoff.md \ doc/connectivity-semantics-and-api.md \ +doc/core/grpc-client-server-polling-engine-usage.md \ doc/core/grpc-error.md \ doc/core/moving-to-c++.md \ doc/core/pending_api_cleanups.md \ diff --git a/tools/doxygen/Doxyfile.core.internal b/tools/doxygen/Doxyfile.core.internal index 9186056733..6b08d49346 100644 --- a/tools/doxygen/Doxyfile.core.internal +++ b/tools/doxygen/Doxyfile.core.internal @@ -771,6 +771,7 @@ doc/compression_cookbook.md \ doc/connection-backoff-interop-test-description.md \ doc/connection-backoff.md \ doc/connectivity-semantics-and-api.md \ +doc/core/grpc-client-server-polling-engine-usage.md \ doc/core/grpc-error.md \ doc/core/moving-to-c++.md \ doc/core/pending_api_cleanups.md \ -- cgit v1.2.3 From c1f880d9fe0e0ce810fb50f7ac6ebe5bedee3922 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 27 Sep 2018 12:28:21 -0700 Subject: Minor change --- doc/core/grpc-client-server-polling-engine-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/core/grpc-client-server-polling-engine-usage.md b/doc/core/grpc-client-server-polling-engine-usage.md index 8de3979a56..fa54f94621 100644 --- a/doc/core/grpc-client-server-polling-engine-usage.md +++ b/doc/core/grpc-client-server-polling-engine-usage.md @@ -26,7 +26,7 @@ This document talks about how polling engine is used in gRPC core (both on clien ## gRPC server - The listening fd (i.e., the socket fd corresponding to the server listening port) is added to each of the server completion queues. Note that in gRPC we use SO_REUSEPORT option and create multiple listening fds but all of them map to the same listening port -- A new incoming channel is randomly assigned to some server completion queue (see picture below) +- A new incoming channel is assigned to some server completion queue picked randomly (note that we currently [round-robin](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_server_posix.cc#L231) over the server completion queues) ![image](../images/grpc-server-cq-fds.png) -- cgit v1.2.3 From 635e0bd1e5149aee941d6b6064bb7cef738e7537 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 27 Sep 2018 12:30:51 -0700 Subject: fix typo --- doc/core/grpc-client-server-polling-engine-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/core/grpc-client-server-polling-engine-usage.md b/doc/core/grpc-client-server-polling-engine-usage.md index fa54f94621..3a560e71a8 100644 --- a/doc/core/grpc-client-server-polling-engine-usage.md +++ b/doc/core/grpc-client-server-polling-engine-usage.md @@ -17,7 +17,7 @@ This document talks about how polling engine is used in gRPC core (both on clien ### Making progress on Async `connect()` on sub-channels (`grpc_pollset_set` usecase) - A gRPC channel is created between a client and a 'target'. The 'target' may resolve in to one or more backend servers. - A sub-channel is the 'connection' from a client to the backend server -- While establishing sub-cannels (i.e connections) to the backends, gRPC issues async [`connect()`](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_client_posix.cc#L296) calls which may not complete right away. When the `connect()` eventually succeeds, the socket fd is makde 'writable' +- While establishing sub-cannels (i.e connections) to the backends, gRPC issues async [`connect()`](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_client_posix.cc#L296) calls which may not complete right away. When the `connect()` eventually succeeds, the socket fd is make 'writable' - This means that the polling engine must be monitoring all these sub-channel `fd`s for writable events and we need to make sure there is a polling thread that monitors all these fds - To accomplish this, the `grpc_pollset_set` is used the following way (see picture below) -- cgit v1.2.3