From a74492e8a41a086f82d73f640d0e69d187c38be6 Mon Sep 17 00:00:00 2001 From: Sree Kuchibhotla Date: Thu, 27 Sep 2018 11:11:01 -0700 Subject: Polling engine usage in client server --- .../grpc-client-server-polling-engine-usage.md | 32 +++++++++++++++++++++ doc/images/grpc-call-channel-cq.png | Bin 0 -> 46078 bytes doc/images/grpc-client-lb-pss.png | Bin 0 -> 56397 bytes doc/images/grpc-server-cq-fds.png | Bin 0 -> 42096 bytes 4 files changed, 32 insertions(+) create mode 100644 doc/core/grpc-client-server-polling-engine-usage.md create mode 100644 doc/images/grpc-call-channel-cq.png create mode 100644 doc/images/grpc-client-lb-pss.png create mode 100644 doc/images/grpc-server-cq-fds.png (limited to 'doc') diff --git a/doc/core/grpc-client-server-polling-engine-usage.md b/doc/core/grpc-client-server-polling-engine-usage.md new file mode 100644 index 0000000000..8de3979a56 --- /dev/null +++ b/doc/core/grpc-client-server-polling-engine-usage.md @@ -0,0 +1,32 @@ +# Polling Engine Usage on gRPC client and Server + +_Author: Sree Kuchibhotla (@sreecha) - Sep 2018_ + + +This document talks about how polling engine is used in gRPC core (both on client and server code paths). + +## gRPC client + +### Relation between Call, Channel (sub-channels), Completion queue, `grpc_pollset` +- A gRPC Call is tied to a channel (more specifically a sub-channel) and a completion queue for the lifetime of the call. +- Once a _sub-channel_ is picked for the call, the file-descriptor (socket fd in case of TCP channels) is added to the pollset corresponding to call's completion queue. (Recall that as per [grpc-cq](grpc-cq.md), a completion queue has a pollset by default) + +![image](../images/grpc-call-channel-cq.png) + + +### Making progress on Async `connect()` on sub-channels (`grpc_pollset_set` usecase) +- A gRPC channel is created between a client and a 'target'. The 'target' may resolve in to one or more backend servers. +- A sub-channel is the 'connection' from a client to the backend server +- While establishing sub-cannels (i.e connections) to the backends, gRPC issues async [`connect()`](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_client_posix.cc#L296) calls which may not complete right away. When the `connect()` eventually succeeds, the socket fd is makde 'writable' + - This means that the polling engine must be monitoring all these sub-channel `fd`s for writable events and we need to make sure there is a polling thread that monitors all these fds + - To accomplish this, the `grpc_pollset_set` is used the following way (see picture below) + +![image](../images/grpc-client-lb-pss.png) + +## gRPC server + +- The listening fd (i.e., the socket fd corresponding to the server listening port) is added to each of the server completion queues. Note that in gRPC we use SO_REUSEPORT option and create multiple listening fds but all of them map to the same listening port +- A new incoming channel is randomly assigned to some server completion queue (see picture below) + +![image](../images/grpc-server-cq-fds.png) + diff --git a/doc/images/grpc-call-channel-cq.png b/doc/images/grpc-call-channel-cq.png new file mode 100644 index 0000000000..d73b987ee9 Binary files /dev/null and b/doc/images/grpc-call-channel-cq.png differ diff --git a/doc/images/grpc-client-lb-pss.png b/doc/images/grpc-client-lb-pss.png new file mode 100644 index 0000000000..188e3654ec Binary files /dev/null and b/doc/images/grpc-client-lb-pss.png differ diff --git a/doc/images/grpc-server-cq-fds.png b/doc/images/grpc-server-cq-fds.png new file mode 100644 index 0000000000..e52bfac81c Binary files /dev/null and b/doc/images/grpc-server-cq-fds.png differ -- cgit v1.2.3