Merge branch 'master' into sreek-pe-doc

6 files changed, 96 insertions, 0 deletions

diff --git a/doc/core/grpc-client-server-polling-engine-usage.md b/doc/core/grpc-client-server-polling-engine-usage.md
new file mode 100644
index 0000000000..3a560e71a8
--- /dev/null
+++ b/doc/core/grpc-client-server-polling-engine-usage.md
@@ -0,0 +1,32 @@
+# Polling Engine Usage on gRPC client and Server
+
+_Author: Sree Kuchibhotla (@sreecha) - Sep 2018_
+
+
+This document talks about how polling engine is used in gRPC core (both on client and server code paths).
+
+## gRPC client
+
+### Relation between Call, Channel (sub-channels), Completion queue, `grpc_pollset` 
+- A gRPC Call is tied to a channel (more specifically a sub-channel) and a completion queue for the lifetime of the call.
+- Once a _sub-channel_ is picked for the call, the file-descriptor (socket fd in case of TCP channels) is added to the pollset corresponding to call's completion queue. (Recall that as per [grpc-cq](grpc-cq.md), a completion queue has a pollset by default)
+
+![image](../images/grpc-call-channel-cq.png)
+
+
+### Making progress on Async `connect()` on sub-channels  (`grpc_pollset_set` usecase)
+- A gRPC channel is created between a client and a 'target'. The 'target' may resolve in to one or more backend servers.
+- A sub-channel is the 'connection' from a client to the backend server
+- While establishing sub-cannels (i.e connections) to the backends, gRPC issues async [`connect()`](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_client_posix.cc#L296) calls which may not complete right away.  When the `connect()` eventually succeeds, the socket fd is make 'writable'
+  - This means that the polling engine must be monitoring all these sub-channel `fd`s for writable events and we need to make sure there is a polling thread that monitors all these fds
+  - To accomplish this, the `grpc_pollset_set` is used the following way (see picture below)
+
+![image](../images/grpc-client-lb-pss.png)
+
+## gRPC server
+
+- The listening fd (i.e., the socket fd corresponding to the server listening port) is added to each of the server completion queues. Note that in gRPC we use SO_REUSEPORT option and create multiple listening fds but all of them map to the same listening port
+- A new incoming channel is assigned to some server completion queue picked randomly (note that we currently [round-robin](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/iomgr/tcp_server_posix.cc#L231) over the server completion queues)
+
+![image](../images/grpc-server-cq-fds.png)
+
diff --git a/doc/core/grpc-cq.md b/doc/core/grpc-cq.md
new file mode 100644
index 0000000000..b485c35456
--- /dev/null
+++ b/doc/core/grpc-cq.md
@@ -0,0 +1,64 @@
+# gRPC Completion Queue
+
+_Author: Sree Kuchibhotla (@sreecha) - Sep 2018_
+
+Code: [completion_queue.cc](https://github.com/grpc/grpc/blob/v1.15.1/src/core/lib/surface/completion_queue.cc)
+
+This document gives an overview of completion queue architecture and focuses mainly on the interaction between completion queue and the Polling engine layer.
+
+## Completion queue attributes
+Completion queue has two attributes
+
+  - Completion_type:
+    - GRPC_CQ_NEXT: grpc_completion_queue_next() can be called (but not grpc_completion_queue_pluck())
+    - GRPC_CQ_PLUCK: grpc_completion_queue_pluck() can be called (but not grpc_completion_queue_next())
+    - GRPC_CQ_CALLBACK: The tags in the queue are function pointers to callbacks. Also, neither next() nor pluck() can be called on this
+
+  - Polling_type:
+    - GRPC_CQ_NON_POLLING: Threads calling completion_queue_next/pluck do not do any polling
+    - GRPC_CQ_DEFAULT_POLLING: Threads calling completion_queue_next/pluck do polling
+    - GRPC_CQ_NON_LISTENING:  Functionally similar to default polling except for a boolean attribute that states that the cq is non-listening. This is used by the grpc-server code to not associate any listening sockets with this completion-queue’s pollset
+
+
+## Details
+
+![image](../images/grpc-cq.png)
+
+
+### **grpc\_completion\_queue\_next()** & **grpc_completion_queue_pluck()** APIS
+
+
+``` C++
+grpc_completion_queue_next(cq, deadline)/pluck(cq, deadline, tag) {
+  while(true) {
+    \\ 1. If an event is queued in the completion queue, dequeue and return
+    \\    (in case of pluck() dequeue only if the tag is the one we are interested in)
+
+    \\ 2. If completion queue shutdown return
+
+    \\ 3. In case of pluck, add (tag, worker) pair to the tag<->worker map on the cq
+
+    \\ 4.  Call grpc_pollset_work(cq’s-pollset, deadline) to do polling
+    \\     Note that if this function found some fds to be readable/writable/error,
+    \\     it would have scheduled those closures (which may queue completion events
+    \\    on SOME completion queue - not necessarily this one)
+  }
+}
+```
+
+### Queuing a completion event (i.e., "tag")
+
+``` C++
+grpc_cq_end_op(cq, tag) {
+  \\ 1. Queue the tag in the event queue
+
+  \\ 2. Find the pollset corresponding to the completion queue
+  \\     (i)  If the cq is of type GRPC_CQ_NEXT, then KICK ANY worker
+  \\          i.e., call grpc_pollset_kick(pollset, nullptr)
+  \\     (ii) If the cq is of type GRPC_CQ_PLUCK, then search the tag<->worker
+  \\          map on the completion queue to find the worker. Then specifically
+  \\          kick that worker i.e  call grpc_pollset_kick(pollset, worker)
+}
+
+```
+ 
diff --git a/doc/images/grpc-call-channel-cq.png b/doc/images/grpc-call-channel-cq.png
new file mode 100644
index 0000000000..d73b987ee9
--- /dev/null
+++ b/doc/images/grpc-call-channel-cq.png
diff --git a/doc/images/grpc-client-lb-pss.png b/doc/images/grpc-client-lb-pss.png
new file mode 100644
index 0000000000..188e3654ec
--- /dev/null
+++ b/doc/images/grpc-client-lb-pss.png
diff --git a/doc/images/grpc-cq.png b/doc/images/grpc-cq.png
new file mode 100644
index 0000000000..2d9e095862
--- /dev/null
+++ b/doc/images/grpc-cq.png
diff --git a/doc/images/grpc-server-cq-fds.png b/doc/images/grpc-server-cq-fds.png
new file mode 100644
index 0000000000..e52bfac81c
--- /dev/null
+++ b/doc/images/grpc-server-cq-fds.png