Merge branch 'master' of github.com:grpc/grpc into publish_compression_spec

8 files changed, 277 insertions, 96 deletions

diff --git a/doc/binary-logging.md b/doc/binary-logging.md
new file mode 100644
index 0000000000..69020d9828
--- /dev/null
+++ b/doc/binary-logging.md
@@ -0,0 +1,59 @@
+# Binary Logging
+
+## Format
+
+The log format is described in [this proto file](src/proto/grpc/binary_log/v1alpha/log.proto). It is intended that multiple parts of the call will be logged in separate files, and then correlated by analysis tools using the rpc\_id.
+
+## API
+
+The binary logger will be a separate library from gRPC, in each language that we support. The user will need to explicitly call into the library to generate logs. The library will provide the ability to log sending or receiving, as relevant, the following on both the client and the server:
+
+ - Initial metadata
+ - Messages
+ - Status with trailing metadata from the server
+ - Additional key/value pairs that are associated with a call but not sent over the wire
+
+The following is an example of what such an API could look like in C++:
+
+```c++
+// The context provides the method_name, deadline, peer, and metadata contents.
+// direction = CLIENT_SEND
+LogRequestHeaders(ClientContext context);
+// direction = SERVER_RECV
+LogRequestHeaders(ServerContext context);
+
+// The context provides the metadata contents
+// direction = CLIENT_RECV
+LogResponseHeaders(ClientContext context);
+// direction = SERVER_SEND
+LogResponseHeaders(ServerContext context);
+
+// The context provides the metadata contents
+// direction = CLIENT_RECV
+LogStatus(ClientContext context, grpc_status_code code, string details);
+// direction = SERVER_SEND
+LogStatus(ServerContext context, grpc_status_code code, string details);
+
+// The context provides the user data contents
+// direction = CLIENT_SEND
+LogUserData(ClientContext context);
+// direction = SERVER_SEND
+LogUserData(ServerContext context);
+
+// direction = CLIENT_SEND
+LogRequestMessage(ClientContext context, uint32_t length, T message);
+// direction = SERVER_RECV
+LogRequestMessage(ServerContext context, uint32_t length, T message);
+// direction = CLIENT_RECV
+LogResponseMessage(ClientContext context, uint32_t length, T message);
+// direction = SERVER_SEND
+LogResponseMessage(ServerContext context, uint32_t length, T message);
+```
+
+In all of those cases, the `rpc_id` is provided by the context, and each combination of method and context argument type implies a single direction, as noted in the comments.
+
+For the message log functions, the `length` argument indicates the length of the complete message, and the `message` argument may be only part of the complete message, stripped of sensitive material and/or shortened for efficiency.
+
+## Language differences
+
+In other languages, more or less data will need to be passed explicitly as separate arguments. In some languages, for example, the metadata will be separate from the context-like object and will need to be passed as a separate argument.
diff --git a/doc/c-style-guide.md b/doc/c-style-guide.md
index 87de8892dd..d6f9bbd7d4 100644
--- a/doc/c-style-guide.md
+++ b/doc/c-style-guide.md
@@ -11,7 +11,7 @@ General
 
 - Layout rules are defined by clang-format, and all code should be passed through
   clang-format. A (docker-based) script to do so is included in 
-  tools/distrib/clang_format_code.sh.
+  [tools/distrib/clang\_format\_code.sh] (../tools/distrib/clang_format_code.sh).
 
 Header Files
 ------------
diff --git a/doc/command_line_tool.md b/doc/command_line_tool.md
new file mode 100644
index 0000000000..89a70548b8
--- /dev/null
+++ b/doc/command_line_tool.md
@@ -0,0 +1,77 @@
+# gRPC command line tool
+
+## Overview
+
+This document describes the command line tool that comes with gRPC repository. It is desireable to have command line
+tools written in other languages to roughly follow the same syntax and flags.
+
+At this point, the tool needs to be built from source, and it should be moved out to grpc-tools repository as a stand
+alone application once it is mature enough.
+
+## Core functionality
+
+The command line tool can do the following things:
+
+- Send unary rpc.
+- Attach metadata and display received metadata.
+- Handle common authentication to server.
+- Find the request/response types from a given proto file.
+- Read proto request in text form.
+- Read request in wire form (for protobuf messages, this means serialized binary form).
+- Display proto response in text form.
+- Write response in wire form to a file.
+
+The command line tool should support the following things:
+
+- List server services and methods through server reflection.
+- Infer request/response types from server reflection result.
+- Fine-grained auth control (such as, use this oauth token to talk to the server).
+- Send streaming rpc.
+
+## Code location
+
+To use the tool, you need to get the grpc repository and in the grpc directory execute
+
+```
+$ make grpc_cli
+```
+
+The main file can be found at
+https://github.com/grpc/grpc/blob/master/test/cpp/util/grpc_cli.cc
+
+## Usage
+
+### Basic usage
+
+Send a rpc to a helloworld server at `localhost:50051`:
+
+```
+$ bins/opt/grpc_cli call localhost:50051 SayHello examples/protos/helloworld.proto \
+    "name: 'world'"  --enable_ssl=false
+```
+
+On success, the tool will print out
+
+```
+Rpc succeeded with OK status
+Response: 
+ message: "Hello world"
+```
+
+The `localhost:50051` part indicates the server you are connecting to. `SayHello` is (part of) the
+gRPC method string. Then there is the path to the proto file containing the service definition,
+if it is not under current directory, you can use `--proto_path` to specify a new search root.
+`"name: 'world'"` is the text format of the request proto message. 
+We are not using ssl here by `--enable_ssl=false`. For information on more
+flags, look at the comments of `grpc_cli.cc`.
+
+### Send non-proto rpc
+
+For using gRPC with protocols other than probobuf, you will need the exact method name string
+and a file containing the raw bytes to be sent on the wire
+
+```
+$ bins/opt/grpc_cli call localhost:50051 /helloworld.Greeter/SayHello --input_binary_file=input.bin \
+    --output_binary_file=output.bin
+```
+On success, you will need to read or decode the response from the `output.bin` file.
diff --git a/doc/connectivity-semantics-and-api.md b/doc/connectivity-semantics-and-api.md
index 5427900394..cc007eaae3 100644
--- a/doc/connectivity-semantics-and-api.md
+++ b/doc/connectivity-semantics-and-api.md
@@ -101,7 +101,7 @@ corresponding reasons. Empty cells denote disallowed transitions.
     <td>Shutdown triggered by application.</td>
   </tr>
   <tr>
-    <th>FATAL_FAILURE</th>
+    <th>SHUTDOWN</th>
     <td></td>
     <td></td>
     <td></td>
diff --git a/doc/cpp-style-guide.md b/doc/cpp-style-guide.md
new file mode 100644
index 0000000000..9408c4abd6
--- /dev/null
+++ b/doc/cpp-style-guide.md
@@ -0,0 +1,85 @@
+GRPC C++ STYLE GUIDE
+=====================
+
+Background
+----------
+
+Here we document style rules for C++ usage in the gRPC C++ bindings
+and tests.
+
+General
+-------
+
+- The majority of gRPC's C++ requirements are drawn from the [Google C++ style
+guide] (https://google.github.io/styleguide/cppguide.html)
+   - However, gRPC has some additional requirements to maintain
+     [portability] (#portability)
+- As in C, layout rules are defined by clang-format, and all code
+should be passed through clang-format. A (docker-based) script to do
+so is included in [tools/distrib/clang\_format\_code.sh]
+(../tools/distrib/clang_format_code.sh).
+
+<a name="portability"></a>
+Portability Restrictions
+-------------------
+
+gRPC supports a large number of compilers, ranging from those that are
+missing many key C++11 features to those that have quite detailed
+analysis. As a result, gRPC compiles with a high level of warnings and
+treat all warnings as errors. gRPC also forbids the use of some common
+C++11 constructs. Here are some guidelines, to be extended as needed:
+- Do not use range-based for. Expressions of the form
+  ```c
+  for (auto& i: vec) {
+    // code
+  }
+  ```
+  
+  are not allowed and should be replaced with code such as
+  ```c
+  for (auto it = vec.begin; it != vec.end(); it++) {
+    auto& i = *it;
+    // code
+  }
+  ```
+  
+- Do not use lambda of any kind (no capture, explicit capture, or
+default capture). Other C++ functional features such as
+`std::function` or `std::bind` are allowed
+- Do not use brace-list initializers.
+- Do not compare a pointer to `nullptr` . This is because gcc 4.4
+  does not support `nullptr` directly and gRPC implements a subset of
+  its features in [include/grpc++/impl/codegen/config.h]
+  (../include/grpc++/impl/codegen/config.h). Instead, pointers should
+  be checked for validity using their implicit conversion to `bool`.
+  In other words, use `if (p)` rather than `if (p != nullptr)`
+- Do not use `final` or `override` as these are not supported by some
+  compilers. Instead use `GRPC_FINAL` and `GRPC_OVERRIDE` . These
+  compile down to the traditional C++ forms for compilers that support
+  them but are just elided if the compiler does not support those features.
+- In the [include] (../../../tree/master/include/grpc++) and [src]
+  (../../../tree/master/src/cpp) directory trees, you should also not
+  use certain STL objects like `std::mutex`, `std::lock_guard`,
+  `std::unique_lock`, `std::nullptr`, `std::thread` . Instead, use
+  `grpc::mutex`, `grpc::lock_guard`, etc., which are gRPC
+  implementations of the prominent features of these objects that are
+  not always available. You can use the `std` versions of those in  [test]
+  (../../../tree/master/test/cpp)
+- Similarly, in the same directories, do not use `std::chrono` unless
+  it is guarded by `#ifndef GRPC_CXX0X_NO_CHRONO` . For platforms that
+  lack`std::chrono,` there is a C-language timer called gpr_timespec that can
+  be used instead.
+- `std::unique_ptr` must be used with extreme care in any kind of
+  collection. For example `vector<std::unique_ptr>` does not work in
+  gcc 4.4 if the vector is constructed to its full size at
+  initialization but does work if elements are added to the vector
+  using functions like `push_back`. `map` and other pair-based
+  collections do not work with `unique_ptr` under gcc 4.4. The issue
+  is that many of these collection implementations assume a copy
+  constructor
+  to be available.
+- Don't use `std::this_thread` . Use `gpr_sleep_until` for sleeping a thread.
+- [Some adjacent character combinations cause problems]
+  (https://en.wikipedia.org/wiki/Digraphs_and_trigraphs#C). If declaring a
+  template against some class relative to the global namespace,
+  `<::name` will be non-portable. Separate the `<` from the `:` and use `< ::name`.
diff --git a/doc/fail_fast.md b/doc/fail_fast.md
new file mode 100644
index 0000000000..3ed4297194
--- /dev/null
+++ b/doc/fail_fast.md
@@ -0,0 +1,15 @@
+gRPC Fail Fast Semantics
+========================
+
+Fail fast requests allow terminating requests (with status UNAVAILABLE) prior
+to the deadline of the request being met.
+
+gRPC implementations of fail fast can terminate requests whenever a channel is
+in the TRANSIENT_FAILURE or SHUTDOWN states. If the channel is in any other
+state (CONNECTING, READY, or IDLE) the request should not be terminated.
+
+Fail fast SHOULD be the default for gRPC implementations, with an option to
+switch to non fail fast.
+
+The opposite of fail fast is 'ignore connectivity'.
+
diff --git a/doc/interop-test-descriptions.md b/doc/interop-test-descriptions.md
index 3beb1d11f4..7fd21c7022 100644
--- a/doc/interop-test-descriptions.md
+++ b/doc/interop-test-descriptions.md
@@ -27,7 +27,7 @@ Clients should accept these arguments:
     * Whether to use a plaintext or encrypted connection
 * --use_test_ca=BOOLEAN
     * Whether to replace platform root CAs with
-      [ca.pem](https://github.com/grpc/grpc/blob/master/src/core/tsi/test_creds/ca.pem)
+      [ca.pem](https://github.com/grpc/grpc/blob/master/src/core/lib/tsi/test_creds/ca.pem)
       as the CA root
 * --default_service_account=ACCOUNT_EMAIL
     * Email of the GCE default service account. Only applicable
@@ -93,26 +93,24 @@ Client asserts:
 ### large_compressed_unary
 
 This test verifies compressed unary calls succeed in sending messages. It
-sends one unary request for every combination of compression algorithm and
-payload type.
+sends one unary request for every payload type, with and without requesting a
+compressed response from the server.
 
 In all scenarios, whether compression was actually performed is determined by
-the compression bit in the response's message flags. The response's compression
-value indicates which algorithm was used if said compression bit is set.
+the compression bit in the response's message flags.
 
 
 Server features:
 * [UnaryCall][]
 * [Compressable Payload][]
 * [Uncompressable Payload][]
-* [Random Payload][]
 
 Procedure:
  1. Client calls UnaryCall with:
 
     ```
     {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
+      request_compressed_response: bool
       response_type: COMPRESSABLE
       response_size: 314159
       payload:{
@@ -123,11 +121,10 @@ Procedure:
     Client asserts:
     * call was successful
     * response payload type is COMPRESSABLE
-    * response compression is consistent with the requested one.
-    * if `response_compression == NONE`, the response MUST NOT have the
+    * if `request_compressed_response` is false, the response MUST NOT have the
+      compressed message flag set.
+    * if `request_compressed_response` is true, the response MUST have the
       compressed message flag set.
-    * if `response_compression != NONE`, the response MUST have the compressed
-      message flag set.
     * response payload body is 314159 bytes in size
     * clients are free to assert that the response payload body contents are
       zero and comparing the entire response message against a golden response
@@ -136,7 +133,7 @@ Procedure:
  2. Client calls UnaryCall with:
     ```
     {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
+      request_compressed_response: bool
       response_type: UNCOMPRESSABLE
       response_size: 314159
       payload:{
@@ -147,29 +144,11 @@ Procedure:
     Client asserts:
     * call was successful
     * response payload type is UNCOMPRESSABLE
-    * response compression is consistent with the requested one.
-    * the response MUST NOT have the compressed message flag set.
+    * the response MAY have the compressed message flag set. Some
+      implementations will choose to compress the payload even when the output
+      size if larger than the input.
     * response payload body is 314159 bytes in size
-    * clients are free to assert that the response payload body contents are
-      identical to the golden uncompressable data at `test/cpp/interop/rnd.dat`.
-
 
- 3. Client calls UnaryCall with:
-    ```
-    {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
-      response_type: RANDOM
-      response_size: 314159
-      payload:{
-        body: 271828 bytes of zeros
-      }
-    }
-    ```
-    Client asserts:
-    * call was successful
-    * response payload type is either COMPRESSABLE or UNCOMPRESSABLE
-    * the behavior is consistent with the randomly chosen incoming payload type,
-      as described in their respective sections.
 
 ### client_streaming
 
@@ -245,7 +224,7 @@ Procedure:
         size: 31415
       }
       response_parameters:{
-        size: 9
+        size: 59
       }
       response_parameters:{
         size: 2653
@@ -272,7 +251,6 @@ Server features:
 * [StreamingOutputCall][]
 * [Compressable Payload][]
 * [Uncompressable Payload][]
-* [Random Payload][]
 
 
 Procedure:
@@ -280,13 +258,13 @@ Procedure:
 
     ```
     {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
+      request_compressed_response: bool
       response_type:COMPRESSABLE
       response_parameters:{
         size: 31415
       }
       response_parameters:{
-        size: 9
+        size: 59
       }
       response_parameters:{
         size: 2653
@@ -301,12 +279,11 @@ Procedure:
     * call was successful
     * exactly four responses
     * response payloads are COMPRESSABLE
-    * response compression is consistent with the requested one.
-    * if `response_compression == NONE`, the response MUST NOT have the
-      compressed message flag set.
-    * if `response_compression != NONE`, the response MUST have the compressed
-      message flag set.
-    * response payload bodies are sized (in order): 31415, 9, 2653, 58979
+    * if `request_compressed_response` is false, the response's messages MUST
+      NOT have the compressed message flag set.
+    * if `request_compressed_response` is true, the response's messages MUST
+      have the compressed message flag set.
+    * response payload bodies are sized (in order): 31415, 59, 2653, 58979
     * clients are free to assert that the response payload body contents are
       zero and comparing the entire response messages against golden responses
 
@@ -315,13 +292,13 @@ Procedure:
 
     ```
     {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
+      request_compressed_response: bool
       response_type:UNCOMPRESSABLE
       response_parameters:{
         size: 31415
       }
       response_parameters:{
-        size: 9
+        size: 59
       }
       response_parameters:{
         size: 2653
@@ -336,40 +313,14 @@ Procedure:
     * call was successful
     * exactly four responses
     * response payloads are UNCOMPRESSABLE
-    * response compressions are consistent with the requested one.
-    * the responses MUST NOT have the compressed message flag set.
-    * response payload bodies are sized (in order): 31415, 9, 2653, 58979
+    * the response MAY have the compressed message flag set. Some
+      implementations will choose to compress the payload even when the output
+      size if larger than the input.
+    * response payload bodies are sized (in order): 31415, 59, 2653, 58979
     * clients are free to assert that the body of the responses are identical to
       the golden uncompressable data at `test/cpp/interop/rnd.dat`.
 
 
- 3. Client calls StreamingOutputCall with:
-
-    ```
-    {
-      response_compression: <one of {NONE, GZIP, DEFLATE}>
-      response_type:RANDOM
-      response_parameters:{
-        size: 31415
-      }
-      response_parameters:{
-        size: 9
-      }
-      response_parameters:{
-        size: 2653
-      }
-      response_parameters:{
-        size: 58979
-      }
-    }
-    ```
-
-    Client asserts:
-    * call was successful
-    * response payload type is either COMPRESSABLE or UNCOMPRESSABLE
-    * the behavior is consistent with the randomly chosen incoming payload type,
-      as described in their respective sections.
-
 ### ping_pong
 
 This test verifies that full duplex bidi is supported.
@@ -399,7 +350,7 @@ Procedure:
     {
       response_type: COMPRESSABLE
       response_parameters:{
-        size: 9
+        size: 59
       }
       payload:{
         body: 8 bytes of zeros
@@ -920,7 +871,7 @@ Servers should accept these arguments:
     * Whether to use a plaintext or encrypted connection
 
 Servers must support TLS with ALPN. They should use
-[server1.pem](https://github.com/grpc/grpc/blob/master/src/core/tsi/test_creds/server1.pem)
+[server1.pem](https://github.com/grpc/grpc/blob/master/src/core/lib/tsi/test_creds/server1.pem)
 for their certificate.
 
 ### EmptyCall
@@ -932,9 +883,9 @@ Server implements EmptyCall which immediately returns the empty message.
 [UnaryCall]: #unarycall
 
 Server implements UnaryCall which immediately returns a SimpleResponse with a
-payload body of size SimpleRequest.response_size bytes and type as appropriate
-for the SimpleRequest.response_type. If the server does not support the
-response_type, then it should fail the RPC with INVALID_ARGUMENT.
+payload body of size `SimpleRequest.response_size` bytes and type as appropriate
+for the `SimpleRequest.response_type`. If the server does not support the
+`response_type`, then it should fail the RPC with `INVALID_ARGUMENT`.
 
 ### StreamingInputCall
 [StreamingInputCall]: #streaminginputcall
@@ -974,15 +925,7 @@ COMPRESSABLE.
 
 When the client requests UNCOMPRESSABLE payload, the response includes a payload
 of the size requested containing uncompressable data and the payload type is
-UNCOMPRESSABLE. A 512 kB dump from /dev/urandom is the current golden data,
-stored at `test/cpp/interop/rnd.dat`
-
-### Random Payload
-[Random Payload]: #random-payload
-
-When the client requests RANDOM payload, the response includes either a randomly
-chosen COMPRESSABLE or UNCOMPRESSABLE payload. The data and the payload type
-will be consistent with this choice.
+UNCOMPRESSABLE.
 
 ### Echo Status
 [Echo Status]: #echo-status
@@ -1004,8 +947,8 @@ key and the corresponding value back to the client as trailing metadata.
 [Observe ResponseParameters.interval_us]: #observe-responseparametersinterval_us
 
 In StreamingOutputCall and FullDuplexCall, server delays sending a
-StreamingOutputCallResponse by the ResponseParameters's interval_us for that
-particular response, relative to the last response sent. That is, interval_us
+StreamingOutputCallResponse by the ResponseParameters's `interval_us` for that
+particular response, relative to the last response sent. That is, `interval_us`
 acts like a sleep *before* sending the response and accumulates from one
 response to the next.
 
@@ -1027,13 +970,13 @@ an email address.
 #### Echo OAuth scope
 [Echo OAuth Scope]: #echo-oauth-scope
 
-If a SimpleRequest has fill_oauth_scope=true and that request was successfully
+If a SimpleRequest has `fill_oauth_scope=true` and that request was successfully
 authenticated via OAuth, then the SimpleResponse should have oauth_scope filled
 with the scope of the method being invoked.
 
 Although a general server-side feature, most test servers won't implement this
-feature. The TLS server grpc-test.sandbox.googleapis.com:443 supports this feature.
-It requires at least the OAuth scope
+feature. The TLS server `grpc-test.sandbox.googleapis.com:443` supports this
+feature. It requires at least the OAuth scope
 `https://www.googleapis.com/auth/xapi.zoo` for authentication to succeed.
 
 Discussion:
diff --git a/doc/statuscodes.md b/doc/statuscodes.md
index 84258c8d12..c918f9ed9a 100644
--- a/doc/statuscodes.md
+++ b/doc/statuscodes.md
@@ -21,6 +21,8 @@ Only a subset of the pre-defined status codes are generated by the gRPC librarie
 | Flow-control protocol violation |	INTERNAL | Both |
 | Error parsing returned status	| UNKNOWN | Client |
 | Incorrect Auth metadata ( Credentials failed to get metadata, Incompatible credentials set on channel and call, Invalid host set in `:authority` metadata, etc.) | UNAUTHENTICATED | Both |
+| Request cardinality violation (method requires exactly one request but client sent some other number of requests) | UNIMPLEMENTED | Server|
+| Response cardinality violation (method requires exactly one response but server sent some other number of responses) | UNIMPLEMENTED | Client|
 | Error parsing response proto	| INTERNAL | Client|
 | Error parsing request proto	| INTERNAL | Server|