diff options
77 files changed, 1137 insertions, 1171 deletions
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 850ea14399..b4e958d3b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,7 +13,7 @@ In order to protect both you and ourselves, you will need to sign the ### Technical requirements You will need several tools to work with this repository. In addition to all of -the packages described in the [INSTALL](INSTALL) file, you will also need +the packages described in the [INSTALL](INSTALL.md) file, you will also need python, and the mako template renderer. To install the latter, using pip, one should simply be able to do `pip install mako`. @@ -21,6 +21,15 @@ In order to run all of the tests we provide, you will need valgrind and clang. More specifically, under debian, you will need the package libc++-dev to properly run all the tests. +Compiling and running grpc C++ tests depend on protobuf 3.0.0, gtest and gflags. +Although gflags is provided in third_party, you will need to manually install +that dependency on your system to run these tests. Under a Debian or Ubuntu +system, you can install the gtests and gflags packages using apt-get: + +```sh + $ [sudo] apt-get install libgflags-dev libgtest-dev +``` + If you are planning to work on any of the languages other than C and C++, you will also need their appropriate development environments. diff --git a/INSTALL b/INSTALL deleted file mode 100644 index e33f8970a9..0000000000 --- a/INSTALL +++ /dev/null @@ -1,217 +0,0 @@ -These instructions only cover building grpc C and C++ libraries under -typical unix systems. If you need more information, please try grpc's -wiki pages: - - https://github.com/google/grpc/wiki - - -************************* -* If you are in a hurry * -************************* - -On Linux (Debian): - - Note: you will need to add the Debian 'jessie-backports' distribution to your sources - file first. - - Add the following line to your `/etc/apt/sources.list` file: - - deb http://http.debian.net/debian jessie-backports main - - Install the gRPC library: - - $ [sudo] apt-get install libgrpc-dev - -OR - - $ git clone https://github.com/grpc/grpc.git - $ cd grpc - $ git submodule update --init - $ make - $ [sudo] make install - -You don't need anything else than GNU Make, gcc and autotools. Under a Debian -or Ubuntu system, this should boil down to the following packages: - - $ [sudo] apt-get install build-essential autoconf libtool - -Building the python wrapper requires the following: - - $ [sudo] apt-get install python-all-dev python-virtualenv - -If you want to install in a different directory than the default /usr/lib, you can -override it on the command line: - - $ [sudo] make install prefix=/opt - - -******************************* -* More detailled instructions * -******************************* - -Setting up dependencies -======================= - -Dependencies to compile the libraries -------------------------------------- - -grpc libraries have few external dependencies. If you need to compile and -install them, they are present in the third_party directory if you have -cloned the github repository recursively. If you didn't clone recursively, -you can still get them later by running the following command: - - $ git submodule update --init - -Note that the Makefile makes it much easier for you to compile from sources -if you were to clone recursively our git repository: it will automatically -compile zlib and OpenSSL, which are core requirements for grpc. Note this -creates grpc libraries that will have zlib and OpenSSL built-in inside of them, -which significantly increases the libraries' size. - -In order to decrease that size, you can manually install zlib and OpenSSL on -your system, so that the Makefile can use them instead. - -Under a Debian or Ubuntu system, one can acquire the development package -for zlib this way: - - # apt-get install zlib1g-dev - -To the best of our knowledge, no distribution has an OpenSSL package that -supports ALPN yet, so you would still have to depend on installing from source -for that particular dependency if you want to reduce the libraries' size. - -The recommended version of OpenSSL that provides ALPN support is available -at this URL: - - https://www.openssl.org/source/openssl-1.0.2.tar.gz - - -Dependencies to compile and run the tests ------------------------------------------ - -Compiling and running grpc plain-C tests dont't require any more dependency. - - -Compiling and running grpc C++ tests depend on protobuf 3.0.0, gtest and -gflags. Although gflags is provided in third_party, you will need to manually -install that dependency on your system to run these tests. - -Under a Debian or Ubuntu system, you can install the gtests and gflags packages -using apt-get: - - # apt-get install libgflags-dev libgtest-dev - -However, protobuf 3.0.0 isn't in a debian package yet, but the Makefile will -automatically try and compile the one present in third_party if you cloned the -repository recursively, and that it detects your system is lacking it. - -Compiling and installing protobuf 3.0.0 requires a few more dependencies in -itself, notably the autoconf suite. If you have apt-get, you can install -these dependencies this way: - - # apt-get install autoconf libtool - -If you want to run the tests using one of the sanitized configurations, you -will need clang and its instrumented libc++: - - # apt-get install clang libc++-dev - -Mac-specific notes: -------------------- - -For a Mac system, git is not available by default. You will first need to -install Xcode from the Mac AppStore and then run the following command from a -terminal: - - $ sudo xcode-select --install - -You should also install "port" following the instructions at -https://www.macports.org . This will reside in /opt/local/bin/port for -most Mac installations. Do the "git submodule" command listed above. - -Then execute the following for all the needed build dependencies - - $ sudo /opt/local/bin/port install autoconf automake libtool gflags cmake - $ mkdir ~/gtest-svn - $ svn checkout http://googletest.googlecode.com/svn/trunk/ gtest-svn - $ mkdir mybuild - $ cd mybuild - $ cmake ../gtest-svn - $ make - $ make gtest.a gtest_main.a - $ sudo cp libgtest.a libgtest_main.a /opt/local/lib - $ sudo mkdir /opt/local/include/gtest - $ sudo cp -pr ../gtest-svn/include/gtest /opt/local/include/gtest - -If you are going to make changes and need to regenerate the projects file, -you will need to install certain modules for python. - - $ sudo easy_install simplejson mako - -Mingw-specific notes: ---------------------- - -While gRPC compiles properly under mingw, some more preparation work is needed. -The recommendation is to use msys2. The installation instructions are available -at that address: http://msys2.github.io/ - -Once this is installed, make sure you are using the following: MinGW-w64 Win64. -You'll be required to install a few more packages: - - $ pacman -S make mingw-w64-x86_64-gcc mingw-w64-x86_64-zlib autoconf automake libtool - -Please also install OpenSSL from that website: - - http://slproweb.com/products/Win32OpenSSL.html - -The package Win64 OpenSSL v1.0.2a should do. At that point you should be able -to compile gRPC with the following: - - $ export LDFLAGS="-L/mingw64/lib -L/c/OpenSSL-Win64" - $ export CPPFLAGS="-I/mingw64/include -I/c/OpenSSL-Win64/include" - $ make - -A word on OpenSSL ------------------ - -Secure HTTP2 requires the TLS extension ALPN (see rfc 7301 and -http://http2.github.io/http2-spec/ section 3.3). Our HTTP2 implementation -relies on OpenSSL's implementation. OpenSSL 1.0.2 is the first released version -of OpenSSL that has ALPN support, and this explains our dependency on it. - -Note that the Makefile supports compiling only the unsecure elements of grpc, -and if you do not have OpenSSL and do not want it, you can still proceed -with installing only the elements you require. However, we strongly recommend -the use of encryption for all network traffic, and discourage the use of grpc -without TLS. - - -Compiling -========= - -If you have all the dependencies mentioned above, you should simply be able -to go ahead and run "make" to compile grpc's C and C++ libraries: - - $ make - - -Testing -======= - -To build and run the tests, you can run the command: - - $ make test - -If you want to be able to run them in parallel, and get better output, you can -also use the python tool we have written: - - $ ./tools/run_tests/run_tests.py - - -Installing -========== - -Once everything is compiled, you should be able to install grpc C and C++ -libraries and headers: - - # make install diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 0000000000..454a8b7b2f --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,57 @@ +#If you are in a hurry + +For language-specific installation instructions for gRPC runtime, please +refer to these documents + + * [C++](examples/cpp): Currently to install gRPC for C++, you need to build from source as described below. + * [C#](src/csharp): NuGet package `Grpc` + * [Go](https://github.com/grpc/grpc-go): `go get google.golang.org/grpc` + * [Java](https://github.com/grpc/grpc-java) + * [Node](src/node): `npm install grpc` + * [Objective-C](src/objective-c) + * [PHP](src/php): `pecl install grpc-beta` + * [Python](src/python/grpcio): `pip install grpcio` + * [Ruby](src/ruby): `gem install grpc` + + +#Pre-requisites + +##Linux + +```sh + $ [sudo] apt-get install build-essential autoconf libtool +``` + +##Mac OSX + +For a Mac system, git is not available by default. You will first need to +install Xcode from the Mac AppStore and then run the following command from a +terminal: + +```sh + $ [sudo] xcode-select --install +``` + +##Protoc + +By default gRPC uses [protocol buffers](https://github.com/google/protobuf), +you will need the `protoc` compiler to generate stub server and client code. + +If you compile gRPC from source, as described below, the Makefile will +automatically try and compile the `protoc` in third_party if you cloned the +repository recursively and it detects that you don't already have it +installed. + + +#Build from Source + +For developers who are interested to contribute, here is how to compile the +gRPC C Core library. + +```sh + $ git clone https://github.com/grpc/grpc.git + $ cd grpc + $ git submodule update --init + $ make + $ [sudo] make install +``` @@ -13,7 +13,7 @@ You can find more detailed documentation and examples in the [doc](doc) and [exa #Installation -See [grpc/INSTALL](INSTALL) for installation instructions for various platforms. +See [INSTALL](INSTALL.md) for installation instructions for various platforms. #Repository Structure & Status diff --git a/doc/interop-test-descriptions.md b/doc/interop-test-descriptions.md index e618e967ee..3beb1d11f4 100644 --- a/doc/interop-test-descriptions.md +++ b/doc/interop-test-descriptions.md @@ -2,9 +2,8 @@ Interoperability Test Case Descriptions ======================================= Client and server use -[test.proto](https://github.com/grpc/grpc/blob/master/test/proto/test.proto) -and the [gRPC over HTTP/2 v2 -protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md). +[test.proto](../src/proto/grpc/testing/test.proto) +and the [gRPC over HTTP/2 v2 protocol](./PROTOCOL-HTTP2.md). Client ------ diff --git a/examples/README.md b/examples/README.md index 84ec80057e..287a80266c 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,450 +1,27 @@ +# Examples -# Getting started +This directory contains code examples for all the C-based gRPC implementations: C++, Node.js, Python, Ruby, Objective-C, PHP, and C#. You can find examples and instructions specific to your +favourite language in the relevant subdirectory. + +Examples for Go and Java gRPC live in their own repositories: -Welcome to the developer documentation for gRPC, a language-neutral, -platform-neutral remote procedure call (RPC) system developed at Google. +* [Java](https://github.com/grpc/grpc-java/tree/master/examples) +* [Android Java](https://github.com/grpc/grpc-java/tree/master/examples/android) +* [Go](https://github.com/grpc/grpc-go/tree/master/examples) -This document introduces you to gRPC with a quick overview and a simple -Hello World example. You'll find more tutorials and reference docs in this repository - more documentation is coming soon! +For more comprehensive documentation, including an [overview](http://www.grpc.io/docs/) and tutorials that use this example code, visit [grpc.io](http://www.grpc.io/docs/). -<a name="quickstart"></a> ## Quick start -You can find quick start guides for each language, including installation instructions, examples, and tutorials here: + +Each example directory has quick start instructions for the appropriate language, including installation instructions and how to run our simplest Hello World example: + * [C++](cpp) -* [Java](https://github.com/grpc/grpc-java/tree/master/examples) -* [Go](https://github.com/grpc/grpc-go/tree/master/examples) * [Ruby](ruby) * [Node.js](node) -* [Android Java](https://github.com/grpc/grpc-java/tree/master/examples/android) * [Python](python/helloworld) * [C#](csharp) * [Objective-C](objective-c/helloworld) * [PHP](php) -## What's in this repository? - -The `examples` directory contains documentation, resources, and examples -for all gRPC users. You can find examples and instructions specific to your -favourite language in the relevant subdirectory. - -You can find out about the gRPC source code repositories in -[`grpc`](https://github.com/grpc/grpc). Each repository provides instructions -for building the appropriate libraries for your language. - - -## What is gRPC? - -In gRPC a *client* application can directly call -methods on a *server* application on a different machine as if it was a -local object, making it easier for you to create distributed applications and -services. As in many RPC systems, gRPC is based around the idea of defining -a *service*, specifying the methods that can be called remotely with their -parameters and return types. On the server side, the server implements this -interface and runs a gRPC server to handle client calls. On the client side, -the client has a *stub* that provides exactly the same methods as the server. - -<!--TODO: diagram--> - -gRPC clients and servers can run and talk to each other in a variety of -environments - from servers inside Google to your own desktop - and can -be written in any of gRPC's [supported languages](#quickstart). So, for -example, you can easily create a gRPC server in Java with clients in Go, -Python, or Ruby. In addition, the latest Google APIs will have gRPC versions -of their interfaces, letting you easily build Google functionality into -your applications. - -<a name="protocolbuffers"></a> -### Working with protocol buffers - -By default gRPC uses *protocol buffers*, Google’s -mature open source mechanism for serializing structured data (although it -can be used with other data formats such as JSON). As you'll -see in our example below, you define gRPC services using *proto files*, -with method parameters and return types specified as protocol buffer message -types. You -can find out lots more about protocol buffers in the [Protocol Buffers -documentation](https://developers.google.com/protocol-buffers/docs/overview). - -#### Protocol buffer versions - -While protocol buffers have been available for open source users for some -time, our examples use a new flavour of protocol buffers called proto3, -which has a slightly simplified syntax, some useful new features, and supports -lots more languages. This is currently available as an alpha release in -Java, C++, Java_nano (Android Java), Python, and Ruby from [the protocol buffers Github -repo](https://github.com/google/protobuf/releases), as well as a Go language -generator from [the golang/protobuf Github repo](https://github.com/golang/protobuf), with more languages in development. You can find out more in the [proto3 language guide](https://developers.google.com/protocol-buffers/docs/proto3), and see -the major differences from the current default version in the [release notes](https://github.com/google/protobuf/releases). More proto3 documentation is coming soon. - -In general, while you *can* use proto2 (the current default protocol buffers version), we recommend that you use proto3 with gRPC as it lets you use the full range of gRPC-supported languages, as well as avoiding compatibility -issues with proto2 clients talking to proto3 servers and vice versa. - -<a name="hello"></a> -## Hello gRPC! - -Now that you know a bit more about gRPC, the easiest way to see how it -works is to look at a simple example. Our Hello World walks you through the -construction of a simple gRPC client-server application, showing you how to: - -- Create a protocol buffers schema that defines a simple RPC service with -a single -Hello World method. -- Create a Java server that implements this interface. -- Create a Java client that accesses the Java server. -- Create a Go client that accesses -the same Java server. - -The complete code for the example is available in the `examples` -directory. We use the Git versioning system for source code management: -however, you don't need to know anything about Git to follow along other -than how to install and run a few git commands. - -This is an introductory example rather than a comprehensive tutorial, so -don't worry if you're not a Go or -Java developer - the concepts are similar for all languages, and you can -find more implementations of our Hello World example in other languages (and full tutorials where available) in -the [language-specific folders](#quickstart) in this repository. Complete tutorials and -reference documentation for all gRPC languages are coming soon. - -<a name="setup"></a> -### Setup - -This section explains how to set up your local machine to work with -the example code. If you just want to read the example, you can go straight -to the [next step](#servicedef). - -#### Install Git - -You can download and install Git from http://git-scm.com/download. Once -installed you should have access to the git command line tool. The main -commands that you will need to use are: - -- git clone ... : clone a remote repository onto your local machine -- git checkout ... : check out a particular branch or a tagged version of -the code to hack on - -#### Install gRPC - -To build and install gRPC plugins and related tools: -- For Java, see the [Java quick start](https://github.com/grpc/grpc-java). -- For Go, see the [Go quick start](https://github.com/grpc/grpc-go). - -#### Get the source code - -The example code for our Java example lives in the `grpc-java` -GitHub repository. Clone this repository to your local machine by running the -following command: - - -``` -git clone https://github.com/grpc/grpc-java.git -``` - -Change your current directory to grpc-java/examples - -``` -cd grpc-java/examples -``` - - - -<a name="servicedef"></a> -### Defining a service - -The first step in creating our example is to define a *service*: an RPC -service specifies the methods that can be called remotely with their parameters -and return types. As you saw in the -[overview](#protocolbuffers) above, gRPC does this using [protocol -buffers](https://developers.google.com/protocol-buffers/docs/overview). We -use the protocol buffers interface definition language (IDL) to define our -service methods, and define the parameters and return -types as protocol buffer message types. Both the client and the -server use interface code generated from the service definition. - -Here's our example service definition, defined using protocol buffers IDL in -[helloworld.proto](https://github.com/grpc/grpc-java/tree/master/examples/src/main/proto). The `Greeter` -service has one method, `SayHello`, that lets the server receive a single -`HelloRequest` -message from the remote client containing the user's name, then send back -a greeting in a single `HelloReply`. This is the simplest type of RPC you -can specify in gRPC - you can find out about other types in the tutorial for your chosen language. - -```proto -syntax = "proto3"; - -option java_package = "io.grpc.examples"; - -package helloworld; - -// The greeter service definition. -service Greeter { - // Sends a greeting - rpc SayHello (HelloRequest) returns (HelloReply) {} -} - -// The request message containing the user's name. -message HelloRequest { - string name = 1; -} - -// The response message containing the greetings -message HelloReply { - string message = 1; -} - -``` - -<a name="generating"></a> -### Generating gRPC code - -Once we've defined our service, we use the protocol buffer compiler -`protoc` to generate the special client and server code we need to create -our application - right now we're going to generate Java code, though you -can generate gRPC code in any gRPC-supported language (as you'll see later -in this example). The generated code contains both stub code for clients to -use and an abstract interface for servers to implement, both with the method -defined in our `Greeter` service. - -(If you didn't install the gRPC plugins and protoc on your system and are just reading along with -the example, you can skip this step and move -onto the next one where we examine the generated code.) - -For simplicity, we've provided a [Gradle build file](https://github.com/grpc/grpc-java/blob/master/examples/build.gradle) with our Java examples that runs `protoc` for you with the appropriate plugin, input, and output: - -```shell -../gradlew build -``` - -This generates the following classes from our .proto, which contain all the generated code -we need to create our example: - -- `Helloworld.java`, which -has all the protocol buffer code to populate, serialize, and retrieve our -`HelloRequest` and `HelloReply` message types -- `GreeterGrpc.java`, which contains (along with some other useful code): - - an interface for `Greeter` servers to implement - - ```java - public static interface Greeter { - public void sayHello(io.grpc.examples.Helloworld.HelloRequest request, - io.grpc.stub.StreamObserver<io.grpc.examples.Helloworld.HelloReply> responseObserver); - } - ``` - - - _stub_ classes that clients can use to talk to a `Greeter` server. As you can see, they also implement the `Greeter` interface. - - ```java - public static class GreeterStub extends - io.grpc.stub.AbstractStub<GreeterStub, GreeterServiceDescriptor> - implements Greeter { - ... - } - ``` - -<a name="server"></a> -### Writing a server - -Now let's write some code! First we'll create a server application to implement -our service. Note that we're not going to go into a lot of detail about how -to create a server in this section. More detailed information will be in the -tutorial for your chosen language: check if there's one available yet in the relevant [quick start](#quickstart). - -Our server application has two classes: - -- a main server class that hosts the service implementation and allows access over the -network: [HelloWorldServer.java](https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java). - - -- a simple service implementation class [GreeterImpl.java](https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java#L51). - - -#### Service implementation - -[GreeterImpl.java](https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java#L51) -actually implements our `Greeter` service's required behaviour. - -As you can see, the class `GreeterImpl` implements the interface -`GreeterGrpc.Greeter` that we [generated](#generating) from our proto -[IDL](https://github.com/grpc/grpc-java/tree/master/examples/src/main/proto) by implementing the method `sayHello`: - -```java - @Override - public void sayHello(HelloRequest req, StreamObserver<HelloReply> responseObserver) { - HelloReply reply = HelloReply.newBuilder().setMessage("Hello " + req.getName()).build(); - responseObserver.onValue(reply); - responseObserver.onCompleted(); - } -``` -- `sayHello` takes two parameters: - - `HelloRequest`: the request - - `StreamObserver<HelloReply>`: a response observer, which is - a special interface for the server to call with its response - -To return our response to the client and complete the call: - -1. We construct and populate a `HelloReply` response object with our exciting -message, as specified in our interface definition. -2. We return the `HelloReply` to the client and then specify that we've finished dealing with the RPC. - - -#### Server implementation - -[HelloWorldServer.java](https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java) -shows the other main feature required to provide a gRPC service; making the service -implementation available from the network. - -```java - /* The port on which the server should run */ - private int port = 50051; - private ServerImpl server; - - private void start() throws Exception { - server = NettyServerBuilder.forPort(port) - .addService(GreeterGrpc.bindService(new GreeterImpl())) - .build().start(); - logger.info("Server started, listening on " + port); - Runtime.getRuntime().addShutdownHook(new Thread() { - @Override - public void run() { - // Use stderr here since the logger may have been reset by its JVM shutdown hook. - System.err.println("*** shutting down gRPC server since JVM is shutting down"); - HelloWorldServer.this.stop(); - System.err.println("*** server shut down"); - } - }); - } - -``` - -Here we create an appropriate gRPC server, binding the `Greeter` service -implementation that we created to a port. Then we start the server running: the server is now ready to receive -requests from `Greeter` service clients on our specified port. We'll cover -how all this works in a bit more detail in our language-specific documentation. - -<a name="client"></a> -### Writing a client - -Client-side gRPC is pretty simple. In this step, we'll use the generated code -to write a simple client that can access the `Greeter` server we created -in the [previous section](#server). You can see the complete client code in -[HelloWorldClient.java](https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldClient.java). - -Again, we're not going to go into much detail about how to implement a client; -we'll leave that for the tutorial. - -#### Connecting to the service - -First let's look at how we connect to the `Greeter` server. First we need -to create a gRPC channel, specifying the hostname and port of the server we -want to connect to. Then we use the channel to construct the stub instance. - - -```java - private final ChannelImpl channel; - private final GreeterGrpc.GreeterBlockingStub blockingStub; - - public HelloWorldClient(String host, int port) { - channel = - NettyChannelBuilder.forAddress(host, port).negotiationType(NegotiationType.PLAINTEXT) - .build(); - blockingStub = GreeterGrpc.newBlockingStub(channel); - } - -``` - -In this case, we create a blocking stub. This means that the RPC call waits -for the server to respond, and will either return a response or raise an -exception. gRPC Java has other kinds of stubs that make non-blocking calls -to the server, where the response is returned asynchronously. - -#### Calling an RPC - -Now we can contact the service and obtain a greeting: - -1. We construct and fill in a `HelloRequest` to send to the service. -2. We call the stub's `hello()` RPC with our request and get a `HelloReply` -back, from which we can get our greeting. - - -```java - HelloRequest req = HelloRequest.newBuilder().setName(name).build(); - HelloReply reply = blockingStub.sayHello(req); - -``` - -<a name="run"></a> -### Try it out! - -Our [Gradle build file](https://github.com/grpc/grpc-java/blob/master/examples/build.gradle) simplifies building and running the examples. - -You can build and run the server from the `grpc-java` root folder with: - -```sh -$ ./gradlew :grpc-examples:helloWorldServer -``` - -and in another terminal window confirm that it receives a message. - -```sh -$ ./gradlew :grpc-examples:helloWorldClient -``` - -### Adding another client - -Finally, let's look at one of gRPC's most useful features - interoperability -between code in different languages. So far, we've just looked at Java code -generated from and implementing our `Greeter` service definition. However, -as you'll see if you look at the language-specific subdirectories -in this repository, we've also generated and implemented `Greeter` -in some of gRPC's other supported languages. Each service -and client uses interface code generated from the same proto -that we used for the Java example. - -So, for example, if we visit the [`go` example -directory](https://github.com/grpc/grpc-go/tree/master/examples) and look at the -[`greeter_client`](https://github.com/grpc/grpc-go/blob/master/examples/greeter_client/main.go), -we can see that like the Java client, it connects to a `Greeter` service -at `localhost:50051` and uses a stub to call the `SayHello` method with a -`HelloRequest`: - -```go -const ( - address = "localhost:50051" - defaultName = "world" -) - -func main() { - // Set up a connection to the server. - conn, err := grpc.Dial(address) - if err != nil { - log.Fatalf("did not connect: %v", err) - } - defer conn.Close() - c := pb.NewGreeterClient(conn) - - // Contact the server and print out its response. - name := defaultName - if len(os.Args) > 1 { - name = os.Args[1] - } - r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: - name}) - if err != nil { - log.Fatalf("could not greet: %v", err) - } - log.Printf("Greeting: %s", r.Message) -} -``` - - -If we run the Java server from earlier in another terminal window, we can -run the Go client and connect to it just like the Java client, even though -it's written in a different language. -``` -$ greeter_client -``` -## Read more! -- You can find links to language-specific tutorials, examples, and other docs in each language's [quick start](#quickstart). -- [gRPC Authentication Support](doc/grpc-auth-support.md) introduces authentication support in gRPC with supported mechanisms and examples. diff --git a/examples/cpp/README.md b/examples/cpp/README.md index 85c495099b..d93cbacf7b 100644 --- a/examples/cpp/README.md +++ b/examples/cpp/README.md @@ -2,7 +2,7 @@ ## Installation -To install gRPC on your system, follow the instructions [here](../../INSTALL). +To install gRPC on your system, follow the instructions to build from source [here](../../INSTALL.md). This also installs the protocol buffer compiler `protoc` (if you don't have it already), and the C++ gRPC plugin for `protoc`. ## Hello C++ gRPC! @@ -23,21 +23,6 @@ Change your current directory to examples/cpp/helloworld $ cd examples/cpp/helloworld/ ``` - -### Generating gRPC code - -To generate the client and server side interfaces: - -```sh -$ make helloworld.grpc.pb.cc helloworld.pb.cc -``` -Which internally invokes the proto-compiler as: - -```sh -$ protoc -I ../../protos/ --grpc_out=. --plugin=protoc-gen-grpc=grpc_cpp_plugin ../../protos/helloworld.proto -$ protoc -I ../../protos/ --cpp_out=. ../../protos/helloworld.proto -``` - ### Client and server implementations The client implementation is at [greeter_client.cc](helloworld/greeter_client.cc). diff --git a/examples/cpp/cpptutorial.md b/examples/cpp/cpptutorial.md index cd1cddb111..ef9ca99c0f 100644 --- a/examples/cpp/cpptutorial.md +++ b/examples/cpp/cpptutorial.md @@ -91,7 +91,7 @@ message Point { Next we need to generate the gRPC client and server interfaces from our .proto service definition. We do this using the protocol buffer compiler `protoc` with a special gRPC C++ plugin. -For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL) first): +For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL.md) first): ```shell $ make route_guide.grpc.pb.cc route_guide.pb.cc diff --git a/examples/cpp/helloworld/README.md b/examples/cpp/helloworld/README.md index 90f3d6b147..8e11f7cdf3 100644 --- a/examples/cpp/helloworld/README.md +++ b/examples/cpp/helloworld/README.md @@ -2,7 +2,7 @@ ### Install gRPC Make sure you have installed gRPC on your system. Follow the instructions here: -[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL). +[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL.md). ### Get the tutorial source code diff --git a/examples/node/README.md b/examples/node/README.md index 7a2bc9794f..c1ef6b05ab 100644 --- a/examples/node/README.md +++ b/examples/node/README.md @@ -20,7 +20,7 @@ TRY IT! - Run the server ```sh - $ # from this directory (grpc_common/node). + $ # from this directory $ node ./greeter_server.js & ``` diff --git a/examples/php/README.md b/examples/php/README.md index 8fb060863a..ea9ccb6790 100644 --- a/examples/php/README.md +++ b/examples/php/README.md @@ -4,16 +4,15 @@ gRPC in 3 minutes (PHP) PREREQUISITES ------------- -This requires PHP 5.5 or greater. +This requires `php` >=5.5, `phpize`, `pecl`, `phpunit` INSTALL ------- - - On Mac OS X, install [homebrew][]. Run the following command to install gRPC. + - Install the gRPC PHP extension ```sh - $ curl -fsSL https://goo.gl/getgrpc | bash -s php + $ [sudo] pecl install grpc-beta ``` - This will download and run the [gRPC install script][] and compile the gRPC PHP extension. - Clone this repository @@ -37,6 +36,7 @@ TRY IT! Please follow the instruction in [Node][] to run the server ``` $ cd examples/node + $ npm install $ nodejs greeter_server.js ``` @@ -58,7 +58,5 @@ TUTORIAL You can find a more detailed tutorial in [gRPC Basics: PHP][] -[homebrew]:http://brew.sh -[gRPC install script]:https://raw.githubusercontent.com/grpc/homebrew-grpc/master/scripts/install [Node]:https://github.com/grpc/grpc/tree/master/examples/node [gRPC Basics: PHP]:http://www.grpc.io/docs/tutorials/basic/php.html diff --git a/examples/python/route_guide/route_guide_server.py b/examples/python/route_guide/route_guide_server.py index f23b98bf36..f95b36b073 100644 --- a/examples/python/route_guide/route_guide_server.py +++ b/examples/python/route_guide/route_guide_server.py @@ -1,4 +1,4 @@ -# Copyright 2015, Google Inc. +# Copyright 2015-2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without @@ -128,7 +128,7 @@ def serve(): while True: time.sleep(_ONE_DAY_IN_SECONDS) except KeyboardInterrupt: - server.stop() + server.stop(0) if __name__ == '__main__': serve() @@ -182,6 +182,7 @@ EXPORTS gpr_event_wait gpr_ref_init gpr_ref + gpr_ref_non_zero gpr_refn gpr_unref gpr_stats_init diff --git a/include/grpc/impl/codegen/sync.h b/include/grpc/impl/codegen/sync.h index d2f19d37d6..6fd7d64b29 100644 --- a/include/grpc/impl/codegen/sync.h +++ b/include/grpc/impl/codegen/sync.h @@ -182,6 +182,10 @@ GPRAPI void gpr_ref_init(gpr_refcount *r, int n); /* Increment the reference count *r. Requires *r initialized. */ GPRAPI void gpr_ref(gpr_refcount *r); +/* Increment the reference count *r. Requires *r initialized. + Crashes if refcount is zero */ +GPRAPI void gpr_ref_non_zero(gpr_refcount *r); + /* Increment the reference count *r by n. Requires *r initialized, n > 0. */ GPRAPI void gpr_refn(gpr_refcount *r, int n); diff --git a/package.xml b/package.xml index 42a0361df0..e65ab73b54 100644 --- a/package.xml +++ b/package.xml @@ -10,11 +10,11 @@ <email>grpc-packages@google.com</email> <active>yes</active> </lead> - <date>2016-02-24</date> + <date>2016-03-01</date> <time>16:06:07</time> <version> - <release>0.8.0</release> - <api>0.8.0</api> + <release>0.14.0</release> + <api>0.14.0</api> </version> <stability> <release>beta</release> @@ -22,7 +22,7 @@ </stability> <license>BSD</license> <notes> -- Simplify gRPC PHP installation #4517 +- Increase unit test code coverage #5225 </notes> <contents> <dir baseinstalldir="/" name="/"> @@ -969,5 +969,20 @@ Update to wrap gRPC C Core version 0.10.0 - Simplify gRPC PHP installation #4517 </notes> </release> + <release> + <version> + <release>0.14.0</release> + <api>0.14.0</api> + </version> + <stability> + <release>beta</release> + <api>beta</api> + </stability> + <date>2016-03-01</date> + <license>BSD</license> + <notes> +- Increase unit test code coverage #5225 + </notes> + </release> </changelog> </package> @@ -154,6 +154,7 @@ INSTALL_REQUIRES = ( SETUP_REQUIRES = ( 'sphinx>=1.3', + 'sphinx_rtd_theme>=0.1.8' ) + INSTALL_REQUIRES COMMAND_CLASS = { @@ -165,6 +166,7 @@ COMMAND_CLASS = { 'build_tagged_ext': precompiled.BuildTaggedExt, 'gather': commands.Gather, 'run_interop': commands.RunInterop, + 'test_lite': commands.TestLite } # Ensure that package data is copied over before any commands have been run: diff --git a/src/core/channel/client_channel.c b/src/core/channel/client_channel.c index eeac3c146c..f021a8ae32 100644 --- a/src/core/channel/client_channel.c +++ b/src/core/channel/client_channel.c @@ -165,7 +165,6 @@ static void cc_on_config_changed(grpc_exec_ctx *exec_ctx, void *arg, channel_data *chand = arg; grpc_lb_policy *lb_policy = NULL; grpc_lb_policy *old_lb_policy; - grpc_resolver *old_resolver; grpc_connectivity_state state = GRPC_CHANNEL_TRANSIENT_FAILURE; int exit_idle = 0; @@ -201,28 +200,25 @@ static void cc_on_config_changed(grpc_exec_ctx *exec_ctx, void *arg, } if (iomgr_success && chand->resolver) { - grpc_resolver *resolver = chand->resolver; - GRPC_RESOLVER_REF(resolver, "channel-next"); grpc_connectivity_state_set(exec_ctx, &chand->state_tracker, state, "new_lb+resolver"); if (lb_policy != NULL) { watch_lb_policy(exec_ctx, chand, lb_policy, state); } - gpr_mu_unlock(&chand->mu_config); GRPC_CHANNEL_STACK_REF(chand->owning_stack, "resolver"); - grpc_resolver_next(exec_ctx, resolver, &chand->incoming_configuration, + grpc_resolver_next(exec_ctx, chand->resolver, + &chand->incoming_configuration, &chand->on_config_changed); - GRPC_RESOLVER_UNREF(exec_ctx, resolver, "channel-next"); + gpr_mu_unlock(&chand->mu_config); } else { - old_resolver = chand->resolver; - chand->resolver = NULL; + if (chand->resolver != NULL) { + grpc_resolver_shutdown(exec_ctx, chand->resolver); + GRPC_RESOLVER_UNREF(exec_ctx, chand->resolver, "channel"); + chand->resolver = NULL; + } grpc_connectivity_state_set(exec_ctx, &chand->state_tracker, GRPC_CHANNEL_FATAL_FAILURE, "resolver_gone"); gpr_mu_unlock(&chand->mu_config); - if (old_resolver != NULL) { - grpc_resolver_shutdown(exec_ctx, old_resolver); - GRPC_RESOLVER_UNREF(exec_ctx, old_resolver, "channel"); - } } if (exit_idle) { @@ -247,11 +243,10 @@ static void cc_start_transport_op(grpc_exec_ctx *exec_ctx, grpc_channel_element *elem, grpc_transport_op *op) { channel_data *chand = elem->channel_data; - grpc_resolver *destroy_resolver = NULL; grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL); - GPR_ASSERT(op->set_accept_stream == NULL); + GPR_ASSERT(op->set_accept_stream == false); if (op->bind_pollset != NULL) { grpc_pollset_set_add_pollset(exec_ctx, chand->interested_parties, op->bind_pollset); @@ -279,7 +274,8 @@ static void cc_start_transport_op(grpc_exec_ctx *exec_ctx, if (op->disconnect && chand->resolver != NULL) { grpc_connectivity_state_set(exec_ctx, &chand->state_tracker, GRPC_CHANNEL_FATAL_FAILURE, "disconnect"); - destroy_resolver = chand->resolver; + grpc_resolver_shutdown(exec_ctx, chand->resolver); + GRPC_RESOLVER_UNREF(exec_ctx, chand->resolver, "channel"); chand->resolver = NULL; if (chand->lb_policy != NULL) { grpc_pollset_set_del_pollset_set(exec_ctx, @@ -290,11 +286,6 @@ static void cc_start_transport_op(grpc_exec_ctx *exec_ctx, } } gpr_mu_unlock(&chand->mu_config); - - if (destroy_resolver) { - grpc_resolver_shutdown(exec_ctx, destroy_resolver); - GRPC_RESOLVER_UNREF(exec_ctx, destroy_resolver, "channel"); - } } typedef struct { diff --git a/src/core/channel/client_uchannel.c b/src/core/channel/client_uchannel.c index b1e7155773..83fcc3a87f 100644 --- a/src/core/channel/client_uchannel.c +++ b/src/core/channel/client_uchannel.c @@ -107,7 +107,7 @@ static void cuc_start_transport_op(grpc_exec_ctx *exec_ctx, grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL); - GPR_ASSERT(op->set_accept_stream == NULL); + GPR_ASSERT(op->set_accept_stream == false); GPR_ASSERT(op->bind_pollset == NULL); if (op->on_connectivity_state_change != NULL) { diff --git a/src/core/channel/subchannel_call_holder.c b/src/core/channel/subchannel_call_holder.c index 81297c8d44..9c087dc2a1 100644 --- a/src/core/channel/subchannel_call_holder.c +++ b/src/core/channel/subchannel_call_holder.c @@ -168,15 +168,15 @@ retry: static void subchannel_ready(grpc_exec_ctx *exec_ctx, void *arg, bool success) { grpc_subchannel_call_holder *holder = arg; - grpc_subchannel_call *call; gpr_mu_lock(&holder->mu); GPR_ASSERT(holder->creation_phase == GRPC_SUBCHANNEL_CALL_HOLDER_PICKING_SUBCHANNEL); - call = GET_CALL(holder); - GPR_ASSERT(call == NULL || call == CANCELLED_CALL); holder->creation_phase = GRPC_SUBCHANNEL_CALL_HOLDER_NOT_CREATING; if (holder->connected_subchannel == NULL) { fail_locked(exec_ctx, holder); + } else if (1 == gpr_atm_acq_load(&holder->subchannel_call)) { + /* already cancelled before subchannel became ready */ + fail_locked(exec_ctx, holder); } else { gpr_atm_rel_store( &holder->subchannel_call, diff --git a/src/core/client_config/subchannel.c b/src/core/client_config/subchannel.c index bec06bf414..d91dd116b8 100644 --- a/src/core/client_config/subchannel.c +++ b/src/core/client_config/subchannel.c @@ -395,7 +395,6 @@ void grpc_subchannel_notify_on_state_change( grpc_exec_ctx *exec_ctx, grpc_subchannel *c, grpc_pollset_set *interested_parties, grpc_connectivity_state *state, grpc_closure *notify) { - int do_connect = 0; external_state_watcher *w; if (state == NULL) { @@ -425,17 +424,13 @@ void grpc_subchannel_notify_on_state_change( w->next->prev = w->prev->next = w; if (grpc_connectivity_state_notify_on_state_change( exec_ctx, &c->state_tracker, state, &w->closure)) { - do_connect = 1; c->connecting = 1; /* released by connection */ GRPC_SUBCHANNEL_WEAK_REF(c, "connecting"); + start_connect(exec_ctx, c); } gpr_mu_unlock(&c->mu); } - - if (do_connect) { - start_connect(exec_ctx, c); - } } void grpc_connected_subchannel_process_transport_op( @@ -635,11 +630,12 @@ static void on_alarm(grpc_exec_ctx *exec_ctx, void *arg, bool iomgr_success) { if (c->disconnected) { iomgr_success = 0; } - gpr_mu_unlock(&c->mu); if (iomgr_success) { update_reconnect_parameters(c); continue_connect(exec_ctx, c); + gpr_mu_unlock(&c->mu); } else { + gpr_mu_unlock(&c->mu); GRPC_SUBCHANNEL_WEAK_UNREF(exec_ctx, c, "connecting"); } } diff --git a/src/core/iomgr/iomgr.c b/src/core/iomgr/iomgr.c index 04580150f3..9c89c2c08a 100644 --- a/src/core/iomgr/iomgr.c +++ b/src/core/iomgr/iomgr.c @@ -41,9 +41,11 @@ #include <grpc/support/string_util.h> #include <grpc/support/sync.h> #include <grpc/support/thd.h> +#include <grpc/support/useful.h> #include "src/core/iomgr/iomgr_internal.h" #include "src/core/iomgr/timer.h" +#include "src/core/support/env.h" #include "src/core/support/string.h" static gpr_mu g_mu; @@ -116,6 +118,9 @@ void grpc_iomgr_shutdown(void) { "memory leaks are likely", count_objects()); dump_objects("LEAKED"); + if (grpc_iomgr_abort_on_leaks()) { + abort(); + } } break; } @@ -154,3 +159,14 @@ void grpc_iomgr_unregister_object(grpc_iomgr_object *obj) { gpr_mu_unlock(&g_mu); gpr_free(obj->name); } + +bool grpc_iomgr_abort_on_leaks(void) { + char *env = gpr_getenv("GRPC_ABORT_ON_LEAKS"); + if (env == NULL) return false; + static const char *truthy[] = {"yes", "Yes", "YES", "true", + "True", "TRUE", "1"}; + for (size_t i = 0; i < GPR_ARRAY_SIZE(truthy); i++) { + if (0 == strcmp(env, truthy[i])) return true; + } + return false; +} diff --git a/src/core/iomgr/iomgr_internal.h b/src/core/iomgr/iomgr_internal.h index e372c18e8a..ac2c46ebe6 100644 --- a/src/core/iomgr/iomgr_internal.h +++ b/src/core/iomgr/iomgr_internal.h @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -34,6 +34,8 @@ #ifndef GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H #define GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H +#include <stdbool.h> + #include "src/core/iomgr/iomgr.h" #include <grpc/support/sync.h> @@ -55,4 +57,6 @@ void grpc_iomgr_platform_flush(void); /** tear down all platform specific global iomgr structures */ void grpc_iomgr_platform_shutdown(void); +bool grpc_iomgr_abort_on_leaks(void); + #endif /* GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H */ diff --git a/src/core/iomgr/pollset_multipoller_with_poll_posix.c b/src/core/iomgr/pollset_multipoller_with_poll_posix.c index 4dddfff230..92d6fb7241 100644 --- a/src/core/iomgr/pollset_multipoller_with_poll_posix.c +++ b/src/core/iomgr/pollset_multipoller_with_poll_posix.c @@ -122,6 +122,7 @@ static void multipoll_with_poll_pollset_maybe_work_and_unlock( } else { h->fds[fd_count++] = h->fds[i]; watchers[pfd_count].fd = h->fds[i]; + GRPC_FD_REF(watchers[pfd_count].fd, "multipoller_start"); pfds[pfd_count].fd = h->fds[i]->fd; pfds[pfd_count].revents = 0; pfd_count++; @@ -135,8 +136,10 @@ static void multipoll_with_poll_pollset_maybe_work_and_unlock( gpr_mu_unlock(&pollset->mu); for (i = 2; i < pfd_count; i++) { - pfds[i].events = (short)grpc_fd_begin_poll(watchers[i].fd, pollset, worker, - POLLIN, POLLOUT, &watchers[i]); + grpc_fd *fd = watchers[i].fd; + pfds[i].events = (short)grpc_fd_begin_poll(fd, pollset, worker, POLLIN, + POLLOUT, &watchers[i]); + GRPC_FD_UNREF(fd, "multipoller_start"); } /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid diff --git a/src/core/iomgr/timer_heap.c b/src/core/iomgr/timer_heap.c index 9d8be5c1fc..b5df566c45 100644 --- a/src/core/iomgr/timer_heap.c +++ b/src/core/iomgr/timer_heap.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -46,7 +46,7 @@ static void adjust_upwards(grpc_timer **first, uint32_t i, grpc_timer *t) { while (i > 0) { uint32_t parent = (uint32_t)(((int)i - 1) / 2); - if (gpr_time_cmp(first[parent]->deadline, t->deadline) >= 0) break; + if (gpr_time_cmp(first[parent]->deadline, t->deadline) <= 0) break; first[i] = first[parent]; first[i]->heap_index = i; i = parent; @@ -62,16 +62,14 @@ static void adjust_downwards(grpc_timer **first, uint32_t i, uint32_t length, grpc_timer *t) { for (;;) { uint32_t left_child = 1u + 2u * i; - uint32_t right_child; - uint32_t next_i; if (left_child >= length) break; - right_child = left_child + 1; - next_i = right_child < length && - gpr_time_cmp(first[left_child]->deadline, - first[right_child]->deadline) < 0 - ? right_child - : left_child; - if (gpr_time_cmp(t->deadline, first[next_i]->deadline) >= 0) break; + uint32_t right_child = left_child + 1; + uint32_t next_i = right_child < length && + gpr_time_cmp(first[left_child]->deadline, + first[right_child]->deadline) > 0 + ? right_child + : left_child; + if (gpr_time_cmp(t->deadline, first[next_i]->deadline) <= 0) break; first[i] = first[next_i]; first[i]->heap_index = i; i = next_i; @@ -95,7 +93,7 @@ static void maybe_shrink(grpc_timer_heap *heap) { static void note_changed_priority(grpc_timer_heap *heap, grpc_timer *timer) { uint32_t i = timer->heap_index; uint32_t parent = (uint32_t)(((int)i - 1) / 2); - if (gpr_time_cmp(heap->timers[parent]->deadline, timer->deadline) < 0) { + if (gpr_time_cmp(heap->timers[parent]->deadline, timer->deadline) > 0) { adjust_upwards(heap->timers, i, timer); } else { adjust_downwards(heap->timers, i, heap->timer_count, timer); diff --git a/src/core/support/sync.c b/src/core/support/sync.c index d368422d9e..69e3e39c5c 100644 --- a/src/core/support/sync.c +++ b/src/core/support/sync.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -98,6 +98,11 @@ void gpr_ref_init(gpr_refcount *r, int n) { gpr_atm_rel_store(&r->count, n); } void gpr_ref(gpr_refcount *r) { gpr_atm_no_barrier_fetch_add(&r->count, 1); } +void gpr_ref_non_zero(gpr_refcount *r) { + gpr_atm prior = gpr_atm_no_barrier_fetch_add(&r->count, 1); + GPR_ASSERT(prior > 0); +} + void gpr_refn(gpr_refcount *r, int n) { gpr_atm_no_barrier_fetch_add(&r->count, n); } diff --git a/src/core/surface/server.c b/src/core/surface/server.c index fb5e0d4b9e..5b13d4ba52 100644 --- a/src/core/surface/server.c +++ b/src/core/surface/server.c @@ -407,8 +407,15 @@ static void destroy_channel(grpc_exec_ctx *exec_ctx, channel_data *chand) { maybe_finish_shutdown(exec_ctx, chand->server); chand->finish_destroy_channel_closure.cb = finish_destroy_channel; chand->finish_destroy_channel_closure.cb_arg = chand; - grpc_exec_ctx_enqueue(exec_ctx, &chand->finish_destroy_channel_closure, true, - NULL); + + grpc_transport_op op; + memset(&op, 0, sizeof(op)); + op.set_accept_stream = true; + op.on_consumed = &chand->finish_destroy_channel_closure; + grpc_channel_next_op(exec_ctx, + grpc_channel_stack_element( + grpc_channel_get_channel_stack(chand->channel), 0), + &op); } static void finish_start_new_rpc(grpc_exec_ctx *exec_ctx, grpc_server *server, @@ -971,7 +978,8 @@ void grpc_server_setup_transport(grpc_exec_ctx *exec_ctx, grpc_server *s, GRPC_CHANNEL_INTERNAL_REF(channel, "connectivity"); memset(&op, 0, sizeof(op)); - op.set_accept_stream = accept_stream; + op.set_accept_stream = true; + op.set_accept_stream_fn = accept_stream; op.set_accept_stream_user_data = chand; op.on_connectivity_state_change = &chand->channel_connectivity_changed; op.connectivity_state = &chand->connectivity_state; diff --git a/src/core/transport/chttp2/internal.h b/src/core/transport/chttp2/internal.h index d76d31be23..b720d1ab3e 100644 --- a/src/core/transport/chttp2/internal.h +++ b/src/core/transport/chttp2/internal.h @@ -358,6 +358,9 @@ struct grpc_chttp2_transport { /** connectivity tracking */ grpc_connectivity_state_tracker state_tracker; } channel_callback; + + /** Transport op to be applied post-parsing */ + grpc_transport_op *post_parsing_op; }; typedef struct { @@ -417,7 +420,7 @@ typedef struct { /** HTTP2 stream id for this stream, or zero if one has not been assigned */ uint32_t id; uint8_t fetching; - uint8_t sent_initial_metadata; + bool sent_initial_metadata; uint8_t sent_message; uint8_t sent_trailing_metadata; uint8_t read_closed; @@ -509,7 +512,7 @@ void grpc_chttp2_publish_reads(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_global *global, grpc_chttp2_transport_parsing *parsing); -void grpc_chttp2_list_add_writable_stream( +bool grpc_chttp2_list_add_writable_stream( grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global); /** Get a writable stream @@ -519,14 +522,13 @@ int grpc_chttp2_list_pop_writable_stream( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_global **stream_global, grpc_chttp2_stream_writing **stream_writing); -void grpc_chttp2_list_remove_writable_stream( +bool grpc_chttp2_list_remove_writable_stream( grpc_chttp2_transport_global *transport_global, - grpc_chttp2_stream_global *stream_global); + grpc_chttp2_stream_global *stream_global) GRPC_MUST_USE_RESULT; -/* returns 1 if stream added, 0 if it was already present */ -int grpc_chttp2_list_add_writing_stream( +void grpc_chttp2_list_add_writing_stream( grpc_chttp2_transport_writing *transport_writing, - grpc_chttp2_stream_writing *stream_writing) GRPC_MUST_USE_RESULT; + grpc_chttp2_stream_writing *stream_writing); int grpc_chttp2_list_have_writing_streams( grpc_chttp2_transport_writing *transport_writing); int grpc_chttp2_list_pop_writing_stream( @@ -770,4 +772,9 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport_parsing *parsing, const uint8_t *opaque_8bytes); +/** add a ref to the stream and add it to the writable list; + ref will be dropped in writing.c */ +void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global, + grpc_chttp2_stream_global *stream_global); + #endif diff --git a/src/core/transport/chttp2/parsing.c b/src/core/transport/chttp2/parsing.c index 8fdebd7f13..0516f39fa9 100644 --- a/src/core/transport/chttp2/parsing.c +++ b/src/core/transport/chttp2/parsing.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -149,7 +149,7 @@ void grpc_chttp2_publish_reads( if (was_zero && !is_zero) { while (grpc_chttp2_list_pop_stalled_by_transport(transport_global, &stream_global)) { - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } @@ -178,7 +178,7 @@ void grpc_chttp2_publish_reads( outgoing_window); is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } stream_global->max_recv_bytes -= (uint32_t)GPR_MIN( diff --git a/src/core/transport/chttp2/stream_lists.c b/src/core/transport/chttp2/stream_lists.c index b284c78818..60fe735cfc 100644 --- a/src/core/transport/chttp2/stream_lists.c +++ b/src/core/transport/chttp2/stream_lists.c @@ -100,11 +100,14 @@ static void stream_list_remove(grpc_chttp2_transport *t, grpc_chttp2_stream *s, } } -static void stream_list_maybe_remove(grpc_chttp2_transport *t, +static bool stream_list_maybe_remove(grpc_chttp2_transport *t, grpc_chttp2_stream *s, grpc_chttp2_stream_list_id id) { if (s->included[id]) { stream_list_remove(t, s, id); + return true; + } else { + return false; } } @@ -125,23 +128,24 @@ static void stream_list_add_tail(grpc_chttp2_transport *t, s->included[id] = 1; } -static int stream_list_add(grpc_chttp2_transport *t, grpc_chttp2_stream *s, - grpc_chttp2_stream_list_id id) { +static bool stream_list_add(grpc_chttp2_transport *t, grpc_chttp2_stream *s, + grpc_chttp2_stream_list_id id) { if (s->included[id]) { - return 0; + return false; } stream_list_add_tail(t, s, id); - return 1; + return true; } /* wrappers for specializations */ -void grpc_chttp2_list_add_writable_stream( +bool grpc_chttp2_list_add_writable_stream( grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global) { GPR_ASSERT(stream_global->id != 0); - stream_list_add(TRANSPORT_FROM_GLOBAL(transport_global), - STREAM_FROM_GLOBAL(stream_global), GRPC_CHTTP2_LIST_WRITABLE); + return stream_list_add(TRANSPORT_FROM_GLOBAL(transport_global), + STREAM_FROM_GLOBAL(stream_global), + GRPC_CHTTP2_LIST_WRITABLE); } int grpc_chttp2_list_pop_writable_stream( @@ -159,20 +163,20 @@ int grpc_chttp2_list_pop_writable_stream( return r; } -void grpc_chttp2_list_remove_writable_stream( +bool grpc_chttp2_list_remove_writable_stream( grpc_chttp2_transport_global *transport_global, grpc_chttp2_stream_global *stream_global) { - stream_list_maybe_remove(TRANSPORT_FROM_GLOBAL(transport_global), - STREAM_FROM_GLOBAL(stream_global), - GRPC_CHTTP2_LIST_WRITABLE); + return stream_list_maybe_remove(TRANSPORT_FROM_GLOBAL(transport_global), + STREAM_FROM_GLOBAL(stream_global), + GRPC_CHTTP2_LIST_WRITABLE); } -int grpc_chttp2_list_add_writing_stream( +void grpc_chttp2_list_add_writing_stream( grpc_chttp2_transport_writing *transport_writing, grpc_chttp2_stream_writing *stream_writing) { - return stream_list_add(TRANSPORT_FROM_WRITING(transport_writing), - STREAM_FROM_WRITING(stream_writing), - GRPC_CHTTP2_LIST_WRITING); + GPR_ASSERT(stream_list_add(TRANSPORT_FROM_WRITING(transport_writing), + STREAM_FROM_WRITING(stream_writing), + GRPC_CHTTP2_LIST_WRITING)); } int grpc_chttp2_list_have_writing_streams( @@ -332,7 +336,7 @@ void grpc_chttp2_list_flush_writing_stalled_by_transport( while (stream_list_pop(transport, &stream, GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) { if (is_window_available) { - grpc_chttp2_list_add_writable_stream(&transport->global, &stream->global); + grpc_chttp2_become_writable(&transport->global, &stream->global); } else { grpc_chttp2_list_add_stalled_by_transport(transport_writing, &stream->writing); diff --git a/src/core/transport/chttp2/writing.c b/src/core/transport/chttp2/writing.c index 356fd8174a..107725cbc7 100644 --- a/src/core/transport/chttp2/writing.c +++ b/src/core/transport/chttp2/writing.c @@ -83,7 +83,8 @@ int grpc_chttp2_unlocking_check_writes( (according to available window sizes) and add to the output buffer */ while (grpc_chttp2_list_pop_writable_stream( transport_global, transport_writing, &stream_global, &stream_writing)) { - uint8_t sent_initial_metadata; + bool sent_initial_metadata = stream_writing->sent_initial_metadata; + bool become_writable = false; stream_writing->id = stream_global->id; stream_writing->read_closed = stream_global->read_closed; @@ -92,16 +93,12 @@ int grpc_chttp2_unlocking_check_writes( outgoing_window, stream_global, outgoing_window); - sent_initial_metadata = stream_writing->sent_initial_metadata; if (!sent_initial_metadata && stream_global->send_initial_metadata) { stream_writing->send_initial_metadata = stream_global->send_initial_metadata; stream_global->send_initial_metadata = NULL; - if (grpc_chttp2_list_add_writing_stream(transport_writing, - stream_writing)) { - GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - } - sent_initial_metadata = 1; + become_writable = true; + sent_initial_metadata = true; } if (sent_initial_metadata) { if (stream_global->send_message != NULL) { @@ -128,10 +125,7 @@ int grpc_chttp2_unlocking_check_writes( stream_writing->flow_controlled_buffer.length > 0) && stream_writing->outgoing_window > 0) { if (transport_writing->outgoing_window > 0) { - if (grpc_chttp2_list_add_writing_stream(transport_writing, - stream_writing)) { - GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - } + become_writable = true; } else { grpc_chttp2_list_add_stalled_by_transport(transport_writing, stream_writing); @@ -141,10 +135,7 @@ int grpc_chttp2_unlocking_check_writes( stream_writing->send_trailing_metadata = stream_global->send_trailing_metadata; stream_global->send_trailing_metadata = NULL; - if (grpc_chttp2_list_add_writing_stream(transport_writing, - stream_writing)) { - GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - } + become_writable = true; } } @@ -153,10 +144,13 @@ int grpc_chttp2_unlocking_check_writes( GRPC_CHTTP2_FLOW_MOVE_STREAM("write", transport_global, stream_writing, announce_window, stream_global, unannounced_incoming_window_for_writing); - if (grpc_chttp2_list_add_writing_stream(transport_writing, - stream_writing)) { - GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); - } + become_writable = true; + } + + if (become_writable) { + grpc_chttp2_list_add_writing_stream(transport_writing, stream_writing); + } else { + GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2_writing"); } } @@ -310,10 +304,7 @@ static void finalize_outbuf(grpc_exec_ctx *exec_ctx, (stream_writing->send_message && !stream_writing->fetching)) && stream_writing->outgoing_window > 0) { if (transport_writing->outgoing_window > 0) { - if (grpc_chttp2_list_add_writing_stream(transport_writing, - stream_writing)) { - /* do nothing - already reffed */ - } + grpc_chttp2_list_add_writing_stream(transport_writing, stream_writing); } else { grpc_chttp2_list_add_writing_stalled_by_transport(transport_writing, stream_writing); diff --git a/src/core/transport/chttp2_transport.c b/src/core/transport/chttp2_transport.c index b9f511e946..19265252ca 100644 --- a/src/core/transport/chttp2_transport.c +++ b/src/core/transport/chttp2_transport.c @@ -142,7 +142,7 @@ static void incoming_byte_stream_update_flow_control( static void fail_pending_writes(grpc_exec_ctx *exec_ctx, grpc_chttp2_stream_global *stream_global); -/* +/******************************************************************************* * CONSTRUCTION/DESTRUCTION/REFCOUNTING */ @@ -432,6 +432,14 @@ static void close_transport_locked(grpc_exec_ctx *exec_ctx, if (t->ep) { allow_endpoint_shutdown_locked(exec_ctx, t); } + + /* flush writable stream list to avoid dangling references */ + grpc_chttp2_stream_global *stream_global; + grpc_chttp2_stream_writing *stream_writing; + while (grpc_chttp2_list_pop_writable_stream( + &t->global, &t->writing, &stream_global, &stream_writing)) { + GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2_writing"); + } } } @@ -521,7 +529,6 @@ static void destroy_stream(grpc_exec_ctx *exec_ctx, grpc_transport *gt, s->global.id) == NULL); } - grpc_chttp2_list_remove_writable_stream(&t->global, &s->global); grpc_chttp2_list_remove_unannounced_incoming_window_available(&t->global, &s->global); grpc_chttp2_list_remove_stalled_by_transport(&t->global, &s->global); @@ -583,7 +590,7 @@ grpc_chttp2_stream_parsing *grpc_chttp2_parsing_accept_stream( return &accepting->parsing; } -/* +/******************************************************************************* * LOCK MANAGEMENT */ @@ -611,10 +618,18 @@ static void unlock(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) { GPR_TIMER_END("unlock", 0); } -/* +/******************************************************************************* * OUTPUT PROCESSING */ +void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global, + grpc_chttp2_stream_global *stream_global) { + if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed && + grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) { + GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing"); + } +} + static void push_setting(grpc_chttp2_transport *t, grpc_chttp2_setting_id id, uint32_t value) { const grpc_chttp2_setting_parameters *sp = @@ -732,7 +747,7 @@ static void maybe_start_some_streams( stream_global->id, STREAM_FROM_GLOBAL(stream_global)); stream_global->in_stream_map = 1; transport_global->concurrent_stream_count++; - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } /* cancel out streams that will never be started */ while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID && @@ -821,7 +836,7 @@ static void perform_stream_op_locked( maybe_start_some_streams(exec_ctx, transport_global); } else { GPR_ASSERT(stream_global->id != 0); - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } else { grpc_chttp2_complete_closure_step( @@ -838,7 +853,7 @@ static void perform_stream_op_locked( exec_ctx, &stream_global->send_message_finished, 0); } else if (stream_global->id != 0) { stream_global->send_message = op->send_message; - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } @@ -858,7 +873,7 @@ static void perform_stream_op_locked( } else if (stream_global->id != 0) { /* TODO(ctiller): check if there's flow control for any outstanding bytes before going writable */ - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } @@ -944,12 +959,10 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx, unlock(exec_ctx, t); } -static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt, - grpc_transport_op *op) { - grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt; - int close_transport = 0; - - lock(t); +static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx, + grpc_chttp2_transport *t, + grpc_transport_op *op) { + bool close_transport = false; grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL); @@ -968,8 +981,8 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt, close_transport = !grpc_chttp2_has_streams(t); } - if (op->set_accept_stream != NULL) { - t->channel_callback.accept_stream = op->set_accept_stream; + if (op->set_accept_stream) { + t->channel_callback.accept_stream = op->set_accept_stream_fn; t->channel_callback.accept_stream_user_data = op->set_accept_stream_user_data; } @@ -990,16 +1003,31 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt, close_transport_locked(exec_ctx, t); } - unlock(exec_ctx, t); - if (close_transport) { - lock(t); close_transport_locked(exec_ctx, t); - unlock(exec_ctx, t); } } -/* +static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt, + grpc_transport_op *op) { + grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt; + + lock(t); + + /* If there's a set_accept_stream ensure that we're not parsing + to avoid changing things out from underneath */ + if (t->parsing_active && op->set_accept_stream) { + GPR_ASSERT(t->post_parsing_op == NULL); + t->post_parsing_op = gpr_malloc(sizeof(*op)); + memcpy(t->post_parsing_op, op, sizeof(*op)); + } else { + perform_transport_op_locked(exec_ctx, t, op); + } + + unlock(exec_ctx, t); +} + +/******************************************************************************* * INPUT PROCESSING */ @@ -1064,7 +1092,6 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, if (!s) { s = grpc_chttp2_stream_map_delete(&t->new_stream_map, id); } - grpc_chttp2_list_remove_writable_stream(&t->global, &s->global); GPR_ASSERT(s); s->global.in_stream_map = 0; if (t->parsing.incoming_stream == &s->parsing) { @@ -1080,6 +1107,9 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t, if (grpc_chttp2_unregister_stream(t, s) && t->global.sent_goaway) { close_transport_locked(exec_ctx, t); } + if (grpc_chttp2_list_remove_writable_stream(&t->global, &s->global)) { + GRPC_CHTTP2_STREAM_UNREF(exec_ctx, &s->global, "chttp2_writing"); + } new_stream_count = grpc_chttp2_stream_map_size(&t->parsing_stream_map) + grpc_chttp2_stream_map_size(&t->new_stream_map); @@ -1331,7 +1361,7 @@ static void update_global_window(void *args, uint32_t id, void *stream) { is_zero = stream_global->outgoing_window <= 0; if (was_zero && !is_zero) { - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } @@ -1392,6 +1422,13 @@ static void recv_data(grpc_exec_ctx *exec_ctx, void *tp, bool success) { /* handle higher level things */ grpc_chttp2_publish_reads(exec_ctx, transport_global, transport_parsing); t->parsing_active = 0; + /* handle delayed transport ops (if there is one) */ + if (t->post_parsing_op) { + grpc_transport_op *op = t->post_parsing_op; + t->post_parsing_op = NULL; + perform_transport_op_locked(exec_ctx, t, op); + gpr_free(op); + } /* if a stream is in the stream map, and gets cancelled, we need to ensure * we are not parsing before continuing the cancellation to keep things in * a sane state */ @@ -1426,7 +1463,7 @@ static void recv_data(grpc_exec_ctx *exec_ctx, void *tp, bool success) { GPR_TIMER_END("recv_data", 0); } -/* +/******************************************************************************* * CALLBACK LOOP */ @@ -1440,7 +1477,7 @@ static void connectivity_state_set( state, reason); } -/* +/******************************************************************************* * POLLSET STUFF */ @@ -1468,7 +1505,7 @@ static void set_pollset(grpc_exec_ctx *exec_ctx, grpc_transport *gt, unlock(exec_ctx, t); } -/* +/******************************************************************************* * BYTE STREAM */ @@ -1508,7 +1545,7 @@ static void incoming_byte_stream_update_flow_control( add_max_recv_bytes); grpc_chttp2_list_add_unannounced_incoming_window_available(transport_global, stream_global); - grpc_chttp2_list_add_writable_stream(transport_global, stream_global); + grpc_chttp2_become_writable(transport_global, stream_global); } } @@ -1623,7 +1660,7 @@ grpc_chttp2_incoming_byte_stream *grpc_chttp2_incoming_byte_stream_create( return incoming_byte_stream; } -/* +/******************************************************************************* * TRACING */ @@ -1709,7 +1746,7 @@ void grpc_chttp2_flowctl_trace(const char *file, int line, const char *phase, gpr_free(prefix); } -/* +/******************************************************************************* * INTEGRATION GLUE */ diff --git a/src/core/transport/metadata.c b/src/core/transport/metadata.c index 14912af7df..807ae071a3 100644 --- a/src/core/transport/metadata.c +++ b/src/core/transport/metadata.c @@ -43,11 +43,13 @@ #include <grpc/support/log.h> #include <grpc/support/string_util.h> #include <grpc/support/time.h> + #include "src/core/profiling/timers.h" #include "src/core/support/murmur_hash.h" #include "src/core/support/string.h" #include "src/core/transport/chttp2/bin_encoder.h" #include "src/core/transport/static_metadata.h" +#include "src/core/iomgr/iomgr_internal.h" /* There are two kinds of mdelem and mdstr instances. * Static instances are declared in static_metadata.{h,c} and @@ -227,6 +229,9 @@ void grpc_mdctx_global_shutdown(void) { if (shard->count != 0) { gpr_log(GPR_DEBUG, "WARNING: %d metadata elements were leaked", shard->count); + if (grpc_iomgr_abort_on_leaks()) { + abort(); + } } gpr_free(shard->elems); } @@ -237,6 +242,9 @@ void grpc_mdctx_global_shutdown(void) { if (shard->count != 0) { gpr_log(GPR_DEBUG, "WARNING: %d metadata strings were leaked", shard->count); + if (grpc_iomgr_abort_on_leaks()) { + abort(); + } } gpr_free(shard->strs); } diff --git a/src/core/transport/transport.c b/src/core/transport/transport.c index 6e154b629a..3b555fa933 100644 --- a/src/core/transport/transport.c +++ b/src/core/transport/transport.c @@ -45,7 +45,7 @@ void grpc_stream_ref(grpc_stream_refcount *refcount, const char *reason) { #else void grpc_stream_ref(grpc_stream_refcount *refcount) { #endif - gpr_ref(&refcount->refs); + gpr_ref_non_zero(&refcount->refs); } #ifdef GRPC_STREAM_REFCOUNT_DEBUG diff --git a/src/core/transport/transport.h b/src/core/transport/transport.h index 8902c5d2f6..ed6e121c9c 100644 --- a/src/core/transport/transport.h +++ b/src/core/transport/transport.h @@ -123,7 +123,7 @@ typedef struct grpc_transport_stream_op { /** Transport op: a set of operations to perform on a transport as a whole */ typedef struct grpc_transport_op { - /** called when processing of this op is done */ + /** Called when processing of this op is done. */ grpc_closure *on_consumed; /** connectivity monitoring - set connectivity_state to NULL to unsubscribe */ grpc_closure *on_connectivity_state_change; @@ -138,9 +138,13 @@ typedef struct grpc_transport_op { grpc_status_code goaway_status; gpr_slice *goaway_message; /** set the callback for accepting new streams; - this is a permanent callback, unlike the other one-shot closures */ - void (*set_accept_stream)(grpc_exec_ctx *exec_ctx, void *user_data, - grpc_transport *transport, const void *server_data); + this is a permanent callback, unlike the other one-shot closures. + If true, the callback is set to set_accept_stream_fn, with its + user_data argument set to set_accept_stream_user_data */ + bool set_accept_stream; + void (*set_accept_stream_fn)(grpc_exec_ctx *exec_ctx, void *user_data, + grpc_transport *transport, + const void *server_data); void *set_accept_stream_user_data; /** add this transport to a pollset */ grpc_pollset *bind_pollset; diff --git a/src/cpp/README.md b/src/cpp/README.md index baeba08315..83d37aa2ed 100644 --- a/src/cpp/README.md +++ b/src/cpp/README.md @@ -6,3 +6,77 @@ This directory contains source code for C++ implementation of gRPC. #Status Beta + +#Pre-requisites + +##Linux + +```sh + $ [sudo] apt-get install build-essential autoconf libtool +``` + +##Mac OSX + +For a Mac system, git is not available by default. You will first need to +install Xcode from the Mac AppStore and then run the following command from a +terminal: + +```sh + $ [sudo] xcode-select --install +``` + +##Protoc + +By default gRPC uses [protocol buffers](https://github.com/google/protobuf), +you will need the `protoc` compiler to generate stub server and client code. + +If you compile gRPC from source, as described below, this also installs the +`protoc` compiler. + +If it hasn't been installed, you can run the following commands to install it. + +```sh +$ cd grpc/third_party/protobuf +$ sudo make install # 'make' should have been run by core grpc +``` + +Alternatively, you can download `protoc` binaries from +[the protocol buffers Github repository](https://github.com/google/protobuf/releases). + +#Installation + +Currently to install gRPC for C++, you need to build from source as described +below. + +#Build from Source + +```sh + $ git clone https://github.com/grpc/grpc.git + $ cd grpc + $ git submodule update --init + $ make + $ [sudo] make install +``` + +#Documentation + +You can find out how to build and run our simplest gRPC C++ example in our +[C++ quick start](https://github.com/grpc/grpc/tree/{{ site.data.config.grpc_release_branch }}/examples/cpp). + +For more detailed documentation on using gRPC in C++ , see our main +documentation site at [grpc.io](http://grpc.io), specifically: + +* [Overview](http://www.grpc.io/docs/): An introduction to gRPC with a simple + Hello World example in all our supported languages, including C++. +* [gRPC Basics - C++](http://www.grpc.io/docs/tutorials/basic/c.html): + A tutorial that steps you through creating a simple gRPC C++ example + application. +* [Asynchronous Basics - C++](http://www.grpc.io/docs/tutorials/async/helloasync-cpp.html): + A tutorial that shows you how to use gRPC C++'s asynchronous/non-blocking + APIs. + + +# Examples + +Code examples for gRPC C++ live in this repository's +[examples/cpp](https://github.com/grpc/grpc/tree/{{ site.data.config.grpc_release_branch }}/examples/cpp) directory. diff --git a/src/csharp/Grpc.Core/Internal/AsyncCallServer.cs b/src/csharp/Grpc.Core/Internal/AsyncCallServer.cs index b72cbd795f..9380c0d0ea 100644 --- a/src/csharp/Grpc.Core/Internal/AsyncCallServer.cs +++ b/src/csharp/Grpc.Core/Internal/AsyncCallServer.cs @@ -193,16 +193,6 @@ namespace Grpc.Core.Internal lock (myLock) { finished = true; - - if (cancelled) - { - // Once we cancel, we don't have to care that much - // about reads and writes. - - // TODO(jtattermusch): is this still necessary? - Cancel(); - } - ReleaseResourcesIfPossible(); } // TODO(jtattermusch): handle error diff --git a/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs b/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs index 18168f9970..0d12c4168c 100644 --- a/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs +++ b/src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs @@ -1,6 +1,6 @@ #region Copyright notice and license -// Copyright 2015, Google Inc. +// Copyright 2015-2016, Google Inc. // All rights reserved. // // Redistribution and use in source and binary forms, with or without @@ -140,14 +140,12 @@ namespace Grpc.IntegrationTesting } [Test] - [Ignore("TODO: see #4427")] public async Task StatusCodeAndMessage() { await InteropClient.RunStatusCodeAndMessageAsync(client); } [Test] - [Ignore("TODO: see #4427")] public void UnimplementedMethod() { InteropClient.RunUnimplementedMethod(UnimplementedService.NewClient(channel)); diff --git a/src/node/README.md b/src/node/README.md index b46b986243..3501b54a66 100644 --- a/src/node/README.md +++ b/src/node/README.md @@ -5,7 +5,7 @@ Beta ## PREREQUISITES -- `node`: This requires `node` to be installed. If you instead have the `nodejs` executable on Debian, you should install the [`nodejs-legacy`](https://packages.debian.org/sid/nodejs-legacy) package. +- `node`: This requires `node` to be installed, version `0.12` or above. If you instead have the `nodejs` executable on Debian, you should install the [`nodejs-legacy`](https://packages.debian.org/sid/nodejs-legacy) package. ## INSTALLATION diff --git a/src/node/interop/interop_client.js b/src/node/interop/interop_client.js index db383e4d00..5602011a8e 100644 --- a/src/node/interop/interop_client.js +++ b/src/node/interop/interop_client.js @@ -290,6 +290,7 @@ function timeoutOnSleepingServer(client, done) { call.write({ payload: {body: zeroBuffer(27182)} }); + call.on('data', function() {}); call.on('error', function(error) { assert(error.code === grpc.status.DEADLINE_EXCEEDED || @@ -336,6 +337,7 @@ function customMetadata(client, done) { ['test_initial_metadata_value']); done(); }); + stream.on('data', function() {}); stream.on('status', function(status) { var echo_trailer = status.metadata.get(ECHO_TRAILING_KEY); assert(echo_trailer.length > 0); @@ -361,6 +363,7 @@ function statusCodeAndMessage(client, done) { done(); }); var duplex = client.fullDuplexCall(); + duplex.on('data', function() {}); duplex.on('status', function(status) { assert(status); assert.strictEqual(status.code, 2); diff --git a/src/node/src/client.js b/src/node/src/client.js index c65dd73650..9cc5a62bdb 100644 --- a/src/node/src/client.js +++ b/src/node/src/client.js @@ -131,9 +131,69 @@ function ClientReadableStream(call, deserialize) { this.finished = false; this.reading = false; this.deserialize = common.wrapIgnoreNull(deserialize); + /* Status generated from reading messages from the server. Overrides the + * status from the server if not OK */ + this.read_status = null; + /* Status received from the server. */ + this.received_status = null; } /** + * Called when all messages from the server have been processed. The status + * parameter indicates that the call should end with that status. status + * defaults to OK if not provided. + * @param {Object!} status The status that the call should end with + */ +function _readsDone(status) { + /* jshint validthis: true */ + if (!status) { + status = {code: grpc.status.OK, details: 'OK'}; + } + this.finished = true; + this.read_status = status; + this._emitStatusIfDone(); +} + +ClientReadableStream.prototype._readsDone = _readsDone; + +/** + * Called to indicate that we have received a status from the server. + */ +function _receiveStatus(status) { + /* jshint validthis: true */ + this.received_status = status; + this._emitStatusIfDone(); +} + +ClientReadableStream.prototype._receiveStatus = _receiveStatus; + +/** + * If we have both processed all incoming messages and received the status from + * the server, emit the status. Otherwise, do nothing. + */ +function _emitStatusIfDone() { + /* jshint validthis: true */ + var status; + if (this.read_status && this.received_status) { + if (this.read_status.code !== grpc.status.OK) { + status = this.read_status; + } else { + status = this.received_status; + } + this.emit('status', status); + if (status.code !== grpc.status.OK) { + var error = new Error(status.details); + error.code = status.code; + error.metadata = status.metadata; + this.emit('error', error); + return; + } + } +} + +ClientReadableStream.prototype._emitStatusIfDone = _emitStatusIfDone; + +/** * Read the next object from the stream. * @access private * @param {*} size Ignored because we use objectMode=true @@ -150,6 +210,7 @@ function _read(size) { if (err) { // Something has gone wrong. Stop reading and wait for status self.finished = true; + self._readsDone(); return; } var data = event.read; @@ -157,8 +218,11 @@ function _read(size) { try { deserialized = self.deserialize(data); } catch (e) { - self.call.cancelWithStatus(grpc.status.INTERNAL, - 'Failed to parse server response'); + self._readsDone({code: grpc.status.INTERNAL, + details: 'Failed to parse server response'}); + } + if (data === null) { + self._readsDone(); } if (self.push(deserialized) && data !== null) { var read_batch = {}; @@ -198,6 +262,11 @@ function ClientDuplexStream(call, serialize, deserialize) { this.serialize = common.wrapIgnoreNull(serialize); this.deserialize = common.wrapIgnoreNull(deserialize); this.call = call; + /* Status generated from reading messages from the server. Overrides the + * status from the server if not OK */ + this.read_status = null; + /* Status received from the server. */ + this.received_status = null; this.on('finish', function() { var batch = {}; batch[grpc.opType.SEND_CLOSE_FROM_CLIENT] = true; @@ -205,6 +274,9 @@ function ClientDuplexStream(call, serialize, deserialize) { }); } +ClientDuplexStream.prototype._readsDone = _readsDone; +ClientDuplexStream.prototype._receiveStatus = _receiveStatus; +ClientDuplexStream.prototype._emitStatusIfDone = _emitStatusIfDone; ClientDuplexStream.prototype._read = _read; ClientDuplexStream.prototype._write = _write; @@ -336,7 +408,7 @@ function makeUnaryRequestFunction(method, serialize, deserialize) { } } if (status.code !== grpc.status.OK) { - error = new Error(response.status.details); + error = new Error(status.details); error.code = status.code; error.metadata = status.metadata; callback(error); @@ -487,22 +559,13 @@ function makeServerStreamRequestFunction(method, serialize, deserialize) { var status_batch = {}; status_batch[grpc.opType.RECV_STATUS_ON_CLIENT] = true; call.startBatch(status_batch, function(err, response) { - response.status.metadata = Metadata._fromCoreRepresentation( - response.status.metadata); - stream.emit('status', response.status); - if (response.status.code !== grpc.status.OK) { - var error = new Error(response.status.details); - error.code = response.status.code; - error.metadata = response.status.metadata; - stream.emit('error', error); + if (err) { + stream.emit('error', err); return; - } else { - if (err) { - // Got a batch error, but OK status. Something went wrong - stream.emit('error', err); - return; - } } + response.status.metadata = Metadata._fromCoreRepresentation( + response.status.metadata); + stream._receiveStatus(response.status); }); return stream; } @@ -552,22 +615,13 @@ function makeBidiStreamRequestFunction(method, serialize, deserialize) { var status_batch = {}; status_batch[grpc.opType.RECV_STATUS_ON_CLIENT] = true; call.startBatch(status_batch, function(err, response) { - response.status.metadata = Metadata._fromCoreRepresentation( - response.status.metadata); - stream.emit('status', response.status); - if (response.status.code !== grpc.status.OK) { - var error = new Error(response.status.details); - error.code = response.status.code; - error.metadata = response.status.metadata; - stream.emit('error', error); + if (err) { + stream.emit('error', err); return; - } else { - if (err) { - // Got a batch error, but OK status. Something went wrong - stream.emit('error', err); - return; - } } + response.status.metadata = Metadata._fromCoreRepresentation( + response.status.metadata); + stream._receiveStatus(response.status); }); return stream; } diff --git a/src/node/test/surface_test.js b/src/node/test/surface_test.js index 530f1f7749..8a232d6fc4 100644 --- a/src/node/test/surface_test.js +++ b/src/node/test/surface_test.js @@ -1000,6 +1000,7 @@ describe('Call propagation', function() { proxy_impl.serverStream = function(parent) { var child = client.serverStream(parent.request, null, {parent: parent}); + child.on('data', function() {}); child.on('error', function(err) { assert(err); assert.strictEqual(err.code, grpc.status.CANCELLED); @@ -1013,6 +1014,7 @@ describe('Call propagation', function() { var proxy_client = new Client('localhost:' + proxy_port, grpc.credentials.createInsecure()); call = proxy_client.serverStream({}); + call.on('data', function() {}); call.on('error', function(err) { done(); }); @@ -1022,6 +1024,7 @@ describe('Call propagation', function() { var call; proxy_impl.bidiStream = function(parent) { var child = client.bidiStream(null, {parent: parent}); + child.on('data', function() {}); child.on('error', function(err) { assert(err); assert.strictEqual(err.code, grpc.status.CANCELLED); @@ -1035,6 +1038,7 @@ describe('Call propagation', function() { var proxy_client = new Client('localhost:' + proxy_port, grpc.credentials.createInsecure()); call = proxy_client.bidiStream(); + call.on('data', function() {}); call.on('error', function(err) { done(); }); @@ -1074,6 +1078,7 @@ describe('Call propagation', function() { proxy_impl.bidiStream = function(parent) { var child = client.bidiStream( null, {parent: parent, propagate_flags: deadline_flags}); + child.on('data', function() {}); child.on('error', function(err) { assert(err); assert(err.code === grpc.status.DEADLINE_EXCEEDED || @@ -1089,6 +1094,7 @@ describe('Call propagation', function() { var deadline = new Date(); deadline.setSeconds(deadline.getSeconds() + 1); var call = proxy_client.bidiStream(null, {deadline: deadline}); + call.on('data', function() {}); call.on('error', function(err) { done(); }); @@ -1130,6 +1136,7 @@ describe('Cancelling surface client', function() { }); it('Should correctly cancel a server stream call', function(done) { var call = client.fib({'limit': 5}); + call.on('data', function() {}); call.on('error', function(error) { assert.strictEqual(error.code, surface_client.status.CANCELLED); done(); @@ -1138,6 +1145,7 @@ describe('Cancelling surface client', function() { }); it('Should correctly cancel a bidi stream call', function(done) { var call = client.divMany(); + call.on('data', function() {}); call.on('error', function(error) { assert.strictEqual(error.code, surface_client.status.CANCELLED); done(); diff --git a/src/objective-c/GRPCClient/private/GRPCCompletionQueue.h b/src/objective-c/GRPCClient/private/GRPCCompletionQueue.h index fe3b8f39d1..7b66cd4c32 100644 --- a/src/objective-c/GRPCClient/private/GRPCCompletionQueue.h +++ b/src/objective-c/GRPCClient/private/GRPCCompletionQueue.h @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without diff --git a/src/objective-c/GRPCClient/private/GRPCCompletionQueue.m b/src/objective-c/GRPCClient/private/GRPCCompletionQueue.m index ea2b01ee1d..ff3031678c 100644 --- a/src/objective-c/GRPCClient/private/GRPCCompletionQueue.m +++ b/src/objective-c/GRPCClient/private/GRPCCompletionQueue.m @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without diff --git a/src/php/README.md b/src/php/README.md index b1823b9226..cf8f2c11b0 100644 --- a/src/php/README.md +++ b/src/php/README.md @@ -9,14 +9,21 @@ Beta ## Environment -Prerequisite: PHP 5.5 or later, `phpunit`, `pecl` +Prerequisite: `php` >=5.5, `phpize`, `pecl`, `phpunit` -**Linux:** +**Linux (Debian):** ```sh $ sudo apt-get install php5 php5-dev php-pear ``` +**Linux (CentOS):** + +```sh +$ yum install php55w +$ yum --enablerepo=remi,remi-php55 install php-devel php-pear +``` + **Mac OS X:** ```sh @@ -24,55 +31,29 @@ $ curl -O http://pear.php.net/go-pear.phar $ sudo php -d detect_unicode=0 go-pear.phar ``` -**PHPUnit: (Both Linux and Mac OS X)** +**PHPUnit:** ```sh -$ curl https://phar.phpunit.de/phpunit.phar -o phpunit.phar -$ chmod +x phpunit.phar -$ sudo mv phpunit.phar /usr/local/bin/phpunit +$ wget https://phar.phpunit.de/phpunit-old.phar +$ chmod +x phpunit-old.phar +$ sudo mv phpunit-old.phar /usr/bin/phpunit ``` ## Quick Install -**Linux (Debian):** - -Add [Debian jessie-backports][] to your `sources.list` file. Example: - -```sh -echo "deb http://http.debian.net/debian jessie-backports main" | \ -sudo tee -a /etc/apt/sources.list -``` - -Install the gRPC Debian package - -```sh -sudo apt-get update -sudo apt-get install libgrpc-dev -``` - Install the gRPC PHP extension ```sh sudo pecl install grpc-beta ``` -**Mac OS X:** - -Install [homebrew][]. Example: - -```sh -ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" -``` +This will compile and install the gRPC PHP extension into the standard PHP extension directory. You should be able to run the [unit tests](#unit-tests), with the PHP extension installed. -Install the gRPC core library and the PHP extension in one step +To run tests with generated stub code from `.proto` files, you will also need the `composer`, `protoc` and `protoc-gen-php` binaries. You can find out how to get these [below](#generated-code-tests). -```sh -$ curl -fsSL https://goo.gl/getgrpc | bash -s php -``` - -This will download and run the [gRPC install script][] and compile the gRPC PHP extension. +## Build from Source -## Build from Source +### gRPC C core library Clone this repository @@ -80,7 +61,7 @@ Clone this repository $ git clone https://github.com/grpc/grpc.git ``` -Build and install the gRPC C core libraries +Build and install the gRPC C core library ```sh $ cd grpc @@ -89,20 +70,15 @@ $ make $ sudo make install ``` -Note: you may encounter a warning about the Protobuf compiler `protoc` 3.0.0+ not being installed. The following might help, and will be useful later on when we need to compile the `protoc-gen-php` tool. +### gRPC PHP extension -```sh -$ cd grpc/third_party/protobuf -$ sudo make install # 'make' should have been run by core grpc -``` - -Install the gRPC PHP extension +Install the gRPC PHP extension from PECL ```sh $ sudo pecl install grpc-beta ``` -OR +Or, compile from source ```sh $ cd grpc/src/php/ext/grpc @@ -112,58 +88,98 @@ $ make $ sudo make install ``` +### Update php.ini + Add this line to your `php.ini` file, e.g. `/etc/php5/cli/php.ini` ```sh extension=grpc.so ``` -Install Composer +## Unit Tests + +You will need the source code to run tests + +```sh +$ git clone https://github.com/grpc/grpc.git +$ cd grpc +$ git pull --recurse-submodules && git submodule update --init --recursive +``` + +Run unit tests ```sh $ cd grpc/src/php +$ ./bin/run_tests.sh +``` + +## Generated Code Tests + +This section specifies the prerequisites for running the generated code tests, as well as how to run the tests themselves. + +### Composer + +If you don't have it already, install `composer` to pull in some runtime dependencies based on the `composer.json` file. + +```sh $ curl -sS https://getcomposer.org/installer | php $ sudo mv composer.phar /usr/local/bin/composer + +$ cd grpc/src/php $ composer install ``` -## Unit Tests +### Protobuf compiler -Run unit tests +Again if you don't have it already, you need to install the protobuf compiler `protoc`, version 3.0.0+. + +If you compiled the gRPC C core library from source above, the `protoc` binary should have been installed as well. If it hasn't been installed, you can run the following commands to install it. ```sh -$ cd grpc/src/php -$ ./bin/run_tests.sh +$ cd grpc/third_party/protobuf +$ sudo make install # 'make' should have been run by core grpc ``` -## Generated Code Tests +Alternatively, you can download `protoc` binaries from [the protocol buffers Github repository](https://github.com/google/protobuf/releases). + -Install `protoc-gen-php` +### PHP protobuf compiler + +You need to install `protoc-gen-php` to generate stub class `.php` files from service definition `.proto` files. ```sh -$ cd grpc/src/php/vendor/datto/protobuf-php +$ cd grpc/src/php/vendor/datto/protobuf-php # if you had run `composer install` in the previous step + +OR + +$ git clone https://github.com/stanley-cheung/Protobuf-PHP # clone from github repo + $ gem install rake ronn $ rake pear:package version=1.0 $ sudo pear install Protobuf-1.0.tgz ``` -Generate client stub code +### Client Stub + +Generate client stub classes from `.proto` files ```sh $ cd grpc/src/php $ ./bin/generate_proto_php.sh ``` -Run a local server serving the math services +### Run test server - - Please see [Node][] on how to run an example server +Run a local server serving the math services. Please see [Node][] for how to run an example server. ```sh -$ cd grpc/src/node +$ cd grpc $ npm install -$ nodejs examples/math_server.js +$ nodejs src/node/test/math/math_server.js ``` +### Run test client + Run the generated code tests ```sh @@ -194,13 +210,15 @@ $ sudo service apache2 restart Make sure the Node math server is still running, as above. ```sh -$ cd grpc/src/node -$ nodejs examples/math_server.js +$ cd grpc +$ npm install +$ nodejs src/node/test/math/math_server.js ``` Make sure you have run `composer install` to generate the `vendor/autoload.php` file ```sh +$ cd grpc/src/php $ composer install ``` @@ -262,13 +280,15 @@ $ sudo service php5-fpm restart Make sure the Node math server is still running, as above. ```sh -$ cd grpc/src/node -$ nodejs examples/math_server.js +$ cd grpc +$ npm install +$ nodejs src/node/test/math/math_server.js ``` Make sure you have run `composer install` to generate the `vendor/autoload.php` file ```sh +$ cd grpc/src/php $ composer install ``` @@ -297,7 +317,4 @@ Connect to `localhost/math_client.php` in your browser, or run this from command $ curl localhost/math_client.php ``` -[homebrew]:http://brew.sh -[gRPC install script]:https://raw.githubusercontent.com/grpc/homebrew-grpc/master/scripts/install [Node]:https://github.com/grpc/grpc/tree/master/src/node/examples -[Debian jessie-backports]:http://backports.debian.org/Instructions/ diff --git a/src/python/grpcio/README.rst b/src/python/grpcio/README.rst index 698c760ebe..3dfae50b4b 100644 --- a/src/python/grpcio/README.rst +++ b/src/python/grpcio/README.rst @@ -6,7 +6,7 @@ Package for gRPC Python. Installation ------------ -gRPC Python is available for Linux and Mac OS X running Python 2.7. +gRPC Python is available for Linux, Mac OS X, and Windows running Python 2.7. From PyPI ~~~~~~~~~ @@ -23,11 +23,15 @@ Else system wide (on Ubuntu)... $ sudo pip install grpcio +n.b. On Windows and on Mac OS X one *must* have a recent release of :code:`pip` +to retrieve the proper wheel from PyPI. Be sure to upgrade to the latest +version! + From Source ~~~~~~~~~~~ Building from source requires that you have the Python headers (usually a -package named `python-dev`). +package named :code:`python-dev`). :: @@ -36,5 +40,19 @@ package named `python-dev`). $ cd $REPO_ROOT $ pip install . -Note that `$REPO_ROOT` can be assigned to whatever directory name floats your -fancy. +Note that :code:`$REPO_ROOT` can be assigned to whatever directory name floats +your fancy. + +Troubleshooting +~~~~~~~~~~~~~~~ + +Help, I ... + +* **... see a** :code:`pkg_resources.VersionConflict` **when I try to install + grpc!** + + This is likely because :code:`pip` doesn't own the offending dependency, + which in turn is likely because your operating system's package manager owns + it. You'll need to force the installation of the dependency: + + :code:`pip install --ignore-installed $OFFENDING_DEPENDENCY` diff --git a/src/python/grpcio/commands.py b/src/python/grpcio/commands.py index aa29c728f2..c1f444f6f1 100644 --- a/src/python/grpcio/commands.py +++ b/src/python/grpcio/commands.py @@ -119,8 +119,7 @@ class SphinxDocumentation(setuptools.Command): import sphinx import sphinx.apidoc metadata = self.distribution.metadata - src_dir = os.path.join( - PYTHON_STEM, self.distribution.package_dir[''], 'grpc') + src_dir = os.path.join(PYTHON_STEM, 'grpc') sys.path.append(src_dir) sphinx.apidoc.main([ '', '--force', '--full', '-H', metadata.name, '-A', metadata.author, @@ -264,6 +263,42 @@ class Gather(setuptools.Command): self.distribution.fetch_build_eggs(self.distribution.tests_require) +class TestLite(setuptools.Command): + """Command to run tests without fetching or building anything.""" + + description = 'run tests without fetching or building anything.' + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + # distutils requires this override. + pass + + def run(self): + self._add_eggs_to_path() + + import tests + loader = tests.Loader() + loader.loadTestsFromNames(['tests']) + runner = tests.Runner() + result = runner.run(loader.suite) + if not result.wasSuccessful(): + sys.exit('Test failure') + + def _add_eggs_to_path(self): + """Adds all egg files under .eggs to sys.path""" + # TODO(jtattemusch): there has to be a cleaner way to do this + import pkg_resources + eggs_dir = os.path.join(PYTHON_STEM, '../../../.eggs') + eggs = [os.path.join(eggs_dir, filename) + for filename in os.listdir(eggs_dir) + if filename.endswith('.egg')] + for egg in eggs: + sys.path.insert(0, pkg_resources.normalize_path(egg)) + + class RunInterop(test.test): description = 'run interop test client/server' diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pxd.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pxd.pxi index 757f1245e8..305475c006 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pxd.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pxd.pxi @@ -1,4 +1,4 @@ -# Copyright 2015, Google Inc. +# Copyright 2015-2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without @@ -31,8 +31,9 @@ cdef class CompletionQueue: cdef grpc_completion_queue *c_completion_queue - cdef object poll_condition - cdef bint is_polling + cdef object pluck_condition + cdef int num_plucking + cdef int num_polling cdef bint is_shutting_down cdef bint is_shutdown diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi index b299dfee22..e11138b1cd 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi @@ -39,8 +39,9 @@ cdef class CompletionQueue: self.c_completion_queue = grpc_completion_queue_create(NULL) self.is_shutting_down = False self.is_shutdown = False - self.poll_condition = threading.Condition() - self.is_polling = False + self.pluck_condition = threading.Condition() + self.num_plucking = 0 + self.num_polling = 0 cdef _interpret_event(self, grpc_event event): cdef OperationTag tag = None @@ -87,19 +88,15 @@ cdef class CompletionQueue: c_deadline = deadline.c_time cdef grpc_event event - # Poll within a critical section - # TODO(atash) consider making queue polling contention a hard error to - # enable easier bug discovery - with self.poll_condition: - while self.is_polling: - self.poll_condition.wait(float(deadline) - time.time()) - self.is_polling = True + # Poll within a critical section to detect contention + with self.pluck_condition: + assert self.num_plucking == 0, 'cannot simultaneously pluck and poll' + self.num_polling += 1 with nogil: event = grpc_completion_queue_next( self.c_completion_queue, c_deadline, NULL) - with self.poll_condition: - self.is_polling = False - self.poll_condition.notify() + with self.pluck_condition: + self.num_polling -= 1 return self._interpret_event(event) def pluck(self, OperationTag tag, Timespec deadline=None): @@ -111,19 +108,18 @@ cdef class CompletionQueue: c_deadline = deadline.c_time cdef grpc_event event - # Poll within a critical section - # TODO(atash) consider making queue polling contention a hard error to - # enable easier bug discovery - with self.poll_condition: - while self.is_polling: - self.poll_condition.wait(float(deadline) - time.time()) - self.is_polling = True + # Pluck within a critical section to detect contention + with self.pluck_condition: + assert self.num_polling == 0, 'cannot simultaneously pluck and poll' + assert self.num_plucking < GRPC_MAX_COMPLETION_QUEUE_PLUCKERS, ( + 'cannot pluck more than {} times simultaneously'.format( + GRPC_MAX_COMPLETION_QUEUE_PLUCKERS)) + self.num_plucking += 1 with nogil: event = grpc_completion_queue_pluck( self.c_completion_queue, <cpython.PyObject *>tag, c_deadline, NULL) - with self.poll_condition: - self.is_polling = False - self.poll_condition.notify() + with self.pluck_condition: + self.num_plucking -= 1 return self._interpret_event(event) def shutdown(self): @@ -144,7 +140,8 @@ cdef class CompletionQueue: grpc_completion_queue_shutdown(self.c_completion_queue) # Pump the queue while not self.is_shutdown: - event = grpc_completion_queue_next( - self.c_completion_queue, c_deadline, NULL) + with nogil: + event = grpc_completion_queue_next( + self.c_completion_queue, c_deadline, NULL) self._interpret_event(event) grpc_completion_queue_destroy(self.c_completion_queue) diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi index 800d0ea2f6..dbf0045710 100644 --- a/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi +++ b/src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi @@ -138,6 +138,8 @@ cdef extern from "grpc/_cython/loader.h": const int GRPC_WRITE_NO_COMPRESS const int GRPC_WRITE_USED_MASK + const int GRPC_MAX_COMPLETION_QUEUE_PLUCKERS + ctypedef struct grpc_completion_queue: # We don't care about the internals (and in fact don't know them) pass diff --git a/src/python/grpcio/grpc/_cython/imports.generated.c b/src/python/grpcio/grpc/_cython/imports.generated.c index 4b1860ce8c..8bd6ae6372 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.c +++ b/src/python/grpcio/grpc/_cython/imports.generated.c @@ -220,6 +220,7 @@ gpr_event_get_type gpr_event_get_import; gpr_event_wait_type gpr_event_wait_import; gpr_ref_init_type gpr_ref_init_import; gpr_ref_type gpr_ref_import; +gpr_ref_non_zero_type gpr_ref_non_zero_import; gpr_refn_type gpr_refn_import; gpr_unref_type gpr_unref_import; gpr_stats_init_type gpr_stats_init_import; @@ -485,6 +486,7 @@ void pygrpc_load_imports(HMODULE library) { gpr_event_wait_import = (gpr_event_wait_type) GetProcAddress(library, "gpr_event_wait"); gpr_ref_init_import = (gpr_ref_init_type) GetProcAddress(library, "gpr_ref_init"); gpr_ref_import = (gpr_ref_type) GetProcAddress(library, "gpr_ref"); + gpr_ref_non_zero_import = (gpr_ref_non_zero_type) GetProcAddress(library, "gpr_ref_non_zero"); gpr_refn_import = (gpr_refn_type) GetProcAddress(library, "gpr_refn"); gpr_unref_import = (gpr_unref_type) GetProcAddress(library, "gpr_unref"); gpr_stats_init_import = (gpr_stats_init_type) GetProcAddress(library, "gpr_stats_init"); diff --git a/src/python/grpcio/grpc/_cython/imports.generated.h b/src/python/grpcio/grpc/_cython/imports.generated.h index a395dce7d6..b70dcccd17 100644 --- a/src/python/grpcio/grpc/_cython/imports.generated.h +++ b/src/python/grpcio/grpc/_cython/imports.generated.h @@ -610,6 +610,9 @@ extern gpr_ref_init_type gpr_ref_init_import; typedef void(*gpr_ref_type)(gpr_refcount *r); extern gpr_ref_type gpr_ref_import; #define gpr_ref gpr_ref_import +typedef void(*gpr_ref_non_zero_type)(gpr_refcount *r); +extern gpr_ref_non_zero_type gpr_ref_non_zero_import; +#define gpr_ref_non_zero gpr_ref_non_zero_import typedef void(*gpr_refn_type)(gpr_refcount *r, int n); extern gpr_refn_type gpr_refn_import; #define gpr_refn gpr_refn_import diff --git a/src/python/grpcio/precompiled.py b/src/python/grpcio/precompiled.py index ae2a0c835a..d34250b02c 100644 --- a/src/python/grpcio/precompiled.py +++ b/src/python/grpcio/precompiled.py @@ -31,6 +31,7 @@ import os import platform import shutil import sys +import sysconfig import setuptools @@ -51,9 +52,15 @@ USE_PRECOMPILED_BINARIES = bool(int(os.environ.get( def _tagged_ext_name(base): uname = platform.uname() - tags = '-'.join((grpc_version.VERSION, uname[0], uname[4])) - flavor = 'ucs2' if sys.maxunicode == 65535 else 'ucs4' - return '{base}-{tags}-{flavor}'.format(base=base, tags=tags, flavor=flavor) + tags = ( + grpc_version.VERSION, + 'py{}'.format(sysconfig.get_python_version()), + uname[0], + uname[4], + ) + ucs = 'ucs{}'.format(sysconfig.get_config_var('Py_UNICODE_SIZE')) + return '{base}-{tags}-{ucs}'.format( + base=base, tags='-'.join(tags), ucs=ucs) class BuildTaggedExt(setuptools.Command): diff --git a/src/python/grpcio/tests/_runner.py b/src/python/grpcio/tests/_runner.py index 4f1ddb57fc..b0dbd92a49 100644 --- a/src/python/grpcio/tests/_runner.py +++ b/src/python/grpcio/tests/_runner.py @@ -1,4 +1,4 @@ -# Copyright 2015, Google Inc. +# Copyright 2015-2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without @@ -143,10 +143,17 @@ class Runner(object): def run(self, suite): """See setuptools' test_runner setup argument for information.""" + # only run test cases with id starting with given prefix + testcase_filter = os.getenv('GRPC_PYTHON_TESTRUNNER_FILTER') + filtered_cases = [] + for case in _loader.iterate_suite_cases(suite): + if not testcase_filter or case.id().startswith(testcase_filter): + filtered_cases.append(case) + # Ensure that every test case has no collision with any other test case in # the augmented results. augmented_cases = [AugmentedCase(case, uuid.uuid4()) - for case in _loader.iterate_suite_cases(suite)] + for case in filtered_cases] case_id_by_case = dict((augmented_case.case, augmented_case.id) for augmented_case in augmented_cases) result_out = StringIO.StringIO() diff --git a/src/python/grpcio/tests/tests.json b/src/python/grpcio/tests/tests.json new file mode 100644 index 0000000000..388d040d5c --- /dev/null +++ b/src/python/grpcio/tests/tests.json @@ -0,0 +1,62 @@ +[ + "_base_interface_test.AsyncEasyTest", + "_base_interface_test.AsyncPeasyTest", + "_base_interface_test.SyncEasyTest", + "_base_interface_test.SyncPeasyTest", + "_beta_features_test.BetaFeaturesTest", + "_beta_features_test.ContextManagementAndLifecycleTest", + "_channel_test.ChannelTest", + "_connectivity_channel_test.ChannelConnectivityTest", + "_core_over_links_base_interface_test.AsyncEasyTest", + "_core_over_links_base_interface_test.AsyncPeasyTest", + "_core_over_links_base_interface_test.SyncEasyTest", + "_core_over_links_base_interface_test.SyncPeasyTest", + "_crust_over_core_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", + "_crust_over_core_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", + "_crust_over_core_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_over_links_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_over_links_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", + "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", + "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", + "_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", + "_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", + "_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", + "_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", + "_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", + "_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", + "_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", + "_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", + "_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", + "_implementations_test.ChannelCredentialsTest", + "_insecure_interop_test.InsecureInteropTest", + "_intermediary_low_test.CancellationTest", + "_intermediary_low_test.EchoTest", + "_intermediary_low_test.ExpirationTest", + "_intermediary_low_test.LonelyClientTest", + "_later_test.LaterTest", + "_logging_pool_test.LoggingPoolTest", + "_lonely_invocation_link_test.LonelyInvocationLinkTest", + "_low_test.HangingServerShutdown", + "_low_test.InsecureServerInsecureClient", + "_not_found_test.NotFoundTest", + "_sanity_test.Sanity", + "_secure_interop_test.SecureInteropTest", + "_transmission_test.RoundTripTest", + "_transmission_test.TransmissionTest", + "_utilities_test.ChannelConnectivityTest", + "beta_python_plugin_test.PythonPluginTest", + "cygrpc_test.InsecureServerInsecureClient", + "cygrpc_test.SecureServerSecureClient", + "cygrpc_test.TypeSmokeTest" +]
\ No newline at end of file diff --git a/src/python/grpcio/tests/unit/_sanity/__init__.py b/src/python/grpcio/tests/unit/_sanity/__init__.py new file mode 100644 index 0000000000..2f88fa0412 --- /dev/null +++ b/src/python/grpcio/tests/unit/_sanity/__init__.py @@ -0,0 +1,30 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + diff --git a/src/python/grpcio/tests/unit/_sanity/_sanity_test.py b/src/python/grpcio/tests/unit/_sanity/_sanity_test.py new file mode 100644 index 0000000000..0a5a715c0e --- /dev/null +++ b/src/python/grpcio/tests/unit/_sanity/_sanity_test.py @@ -0,0 +1,53 @@ +# Copyright 2016, Google Inc. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following disclaimer +# in the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Google Inc. nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import json +import unittest + +import tests + + +class Sanity(unittest.TestCase): + + def testTestsJsonUpToDate(self): + """Autodiscovers all test suites and checks that tests.json is up to date""" + loader = tests.Loader() + loader.loadTestsFromNames(['tests']) + test_suite_names = [ + test_case_class.id().rsplit('.', 1)[0] + for test_case_class in tests._loader.iterate_suite_cases(loader.suite)] + test_suite_names = sorted(set(test_suite_names)) + + with open('src/python/grpcio/tests/tests.json') as tests_json_file: + tests_json = json.load(tests_json_file) + self.assertListEqual(test_suite_names, tests_json) + + +if __name__ == '__main__': + unittest.main(verbosity=2) diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.c b/src/ruby/ext/grpc/rb_grpc_imports.generated.c index 1af34d97fb..56db4ec686 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.c +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.c @@ -220,6 +220,7 @@ gpr_event_get_type gpr_event_get_import; gpr_event_wait_type gpr_event_wait_import; gpr_ref_init_type gpr_ref_init_import; gpr_ref_type gpr_ref_import; +gpr_ref_non_zero_type gpr_ref_non_zero_import; gpr_refn_type gpr_refn_import; gpr_unref_type gpr_unref_import; gpr_stats_init_type gpr_stats_init_import; @@ -481,6 +482,7 @@ void grpc_rb_load_imports(HMODULE library) { gpr_event_wait_import = (gpr_event_wait_type) GetProcAddress(library, "gpr_event_wait"); gpr_ref_init_import = (gpr_ref_init_type) GetProcAddress(library, "gpr_ref_init"); gpr_ref_import = (gpr_ref_type) GetProcAddress(library, "gpr_ref"); + gpr_ref_non_zero_import = (gpr_ref_non_zero_type) GetProcAddress(library, "gpr_ref_non_zero"); gpr_refn_import = (gpr_refn_type) GetProcAddress(library, "gpr_refn"); gpr_unref_import = (gpr_unref_type) GetProcAddress(library, "gpr_unref"); gpr_stats_init_import = (gpr_stats_init_type) GetProcAddress(library, "gpr_stats_init"); diff --git a/src/ruby/ext/grpc/rb_grpc_imports.generated.h b/src/ruby/ext/grpc/rb_grpc_imports.generated.h index 38aabfaca8..b972f60fc3 100644 --- a/src/ruby/ext/grpc/rb_grpc_imports.generated.h +++ b/src/ruby/ext/grpc/rb_grpc_imports.generated.h @@ -610,6 +610,9 @@ extern gpr_ref_init_type gpr_ref_init_import; typedef void(*gpr_ref_type)(gpr_refcount *r); extern gpr_ref_type gpr_ref_import; #define gpr_ref gpr_ref_import +typedef void(*gpr_ref_non_zero_type)(gpr_refcount *r); +extern gpr_ref_non_zero_type gpr_ref_non_zero_import; +#define gpr_ref_non_zero gpr_ref_non_zero_import typedef void(*gpr_refn_type)(gpr_refcount *r, int n); extern gpr_refn_type gpr_refn_import; #define gpr_refn gpr_refn_import diff --git a/summerofcode/ideas.md b/summerofcode/ideas.md index c7f368e7cc..83f2cecd48 100644 --- a/summerofcode/ideas.md +++ b/summerofcode/ideas.md @@ -19,7 +19,9 @@ gRPC C Core: 1. Port gRPC to one of the major BSD platforms ([FreeBSD](https://freebsd.org), [NetBSD](https://netbsd.org), and [OpenBSD](https://openbsd.org)) and create packages for them. Add [kqueue](https://www.freebsd.org/cgi/man.cgi?query=kqueue) support in the process. * **Required skills:** C programming language, BSD operating system. - * **Likely mentors:** [Craig Tiller](https://github.com/ctiller), [Nicolas Noble](https://github.com/nicolasnoble). + * **Likely mentors:** [Craig Tiller](https://github.com/ctiller), + [Nicolas Noble](https://github.com/nicolasnoble), + [Vijay Pai](https://github.com/vjpai). 1. Fix gRPC C-core's URI parser. The current parser does not qualify as a standard parser according to [RFC3986]( https://tools.ietf.org/html/rfc3986). Write test suites to verify this and make changes necessary to make the URI parser compliant. * **Required skills:** C programming language, HTTP standard compliance. * **Likely mentors:** [Craig Tiller](https://github.com/ctiller). diff --git a/templates/README.md b/templates/README.md index 6740972cfb..eedc6e9c09 100644 --- a/templates/README.md +++ b/templates/README.md @@ -6,78 +6,92 @@ was going to single handedly cover all of our usage cases. So instead we decided to work the following way: -* A build.json file at the root is the source of truth for listing all of the -target and files needed to build grpc and its tests, as well as basic system -dependencies description. +* A `build.yaml` file at the root is the source of truth for listing all the +targets and files needed to build grpc and its tests, as well as a basic system +for dependency description. * Each project file (Makefile, Visual Studio project files, Bazel's BUILD) is -a plain-text template that uses the build.json file to generate the final -output file. +a [YAML](http://yaml.org) file used by the `build.yaml` file to generate the +final output file. This way we can maintain as many project system as we see fit, without having to manually maintain them when we add or remove new code to the repository. Only the structure of the project file is relevant to the template. The actual list of source code and targets isn't. -We currently have template files for GNU Make, Visual Studio 2010 to 2015, -and Bazel. In the future, we would like to expand to generating gyp or cmake -project files (or potentially both), XCode project files, and an Android.mk -file to be able to compile gRPC using Android's NDK. +We currently have template files for GNU Make, Visual Studio 2013, +[Bazel](http://bazel.io) and [gyp](https://gyp.gsrc.io/) (albeit only for +Node.js). In the future, we +would like to expand to also generate [cmake](https://cmake.org) +project files, XCode project files, and an Android.mk file allowing to compile +gRPC using Android's NDK. We'll gladly accept contribution that'd create additional project files using that system. -# Structure of build.json +# Structure of `build.yaml` -The build.json file has the following structure: +The `build.yaml` file has the following structure: ``` -{ - "settings": { ... }, # global settings, such as version number - "filegroups": [ ... ], # groups of file that is automatically expanded - "libs": [ ... ], # list of libraries to build - "targets": [ ... ], # list of targets to build -} +settings: # global settings, such as version number + ... +filegroups: # groups of files that are automatically expanded + ... +libs: # list of libraries to build + ... +target: # list of targets to build + ... ``` The `filegroups` are helpful to re-use a subset of files in multiple targets. One `filegroups` entry has the following structure: ``` -{ - "name": "arbitrary string", # the name of the filegroup - "public_headers": [ ... ], # list of public headers defined in that filegroup - "headers": [ ... ], # list of headers defined in that filegroup - "src": [ ... ], # list of source files defined in that filegroup -} +- name: "arbitrary string", # the name of the filegroup + public_headers: # list of public headers defined in that filegroup + - ... + headers: # list of headers defined in that filegroup + - ... + src: # list of source files defined in that filegroup + - ... ``` -The `libs` array contains the list of all the libraries we describe. Some may be +The `libs` collection contains the list of all the libraries we describe. Some may be helper libraries for the tests. Some may be installable libraries. Some may be helper libraries for installable binaries. The `targets` array contains the list of all the binary targets we describe. Some may be installable binaries. -One `libs` or `targets` entry has the following structure: +One `libs` or `targets` entry has the following structure (see below for +details): ``` -{ - "name": "arbitrary string", # the name of the library - "build": "build type", # in which situation we want that library to be - # built and potentially installed - "language": "...", # the language tag; "c" or "c++" - "public_headers": [ ... ], # list of public headers to install - "headers": [ ... ], # list of headers used by that target - "src": [ ... ], # list of files to compile - "secure": "...", # "yes", "no" or "check" - "baselib": boolean, # this is a low level library that has system - # dependencies - "vs_project_guid: "...", # Visual Studio's unique guid for that project - "filegroups": [ ... ], # list of filegroups to merge to that project - # note that this will be expanded automatically - "deps": [ ... ], # list of libraries this target depends on -} +name: "arbitrary string", # the name of the library +build: "build type", # in which situation we want that library to be + # built and potentially installed (see below). +language: "...", # the language tag; "c" or "c++" +public_headers: # list of public headers to install +headers: # list of headers used by that target +src: # list of files to compile +secure: boolean, # see below +baselib: boolean, # this is a low level library that has system + # dependencies +vs_project_guid: '{...}', # Visual Studio's unique guid for that project +filegroups: # list of filegroups to merge to that project + # note that this will be expanded automatically +deps: # list of libraries this target depends on +deps_linkage: "..." # "static" or "dynamic". Used by the Makefile only to + # determine the way dependencies are linkned. Defaults + # to "dynamic". +dll: "..." # see below. +dll_def: "..." # Visual Studio's dll definition file. +vs_props: # List of property sheets to attach to that project. +vs_config_type: "..." # DynamicLibrary/StaticLibrary. Used only when + # creating a library. Specifies if we're building a + # static library or a dll. Use in conjunction with `dll_def`. +vs_packages: # List of nuget packages this project depends on. ``` ## The `"build"` tag @@ -86,8 +100,9 @@ Currently, the "`build`" tag have these meanings: * `"all"`: library to build on `"make all"`, and install on the system. * `"protoc"`: a protoc plugin to build on `"make all"` and install on the system. -* `"priviate"`: a library to only build for tests. +* `"private"`: a library to only build for tests. * `"test"`: a test binary to run on `"make test"`. +* `"tool"`: a binary to be built upon `"make tools"`. All of the targets should always be present in the generated project file, if possible and applicable. But the build tag is what should group the targets @@ -111,6 +126,18 @@ should merge OpenSSL, protobuf or zlib inside that library. That effect depends on the `"language"` tag. OpenSSL and zlib are for `"c"` libraries, while protobuf is for `"c++"` ones. +## The `"dll"` tag + +Used only by Visual Studio's project files. "true" means the project will be +built with both static and dynamic runtimes. "false" means it'll only be built +with static runtime. "only" means it'll only be built with the dll runtime. + +## The `"dll_def"` tag + +Specifies the visual studio's dll definition file. When creating a DLL, you +sometimes (not always) need a def file (see grpc.def). + + # The template system We're currently using the [mako templates](http://www.makotemplates.org/) diff --git a/templates/package.xml.template b/templates/package.xml.template index 067c8839d5..d309bfddbc 100644 --- a/templates/package.xml.template +++ b/templates/package.xml.template @@ -12,11 +12,11 @@ <email>grpc-packages@google.com</email> <active>yes</active> </lead> - <date>2016-02-24</date> + <date>2016-03-01</date> <time>16:06:07</time> <version> - <release>0.8.0</release> - <api>0.8.0</api> + <release>0.14.0</release> + <api>0.14.0</api> </version> <stability> <release>beta</release> @@ -24,7 +24,7 @@ </stability> <license>BSD</license> <notes> - - Simplify gRPC PHP installation #4517 + - Increase unit test code coverage #5225 </notes> <contents> <dir baseinstalldir="/" name="/"> @@ -155,5 +155,20 @@ - Simplify gRPC PHP installation #4517 </notes> </release> + <release> + <version> + <release>0.14.0</release> + <api>0.14.0</api> + </version> + <stability> + <release>beta</release> + <api>beta</api> + </stability> + <date>2016-03-01</date> + <license>BSD</license> + <notes> + - Increase unit test code coverage #5225 + </notes> + </release> </changelog> </package> diff --git a/test/core/iomgr/timer_heap_test.c b/test/core/iomgr/timer_heap_test.c index 077a9fd6bd..cd34696f7d 100644 --- a/test/core/iomgr/timer_heap_test.c +++ b/test/core/iomgr/timer_heap_test.c @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -38,6 +38,8 @@ #include <grpc/support/alloc.h> #include <grpc/support/log.h> +#include <grpc/support/useful.h> + #include "test/core/util/test_config.h" static gpr_timespec random_deadline(void) { @@ -57,79 +59,6 @@ static grpc_timer *create_test_elements(size_t num_elements) { return elems; } -static int cmp_elem(const void *a, const void *b) { - int i = *(const int *)a; - int j = *(const int *)b; - return i - j; -} - -static size_t *all_top(grpc_timer_heap *pq, size_t *n) { - size_t *vec = NULL; - size_t *need_to_check_children; - size_t num_need_to_check_children = 0; - - *n = 0; - if (pq->timer_count == 0) return vec; - need_to_check_children = - gpr_malloc(pq->timer_count * sizeof(*need_to_check_children)); - need_to_check_children[num_need_to_check_children++] = 0; - vec = gpr_malloc(pq->timer_count * sizeof(*vec)); - while (num_need_to_check_children > 0) { - size_t ind = need_to_check_children[0]; - size_t leftchild, rightchild; - num_need_to_check_children--; - memmove(need_to_check_children, need_to_check_children + 1, - num_need_to_check_children * sizeof(*need_to_check_children)); - vec[(*n)++] = ind; - leftchild = 1u + 2u * ind; - if (leftchild < pq->timer_count) { - if (gpr_time_cmp(pq->timers[leftchild]->deadline, - pq->timers[ind]->deadline) >= 0) { - need_to_check_children[num_need_to_check_children++] = leftchild; - } - rightchild = leftchild + 1; - if (rightchild < pq->timer_count && - gpr_time_cmp(pq->timers[rightchild]->deadline, - pq->timers[ind]->deadline) >= 0) { - need_to_check_children[num_need_to_check_children++] = rightchild; - } - } - } - - gpr_free(need_to_check_children); - - return vec; -} - -static void check_pq_top(grpc_timer *elements, grpc_timer_heap *pq, - uint8_t *inpq, size_t num_elements) { - gpr_timespec max_deadline = gpr_inf_past(GPR_CLOCK_REALTIME); - size_t *max_deadline_indices = - gpr_malloc(num_elements * sizeof(*max_deadline_indices)); - size_t *top_elements; - size_t num_max_deadline_indices = 0; - size_t num_top_elements; - size_t i; - for (i = 0; i < num_elements; ++i) { - if (inpq[i] && gpr_time_cmp(elements[i].deadline, max_deadline) >= 0) { - if (gpr_time_cmp(elements[i].deadline, max_deadline) > 0) { - num_max_deadline_indices = 0; - max_deadline = elements[i].deadline; - } - max_deadline_indices[num_max_deadline_indices++] = elements[i].heap_index; - } - } - qsort(max_deadline_indices, num_max_deadline_indices, - sizeof(*max_deadline_indices), cmp_elem); - top_elements = all_top(pq, &num_top_elements); - GPR_ASSERT(num_top_elements == num_max_deadline_indices); - for (i = 0; i < num_top_elements; i++) { - GPR_ASSERT(max_deadline_indices[i] == top_elements[i]); - } - gpr_free(max_deadline_indices); - gpr_free(top_elements); -} - static int contains(grpc_timer_heap *pq, grpc_timer *el) { size_t i; for (i = 0; i < pq->timer_count; i++) { @@ -145,15 +74,19 @@ static void check_valid(grpc_timer_heap *pq) { size_t right_child = left_child + 1u; if (left_child < pq->timer_count) { GPR_ASSERT(gpr_time_cmp(pq->timers[i]->deadline, - pq->timers[left_child]->deadline) >= 0); + pq->timers[left_child]->deadline) <= 0); } if (right_child < pq->timer_count) { GPR_ASSERT(gpr_time_cmp(pq->timers[i]->deadline, - pq->timers[right_child]->deadline) >= 0); + pq->timers[right_child]->deadline) <= 0); } } } +/******************************************************************************* + * test1 + */ + static void test1(void) { grpc_timer_heap pq; const size_t num_test_elements = 200; @@ -162,6 +95,8 @@ static void test1(void) { grpc_timer *test_elements = create_test_elements(num_test_elements); uint8_t *inpq = gpr_malloc(num_test_elements); + gpr_log(GPR_INFO, "test1"); + grpc_timer_heap_init(&pq); memset(inpq, 0, num_test_elements); GPR_ASSERT(grpc_timer_heap_is_empty(&pq)); @@ -172,7 +107,6 @@ static void test1(void) { check_valid(&pq); GPR_ASSERT(contains(&pq, &test_elements[i])); inpq[i] = 1; - check_pq_top(test_elements, &pq, inpq, num_test_elements); } for (i = 0; i < num_test_elements; ++i) { /* Test that check still succeeds even for element that wasn't just @@ -182,7 +116,7 @@ static void test1(void) { GPR_ASSERT(pq.timer_count == num_test_elements); - check_pq_top(test_elements, &pq, inpq, num_test_elements); + check_valid(&pq); for (i = 0; i < num_test_operations; ++i) { size_t elem_num = (size_t)rand() % num_test_elements; @@ -193,14 +127,12 @@ static void test1(void) { grpc_timer_heap_add(&pq, el); GPR_ASSERT(contains(&pq, el)); inpq[elem_num] = 1; - check_pq_top(test_elements, &pq, inpq, num_test_elements); check_valid(&pq); } else { GPR_ASSERT(contains(&pq, el)); grpc_timer_heap_remove(&pq, el); GPR_ASSERT(!contains(&pq, el)); inpq[elem_num] = 0; - check_pq_top(test_elements, &pq, inpq, num_test_elements); check_valid(&pq); } } @@ -210,7 +142,108 @@ static void test1(void) { gpr_free(inpq); } +/******************************************************************************* + * test2 + */ + +typedef struct { + grpc_timer elem; + bool inserted; +} elem_struct; + +static elem_struct *search_elems(elem_struct *elems, size_t count, + bool inserted) { + size_t *search_order = gpr_malloc(count * sizeof(*search_order)); + for (size_t i = 0; i < count; i++) { + search_order[i] = i; + } + for (size_t i = 0; i < count * 2; i++) { + size_t a = (size_t)rand() % count; + size_t b = (size_t)rand() % count; + GPR_SWAP(size_t, search_order[a], search_order[b]); + } + elem_struct *out = NULL; + for (size_t i = 0; out == NULL && i < count; i++) { + if (elems[search_order[i]].inserted == inserted) { + out = &elems[search_order[i]]; + } + } + gpr_free(search_order); + return out; +} + +static void test2(void) { + gpr_log(GPR_INFO, "test2"); + + grpc_timer_heap pq; + + elem_struct elems[1000]; + size_t num_inserted = 0; + + grpc_timer_heap_init(&pq); + memset(elems, 0, sizeof(elems)); + + for (size_t round = 0; round < 10000; round++) { + int r = rand() % 1000; + if (r <= 550) { + /* 55% of the time we try to add something */ + elem_struct *el = search_elems(elems, GPR_ARRAY_SIZE(elems), false); + if (el != NULL) { + el->elem.deadline = random_deadline(); + grpc_timer_heap_add(&pq, &el->elem); + el->inserted = true; + num_inserted++; + check_valid(&pq); + } + } else if (r <= 650) { + /* 10% of the time we try to remove something */ + elem_struct *el = search_elems(elems, GPR_ARRAY_SIZE(elems), true); + if (el != NULL) { + grpc_timer_heap_remove(&pq, &el->elem); + el->inserted = false; + num_inserted--; + check_valid(&pq); + } + } else { + /* the remaining times we pop */ + if (num_inserted > 0) { + grpc_timer *top = grpc_timer_heap_top(&pq); + grpc_timer_heap_pop(&pq); + for (size_t i = 0; i < GPR_ARRAY_SIZE(elems); i++) { + if (top == &elems[i].elem) { + GPR_ASSERT(elems[i].inserted); + elems[i].inserted = false; + } + } + num_inserted--; + check_valid(&pq); + } + } + + if (num_inserted) { + gpr_timespec *min_deadline = NULL; + for (size_t i = 0; i < GPR_ARRAY_SIZE(elems); i++) { + if (elems[i].inserted) { + if (min_deadline == NULL) { + min_deadline = &elems[i].elem.deadline; + } else { + if (gpr_time_cmp(elems[i].elem.deadline, *min_deadline) < 0) { + min_deadline = &elems[i].elem.deadline; + } + } + } + } + GPR_ASSERT( + 0 == gpr_time_cmp(grpc_timer_heap_top(&pq)->deadline, *min_deadline)); + } + } + + grpc_timer_heap_destroy(&pq); +} + static void shrink_test(void) { + gpr_log(GPR_INFO, "shrink_test"); + grpc_timer_heap pq; size_t i; size_t expected_size; @@ -274,6 +307,7 @@ int main(int argc, char **argv) { for (i = 0; i < 5; i++) { test1(); + test2(); shrink_test(); } diff --git a/test/cpp/end2end/end2end_test.cc b/test/cpp/end2end/end2end_test.cc index 42757974b2..dc2c4f6426 100644 --- a/test/cpp/end2end/end2end_test.cc +++ b/test/cpp/end2end/end2end_test.cc @@ -437,9 +437,10 @@ class End2endServerTryCancelTest : public End2endTest { break; case CANCEL_AFTER_PROCESSING: - // Server cancelled after writing all messages. Client must have read - // all messages - EXPECT_EQ(num_msgs_read, kNumResponseStreamsMsgs); + // Even though the Server cancelled after writing all messages, the RPC + // may be cancelled before the Client got a chance to read all the + // messages. + EXPECT_LE(num_msgs_read, kNumResponseStreamsMsgs); break; default: { @@ -519,7 +520,11 @@ class End2endServerTryCancelTest : public End2endTest { case CANCEL_AFTER_PROCESSING: EXPECT_EQ(num_msgs_sent, num_messages); - EXPECT_EQ(num_msgs_read, num_msgs_sent); + + // The Server cancelled after reading the last message and after writing + // the message to the client. However, the RPC cancellation might have + // taken effect before the client actually read the response. + EXPECT_LE(num_msgs_read, num_msgs_sent); break; default: diff --git a/test/cpp/end2end/test_service_impl.cc b/test/cpp/end2end/test_service_impl.cc index 66d11d0dfc..7c3e514eff 100644 --- a/test/cpp/end2end/test_service_impl.cc +++ b/test/cpp/end2end/test_service_impl.cc @@ -326,7 +326,11 @@ void TestServiceImpl::ServerTryCancel(ServerContext* context) { EXPECT_FALSE(context->IsCancelled()); context->TryCancel(); gpr_log(GPR_INFO, "Server called TryCancel() to cancel the request"); - EXPECT_TRUE(context->IsCancelled()); + // Now wait until it's really canceled + while (!context->IsCancelled()) { + gpr_sleep_until(gpr_time_add(gpr_now(GPR_CLOCK_REALTIME), + gpr_time_from_micros(1000, GPR_TIMESPAN))); + } } } // namespace testing diff --git a/test/cpp/interop/reconnect_interop_client.cc b/test/cpp/interop/reconnect_interop_client.cc index 1f6b352db1..79a60cc860 100644 --- a/test/cpp/interop/reconnect_interop_client.cc +++ b/test/cpp/interop/reconnect_interop_client.cc @@ -1,6 +1,6 @@ /* * - * Copyright 2015, Google Inc. + * Copyright 2015-2016, Google Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without diff --git a/test/cpp/qps/qps_worker.cc b/test/cpp/qps/qps_worker.cc index 9442017ddf..b83e9d1dd7 100644 --- a/test/cpp/qps/qps_worker.cc +++ b/test/cpp/qps/qps_worker.cc @@ -101,6 +101,19 @@ static std::unique_ptr<Server> CreateServer(const ServerConfig& config) { abort(); } +class ScopedProfile GRPC_FINAL { + public: + ScopedProfile(const char* filename, bool enable) : enable_(enable) { + if (enable_) grpc_profiler_start(filename); + } + ~ScopedProfile() { + if (enable_) grpc_profiler_stop(); + } + + private: + const bool enable_; +}; + class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { public: WorkerServiceImpl(int server_port, QpsWorker* worker) @@ -114,9 +127,8 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { return Status(StatusCode::RESOURCE_EXHAUSTED, ""); } - grpc_profiler_start("qps_client.prof"); + ScopedProfile profile("qps_client.prof", false); Status ret = RunClientBody(ctx, stream); - grpc_profiler_stop(); return ret; } @@ -128,9 +140,8 @@ class WorkerServiceImpl GRPC_FINAL : public WorkerService::Service { return Status(StatusCode::RESOURCE_EXHAUSTED, ""); } - grpc_profiler_start("qps_server.prof"); + ScopedProfile profile("qps_server.prof", false); Status ret = RunServerBody(ctx, stream); - grpc_profiler_stop(); return ret; } diff --git a/tools/README.md b/tools/README.md index a0c41eb79f..df4c6ef7d2 100644 --- a/tools/README.md +++ b/tools/README.md @@ -1,7 +1,5 @@ buildgen: template renderer for our build system. -distpackages: script to generate debian packages. - distrib: scripts to distribute language-specific packages. dockerfile: Docker files to test gRPC. @@ -12,6 +10,4 @@ gce: scripts to help setup testing infrastructure on GCE. jenkins: support for running tests on Jenkins. -profile_analyzer: pretty printer for gRPC profiling data. - run_tests: scripts to run gRPC tests in parallel. diff --git a/tools/distrib/python/docgen.py b/tools/distrib/python/docgen.py index 4ac8f9c64a..161d83e3b7 100755 --- a/tools/distrib/python/docgen.py +++ b/tools/distrib/python/docgen.py @@ -1,5 +1,5 @@ #!/usr/bin/env python2.7 -# Copyright 2015, Google Inc. +# Copyright 2015-2016, Google Inc. # All rights reserved. # # Redistribution and use in source and binary forms, with or without @@ -51,29 +51,35 @@ SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) PROJECT_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, '..', '..', '..')) CONFIG = args.config -SETUP_PATH = os.path.join(PROJECT_ROOT, 'src/python/grpcio/setup.py') -DOC_PATH = os.path.join(PROJECT_ROOT, 'src/python/grpcio/doc/build') +SETUP_PATH = os.path.join(PROJECT_ROOT, 'setup.py') +REQUIREMENTS_PATH = os.path.join(PROJECT_ROOT, 'requirements.txt') +DOC_PATH = os.path.join(PROJECT_ROOT, 'doc/build') INCLUDE_PATH = os.path.join(PROJECT_ROOT, 'include') LIBRARY_PATH = os.path.join(PROJECT_ROOT, 'libs/{}'.format(CONFIG)) VIRTUALENV_DIR = os.path.join(SCRIPT_DIR, 'distrib_virtualenv') VIRTUALENV_PYTHON_PATH = os.path.join(VIRTUALENV_DIR, 'bin', 'python') +VIRTUALENV_PIP_PATH = os.path.join(VIRTUALENV_DIR, 'bin', 'pip') environment = os.environ.copy() environment.update({ 'CONFIG': CONFIG, 'CFLAGS': '-I{}'.format(INCLUDE_PATH), 'LDFLAGS': '-L{}'.format(LIBRARY_PATH), - 'LD_LIBRARY_PATH': LIBRARY_PATH + 'LD_LIBRARY_PATH': LIBRARY_PATH, + 'GRPC_PYTHON_BUILD_WITH_CYTHON': '1', }) subprocess_arguments_list = [ {'args': ['make'], 'cwd': PROJECT_ROOT}, {'args': ['virtualenv', VIRTUALENV_DIR], 'env': environment}, + {'args': [VIRTUALENV_PIP_PATH, 'install', '-r', REQUIREMENTS_PATH], + 'env': environment}, {'args': [VIRTUALENV_PYTHON_PATH, SETUP_PATH, 'build'], 'env': environment}, {'args': [VIRTUALENV_PYTHON_PATH, SETUP_PATH, 'doc'], 'env': environment}, ] for subprocess_arguments in subprocess_arguments_list: + print('Running command: {}'.format(subprocess_arguments['args'])) subprocess.check_call(**subprocess_arguments) if args.submit: diff --git a/tools/run_tests/build_node.bat b/tools/run_tests/build_node.bat index 6896bc1d1b..82e8208348 100644 --- a/tools/run_tests/build_node.bat +++ b/tools/run_tests/build_node.bat @@ -27,4 +27,16 @@ @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -npm install --build-from-source
\ No newline at end of file +set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm + +del /f /q BUILD || rmdir build /s /q + +call npm install --build-from-source + +@rem delete the redundant openssl headers +for /f "delims=v" %%v in ('node --version') do ( + rmdir "%USERPROFILE%\.node-gyp\%%v\include\node\openssl" /S /Q +) + +@rem rebuild, because it probably failed the first time +call npm install --build-from-source
\ No newline at end of file diff --git a/tools/run_tests/build_python.sh b/tools/run_tests/build_python.sh index e0fcbb602d..f120fc7ed6 100755 --- a/tools/run_tests/build_python.sh +++ b/tools/run_tests/build_python.sh @@ -45,3 +45,6 @@ export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 tox --notest $ROOT/.tox/py27/bin/python $ROOT/setup.py build +$ROOT/.tox/py27/bin/python $ROOT/setup.py build_py +$ROOT/.tox/py27/bin/python $ROOT/setup.py build_ext --inplace +$ROOT/.tox/py27/bin/python $ROOT/setup.py gather --test diff --git a/tools/run_tests/jobset.py b/tools/run_tests/jobset.py index adf178bb3c..a3b246dc08 100755 --- a/tools/run_tests/jobset.py +++ b/tools/run_tests/jobset.py @@ -384,7 +384,8 @@ class Jobset(object): self._travis, self._add_env) self._running.add(job) - self.resultset[job.GetSpec().shortname] = [] + if not self.resultset.has_key(job.GetSpec().shortname): + self.resultset[job.GetSpec().shortname] = [] return True def reap(self): diff --git a/tools/run_tests/post_test_node.bat b/tools/run_tests/post_test_node.bat deleted file mode 100644 index 1a2a5491fa..0000000000 --- a/tools/run_tests/post_test_node.bat +++ /dev/null @@ -1,30 +0,0 @@ -@rem Copyright 2016, Google Inc. -@rem All rights reserved. -@rem -@rem Redistribution and use in source and binary forms, with or without -@rem modification, are permitted provided that the following conditions are -@rem met: -@rem -@rem * Redistributions of source code must retain the above copyright -@rem notice, this list of conditions and the following disclaimer. -@rem * Redistributions in binary form must reproduce the above -@rem copyright notice, this list of conditions and the following disclaimer -@rem in the documentation and/or other materials provided with the -@rem distribution. -@rem * Neither the name of Google Inc. nor the names of its -@rem contributors may be used to endorse or promote products derived from -@rem this software without specific prior written permission. -@rem -@rem THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -@rem "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -@rem LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -@rem A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -@rem OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -@rem SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -@rem LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -@rem DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -@rem THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -@rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -@rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -rmdir node_modules /S /Q
\ No newline at end of file diff --git a/tools/run_tests/pre_build_node.bat b/tools/run_tests/pre_build_node.bat index 6e7cbe5d42..a29456f9ed 100644 --- a/tools/run_tests/pre_build_node.bat +++ b/tools/run_tests/pre_build_node.bat @@ -27,13 +27,8 @@ @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -@rem Expire cache after 1 week -npm update --cache-min 604800 +set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm -npm install node-gyp-install -.\node_modules\.bin\node-gyp-install.cmd +@rem Expire cache after 1 week +call npm update --cache-min 604800 -@rem delete the redundant openssl headers -for /f "delims=v" %%v in ('node --version') do ( - rmdir "%HOMEDRIVE%%HOMEPATH%\.node-gyp\%%v\include\node\openssl" /S /Q -)
\ No newline at end of file diff --git a/tools/run_tests/run_interop_tests.py b/tools/run_tests/run_interop_tests.py index df3ab90a83..1dc772a856 100755 --- a/tools/run_tests/run_interop_tests.py +++ b/tools/run_tests/run_interop_tests.py @@ -60,6 +60,8 @@ _SKIP_COMPRESSION = ['large_compressed_unary', _SKIP_ADVANCED = ['custom_metadata', 'status_code_and_message', 'unimplemented_method'] +_TEST_TIMEOUT = 3*60 + class CXXLanguage: def __init__(self): @@ -459,7 +461,7 @@ def cloud_to_prod_jobspec(language, test_case, server_host_name, environ=environ, shortname='%s:%s:%s:%s' % (suite_name, server_host_name, language, test_case), - timeout_seconds=90, + timeout_seconds=_TEST_TIMEOUT, flake_retries=5 if args.allow_flakes else 0, timeout_retries=2 if args.allow_flakes else 0, kill_handler=_job_kill_handler) @@ -495,7 +497,7 @@ def cloud_to_cloud_jobspec(language, test_case, server_name, server_host, environ=environ, shortname='cloud_to_cloud:%s:%s_server:%s' % (language, server_name, test_case), - timeout_seconds=90, + timeout_seconds=_TEST_TIMEOUT, flake_retries=5 if args.allow_flakes else 0, timeout_retries=2 if args.allow_flakes else 0, kill_handler=_job_kill_handler) diff --git a/tools/run_tests/run_node.bat b/tools/run_tests/run_node.bat index 4177736356..0987fbee55 100644 --- a/tools/run_tests/run_node.bat +++ b/tools/run_tests/run_node.bat @@ -27,6 +27,7 @@ @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm set JUNIT_REPORT_PATH=src\node\report.xml set JUNIT_REPORT_STACK=1 .\node_modules\.bin\mocha.cmd --reporter mocha-jenkins-reporter --timeout 8000 src\node\test
\ No newline at end of file diff --git a/tools/run_tests/run_python.sh b/tools/run_tests/run_python.sh index ffe9c12af1..ace49e1514 100755 --- a/tools/run_tests/run_python.sh +++ b/tools/run_tests/run_python.sh @@ -42,7 +42,12 @@ export LDFLAGS="-L$ROOT/libs/$CONFIG" export GRPC_PYTHON_BUILD_WITH_CYTHON=1 export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 -tox +if [ "$CONFIG" = "gcov" ] +then + tox +else + $ROOT/.tox/py27/bin/python $ROOT/setup.py test_lite +fi mkdir -p $ROOT/reports rm -rf $ROOT/reports/python-coverage diff --git a/tools/run_tests/run_tests.py b/tools/run_tests/run_tests.py index 08a5ff0e8f..2df82adf0a 100755 --- a/tools/run_tests/run_tests.py +++ b/tools/run_tests/run_tests.py @@ -290,10 +290,7 @@ class NodeLanguage(object): return [['tools/run_tests/build_node.sh', self.node_version]] def post_tests_steps(self): - if self.platform == 'windows': - return [['tools\\run_tests\\post_test_node.bat']] - else: - return [] + return [] def makefile_name(self): return 'Makefile' @@ -353,15 +350,27 @@ class PythonLanguage(object): _check_compiler(self.args.compiler, ['default']) def test_specs(self): + # load list of known test suites + with open('src/python/grpcio/tests/tests.json') as tests_json_file: + tests_json = json.load(tests_json_file) environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS) environment['PYVER'] = '2.7' - return [self.config.job_spec( - ['tools/run_tests/run_python.sh'], - None, - environ=environment, - shortname='py.test', - timeout_seconds=15*60 - )] + if self.config.build_config != 'gcov': + return [self.config.job_spec( + ['tools/run_tests/run_python.sh'], + None, + environ=dict(environment.items() + + [('GRPC_PYTHON_TESTRUNNER_FILTER', suite_name)]), + shortname='py.test.%s' % suite_name, + timeout_seconds=5*60) + for suite_name in tests_json] + else: + return [self.config.job_spec(['tools/run_tests/run_python.sh'], + None, + environ=environment, + shortname='py.test.coverage', + timeout_seconds=15*60)] + def pre_build_steps(self): return [] |