diff options
-rw-r--r-- | site/designs/_posts/2016-08-04-extensibility-for-native-rules.md | 154 |
1 files changed, 154 insertions, 0 deletions
diff --git a/site/designs/_posts/2016-08-04-extensibility-for-native-rules.md b/site/designs/_posts/2016-08-04-extensibility-for-native-rules.md new file mode 100644 index 0000000000..ed5877e4ee --- /dev/null +++ b/site/designs/_posts/2016-08-04-extensibility-for-native-rules.md @@ -0,0 +1,154 @@ +--- +layout: contribute +title: Extensibility For Native Rules +--- +# Extensibility For Native Rules + +**Status**: Reviewed, not yet implemeted + +**Author**: [Dmitry Lomov](mailto:dslomov@google.com) + +## Motivation + +There is a number of requests that require Skylark API to access functionality +of native rules from Skylark rules. Typical scenarios can be illustrated by the +following "sandwich": + +```python +bread_library(name = "top", …) +java_library(name = "meat", deps = [":top", …], …) +bread_library(name = "bottom", deps = [":meat", …]) +``` + +Here bread\_library is a rule written in Skylark. Here we need three things: + +* implementation of bread\_library should be able to produce jar files and other + artifacts in the same way as native java\_library rule would; in other words + the implementation should be able to delegate to the a native implementation +* java\_library should allow depending on bread\_library; importantly, that + dependence should be *meaningful*, that is if bread\_library produces Java + artifacts, such as jar files, java\_library should be able to compile against + those like it would against a java\_library dependency +* bread\_library should be able to depend on java\_library; it should be able to + access all information it needs; when delegating to native implementation, + there should be a simple way to pass information from a dependency to to the + native implementation + +In this document we present some ideas about what how this all might look like, +and suggest some practical steps we can take to get there. + +## Extensible Native Rules + +This proposal assumes [Declared Providers +](/designs/declared-providers.html) +are implemented. Here is how the implementation of bread\_library might look +like: + +```python +# Implementation of a rule that transpiles to Java and invokes a native +# compilation +def _bread_library_impl(ctx): + bread_sources = [f for src in ctx.attrs.src for f in src.files] + generated_java_files = _invoke_bread_transpiler(ctx, bread_sources) + # lang.java.provider is a declared provider for Java + java_deps = [target[lang.java.provider] for target in ctx.attrs.deps] + # create a native compilation action + java_p = lang.java.compile(ctx, + srcs = generated_java_files, + # information about dependencies is just a lang.java.provider + deps = java_deps, + ...) + # java_p is a lang.java.provider representing the result of compilation + # action we return that provider and immediately java_libary rule can depend + # on us + return [java_p, ...] + +# Implementation of a rule that compiles to JVM bytecode directly +def _scala_library_impl(ctx): + # collect dependency jars to pass to the compile action + dep_jars = [dep[java.lang.provider].jar for dep in ctx.attrs.deps] + jar_output = ctx.new_file(...) + ... construct compilation actions ... + # build a provider that passes all transitive information + transitive_p = lang.java.transitive( + [dep[java.lang.provider] for dep in ctx.attrs.deps]) + java_p = lang.java.provider( + transitive_p, + jar = jar_output, + # update transitive information that we care about + transitive_jars = + transitive_p.transitive_jars | set(jar_output), + ... whatever other information is needed ...) + # return java.lang.provider + return [java_p, ...] +``` + +The provider is the glue ("butter") that connects Skylark rules to native rules +and also to the native rule implementations exposed to Skylark. Note how the +native rule implementation (lang.java.compile) both consumes the entire +providers from dependencies and returns the provider that needs to be returned +from the rule. `lang.java.transitive` is a function that passes all the +transitive information correctly from dependencies. The [existing '.java' +provider](http://www.bazel.io/docs/skylark/lib/JavaSkylarkApiProvider.html) +becomes the same thing as lang.java.provider. + +Note: for the sake for this document we are placing things in lang.java. There +are other alternatives to this, e.g. "magical" .bzl files from which +java\_provider and java\_compile function are exported. + +## How to get there + +Our current native rules are not as neat as described above. Making them +extensible in one go is a difficult and long term project (or rather, projects: +one for each language). Here is a suggested steps for extensibility of +particular language implementation (we continue to use Java as a running +example). + +### Phase 1: Expose native compilation actions + +At this step, lang.java.provider is a *black box*. Skylark rules cannot +construct the lang.java.provider directly: the only way to create it is to +invoke lang.java.compile function. + +Native implementations of Java rules are rewritten so that they can link to deps +that return lang.java.provider and that they return lang.java.provider. The +implementation of the provider can just be a bag of all providers that Java +rules normally return - since that bag is not openable by Skylark, we can +refactor it later without much difficulty . + +Native compilation function (java.lang.compile) is pretty much +JavaLibrary.create refactored so that it gets its dependent providers not from +attributes but from a list of bags. JavaLibrary.create just collects the bags +from deps and passes those to that function. + +At the end of this phase, implementing code generators (and code generating +aspects) such as java\_proto\_library becomes possible. This also covers many +(most?) use cases where people use macros to delegate to native rule +implementations + +No huge refactoring of language rule implementation is needed, but the stage is +set for gradual opening up in the future. + +### Phase 1a: Implementing JavaSkylarkApiProvider on top of black box + +(Optional) As lang.java.provider is just a bag of existing providers, it is easy +to just implement everything in 'target.java' on top of it, if desired. + +### Phase 2: Evolving the API and opening up + +The next step in the API evolution is making the black box provider less black. +This means introducing a constructor for lang.java.provider as well as accessors +to fields. + +The API can be designed gradually and thoughtfully, only exposing the things we +need and adding carefully: as an example sequence first just java libraries, +then resources, then JNI, then support for tests. Existence of +lang.java.transitive is crucial at this stage as it allows merging of transitive +information from dependencies that is not yet exposed to Skylark. + +As API exposure gradually progresses, the exposed Skylark API reaches parity +with internal API. + +Through the execution of this phase, more and more use cases are covered, and at +the end the rules are fully extensible. + |