From cdb49c5e8392132a868c6199e617951c0aa268e1 Mon Sep 17 00:00:00 2001 From: Damien Martin-Guillerez Date: Thu, 31 Mar 2016 19:22:16 +0000 Subject: Adds a blog post about C++ autoconf with Skylark remote repository Also a bit of best practices (for issue #1081) that should be integrated into the main documentation later when everything gets more stable. -- MOS_MIGRATED_REVID=118706269 --- site/blog/_posts/2016-03-31-autoconfiguration.md | 89 ++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 site/blog/_posts/2016-03-31-autoconfiguration.md (limited to 'site/blog') diff --git a/site/blog/_posts/2016-03-31-autoconfiguration.md b/site/blog/_posts/2016-03-31-autoconfiguration.md new file mode 100644 index 0000000000..0821f7bb32 --- /dev/null +++ b/site/blog/_posts/2016-03-31-autoconfiguration.md @@ -0,0 +1,89 @@ +--- +layout: posts +title: Using Skylark remote repositories to auto-detect the C++ toolchain. +--- + +[Skylark remote repositories](/docs/skylark/repository_rules.html) let you +create custom [external repositories](/docs/external.html) using +[Skylark](/docs/skylark/index.html). This not only enables creating rules for +custom package systems such as [PyPi](https://pypi.python.org) but also generating +a repository to reflect the toolchain installed on the workstation Bazel is running +on. We explain here how we implemented [auto-configuration for the C++ +toolchain](https://github.com/bazelbuild/bazel/blob/master/tools/cpp/cc_configure.bzl). + +## Principles + +
+C++ toolchain: the set of binaries and libraries required to build C++ code. + +Crosstool: a compiler capable of building for a certain architecture, which +can be different from the host architecture (e.g., gcc running on Linux and +building for Raspberry Pi). +
+ +C++ toolchains are configured in Bazel using a [crosstool target](https://github.com/bazelbuild/bazel/blob/8fa5ae6a6364100f2a7f9130e62eb0edb447339a/tools/cpp/BUILD#L32) +and a [CROSSTOOL file](https://github.com/bazelbuild/bazel/blob/master/tools/cpp/CROSSTOOL). + +This crosstool target (:default_toolchain) is the first step in moving the contents +of the CROSSTOOL file entirely into BUILD file rules. The CROSSTOOL file defines +where to find the C++ compiler, its include directories and also the various flag +to use at each compilation step. + +When your C++ compiler is not in the standard location, then this static +CROSSTOOL file cannot find it. To cope with the variety of installation out +there, we created a `cc_configure` Skylark repository rule that will generates +a `@local_config_cc//tools/cpp` package containing a generated CROSSTOOL file +based on the information we gathered from the operating system. + + +## Implementation + +The [`cc_configure`](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L291) +rule is actually a macro wrapping the [`cc_autoconf`](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L288) +enforcing the `local_config_cc` name for the repository. The +[implementation](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L255) +of the `cc_autoconf` rule does the following step: + + - [Detect the `cpu_value`](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L85) + using the [`repository_ctx.os.name`](/docs/skylark/lib/repository_os.html#name) attribute. + - Generates a [static package](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L85) + if we do not support the target platform. + - Detect the [C++ compiler path](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L235) + using [`repository_ctx.which`](/docs/skylark/lib/repository_ctx.html#which) and the `CC` environment variable with + [`repository_ctx.os.environ`](/docs/skylark/lib/repository_os.html#environ). + - Detect some [more tool paths](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L53), + still using [`repository_ctx.which`](/docs/skylark/lib/repository_ctx.html#which). + - Generates the [various flag for the `CROSSTOOL` file](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L127), + [testing flags against the detected compiler](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L114) + using [`repository_ctx.execute`](/docs/skylark/lib/repository_ctx.html#execute). We also + [detect the include directories](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L101) + with [`repository_ctx.execute`](/docs/skylark/lib/repository_ctx.html#execute). + - With the gathered information, generate the C++ tools package: its [BUILD file](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L274), + [wrapper script for Darwin](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L278) and + [CROSSTOOL file](https://github.com/bazelbuild/bazel/blob/9116b3e99af2fd31d92c9bb7c37905a1675456c1/tools/cpp/cc_configure.bzl#L279) using + [`repository_ctx.template`](/docs/skylark/lib/repository_ctx.html#template). + +So using the function provided by [`repository_ctx`](/docs/skylark/lib/repository_ctx.html), we can discover +the binaries on the system, what version they are, and which options they support, then generate a +configuration to match the local C++ toolchain. + + +## Creating your own repository rules + +When creating a Skylark remote repository, a few things should be taken in considerations: + + - The Skylark implementation of a remote repository is run during the loading phase of + the repository, which means that unless the rule definition is changed in the WORKSPACE + file or the implementation fails, it will not be re-run unless the user does a + `bazel clean --expunge`. We are thinking of further command to force re-run that loading + phase for a specific remote repository ([#974](https://github.com/bazelbuild/bazel/issues/974)). + - Skylark remote repository can do a lot of non hermetic operation, it is recommended + to check as many things as possible to ensure hermeticity (and overall, we recommend + using a vendored toolchain instead of using auto-detected one if reproducibility is important). + For example, it is recommended to use the `sha256` argument of the + [`repository_ctx.download`](/docs/skylark/lib/repository_ctx.html#download) method. + - Naming a rule can be complex and we recommend to not use standard suffix of classical + rules for remote repositories (e.g. `*_library` or `*_binary`). If you create a + package rule, a good name would probably be `xxx_package` (e.g., `pypi_package`). If + you create an autoconfiguration rule, `xxx_configure` is probably the best name + (e.g. `cc_configure`). -- cgit v1.2.3