Add "toolchain-identifier" attribute to cc_toolchain rule

Currently the selection of C++ toolchain configuration depends on 2 different things:
1. CROSSTOOL file: a proto text file that contains multiple CrosstoolConfig.CToolchain messages, out of which we want to select the appropriate CToolchain;
2. cc_toolchain_suite rule, specified by --crosstool_top option: this rule contains "toolchains" map attribute. The map's keys are of type <cpu>|<compiler>, or just <cpu>, and the values are labels of cc_toolchain rules (which contain additional C++ information).

If there is an entry in cc_toolchain_suite.toolchains that corresponds to the --cpu and --compiler options, we use it, otherwise we loop through all the CToolchains in the CROSSTOOL file, select the one that corresponds to the --cpu and --compiler options, and then get the cc_toolchain label from cc_toolchain_suite.toolchains[toolchain.targetCpu|toolchain.compiler].
In both cases we read the CROSSTOOL file and pass all its information forward, to be used in creation of CppConfiguration and CcToolchainProvider creation.

As part of the efforts of rewriting CROSSTOOL in Skylark, we need to make obtaining the cc_toolchain label independent of the CROSSTOOL file and toolchain selection. As a step towards that goal, we add a new "toolchain_identifier" attribute to cc_toolchain, which uniquely identifies a CToolchain in the CROSSTOOL file.

Now the process of getting the CToolchain goes as follows:
Check for existence of cc_toolchain_suite.toolchains[<cpu>|<compiler>], if --compiler is specified, otherwise check for cc_toolchain_suite.toolchains[<cpu>].

1. if a value is found, load the cc_toolchain rule and look for the toolchain_identifier attribute.
    1.a if the attribute exists, loop through all the CToolchains in CROSSTOOL and select the one with the matching toolchain identifier.
    1.b otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu and --compiler values.

2. If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.

In the future we will get rid of 2. by making sure that cc_toolchain_suite.toolchains[<cpu>|<compiler>] and cc_toolchain_suite.toolchains[<cpu>] are always defined.
1.b will be removed by making the cc_toolchain.toolchain_identifier attribute mandatory
After this deriving the cc_toolchain label will be independent of the CROSSTOOL file

1.a will ultimately be replaced by an attribute that points to a skylark_crosstool rule, that will contain all the information that CROSSTOOL contains, allowing us to remove CROSSTOOL altogether.

Work towards issue #5380
RELNOTES: None.
PiperOrigin-RevId: 200388550

4 files changed, 112 insertions, 16 deletions

diff --git a/src/main/java/com/google/devtools/build/lib/rules/cpp/CcToolchainRule.java b/src/main/java/com/google/devtools/build/lib/rules/cpp/CcToolchainRule.java
index a22c7a3f32..f5ca99384c 100644
--- a/src/main/java/com/google/devtools/build/lib/rules/cpp/CcToolchainRule.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/cpp/CcToolchainRule.java
@@ -176,7 +176,6 @@ public final class CcToolchainRule implements RuleDefinition {
         // TODO(b/78578234): Make this the default and remove the late-bound versions.
         .add(attr("libc_top", LABEL).allowedFileTypes())
         .add(attr(LIBC_TOP_ATTR, LABEL).value(LIBC_TOP_VALUE))
-
         .add(attr(FDO_OPTIMIZE_ATTR, LABEL).singleArtifact().value(FDO_OPTIMIZE_VALUE))
         .add(
             attr(FDO_PROFILE_ATTR, LABEL)
@@ -194,6 +193,10 @@ public final class CcToolchainRule implements RuleDefinition {
                 .value(CppRuleClasses.LIPO_CONTEXT_COLLECTOR)
                 .skipPrereqValidatorCheck())
         .add(attr("proto", Type.STRING))
+        .add(
+            attr("toolchain_identifier", Type.STRING)
+                .nonconfigurable("Used in configuration creation")
+                .value(""))
         .build();
   }
 
diff --git a/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java b/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java
index 317d52db40..73d6c538c1 100644
--- a/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java
@@ -55,6 +55,35 @@ import javax.annotation.Nullable;
  * This class represents the C/C++ parts of the {@link BuildConfiguration}, including the host
  * architecture, target architecture, compiler version, and a standard library version. It has
  * information about the tools locations and the flags required for compiling.
+ *
+ * <p>Before {@link CppConfiguration} is created, two things need to be done:
+ *
+ * <ol>
+ *   <li>choosing a {@link CcToolchainRule} label from {@code toolchains} map attribute of {@link
+ *       CcToolchainSuiteRule}.
+ *   <li>selection of a {@link
+ *       com.google.devtools.build.lib.view.config.crosstool.CrosstoolConfig.CToolchain} from the
+ *       CROSSTOOL file.
+ * </ol>
+ *
+ * <p>The process goes as follows:
+ *
+ * <p>Check for existence of {@link CcToolchainSuiteRule}.toolchains[<cpu>|<compiler>], if
+ * --compiler is specified, otherwise check for {@link CcToolchainSuiteRule}.toolchains[<cpu>].
+ *
+ * <ul>
+ *   <li>if a value is found, load the {@link CcToolchainRule} rule and look for the {@code
+ *       toolchain_identifier} attribute.
+ *   <li>
+ *       <ul>
+ *         <li>if the attribute exists, loop through all the {@code CToolchain}s in CROSSTOOL and
+ *             select the one with the matching toolchain identifier.
+ *         <li>otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu
+ *             and --compiler values.
+ *       </ul>
+ *   <li>If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and
+ *       --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.
+ * </ul>
  */
 @AutoCodec
 @Immutable
diff --git a/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java b/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java
index e20e3a0d4c..48ec524c0b 100644
--- a/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java
@@ -32,6 +32,7 @@ import com.google.devtools.build.lib.packages.NoSuchThingException;
 import com.google.devtools.build.lib.packages.NonconfigurableAttributeMapper;
 import com.google.devtools.build.lib.packages.Rule;
 import com.google.devtools.build.lib.packages.Target;
+import com.google.devtools.build.lib.syntax.Type;
 import com.google.devtools.build.lib.vfs.FileSystemUtils;
 import com.google.devtools.build.lib.vfs.PathFragment;
 import com.google.devtools.build.lib.view.config.crosstool.CrosstoolConfig;
@@ -142,9 +143,6 @@ public class CppConfigurationLoader implements ConfigurationFragmentFactory {
     if (file == null) {
       return null;
     }
-    CrosstoolConfig.CToolchain toolchain =
-        CrosstoolConfigurationLoader.selectToolchain(
-            file.getProto(), options, cpuTransformer.getTransformer());
 
     PathFragment fdoPath = null;
     Label fdoProfileLabel = null;
@@ -168,18 +166,17 @@ public class CppConfigurationLoader implements ConfigurationFragmentFactory {
 
     Label ccToolchainLabel;
     Target crosstoolTop;
-
+    CrosstoolConfig.CToolchain toolchain = null;
     try {
       crosstoolTop = env.getTarget(crosstoolTopLabel);
     } catch (NoSuchThingException e) {
       throw new IllegalStateException(e);  // Should have been found out during redirect chasing
     }
 
+    String desiredCpu = cpuTransformer.getTransformer().apply(options.get(Options.class).cpu);
     if (crosstoolTop instanceof Rule
         && ((Rule) crosstoolTop).getRuleClass().equals("cc_toolchain_suite")) {
       Rule ccToolchainSuite = (Rule) crosstoolTop;
-
-      String desiredCpu = cpuTransformer.getTransformer().apply(options.get(Options.class).cpu);
       String key =
           desiredCpu + (cppOptions.cppCompiler == null ? "" : ("|" + cppOptions.cppCompiler));
       Map<String, Label> toolchains =
@@ -187,13 +184,21 @@ public class CppConfigurationLoader implements ConfigurationFragmentFactory {
               .get("toolchains", BuildType.LABEL_DICT_UNARY);
       ccToolchainLabel = toolchains.get(key);
       if (ccToolchainLabel == null) {
+        // If the cc_toolchain_suite does not contain entry for --cpu|--compiler (or only --cpu if
+        // --compiler is not present) we select the toolchain by looping through all the toolchains
+        // in the CROSSTOOL file and selecting the one that matches --cpu (and --compiler, if
+        // present). Then we use the toolchain.target_cpu|toolchain.compiler key to get the
+        // cc_toolchain label.
+        toolchain =
+            CrosstoolConfigurationLoader.selectToolchain(
+                file.getProto(), options, cpuTransformer.getTransformer());
         ccToolchainLabel = toolchains.get(toolchain.getTargetCpu() + "|" + toolchain.getCompiler());
       }
       if (ccToolchainLabel == null) {
         String errorMessage =
             String.format(
                 "cc_toolchain_suite '%s' does not contain a toolchain for CPU '%s'",
-                crosstoolTopLabel, toolchain.getTargetCpu());
+                crosstoolTopLabel, desiredCpu);
         if (cppOptions.cppCompiler != null) {
           errorMessage = errorMessage + " and compiler " + cppOptions.cppCompiler;
         }
@@ -221,6 +226,22 @@ public class CppConfigurationLoader implements ConfigurationFragmentFactory {
           "The label '%s' is not a cc_toolchain rule", ccToolchainLabel));
     }
 
+    if (toolchain == null) {
+      // If cc_toolchain_suite contains an entry for the given --cpu and --compiler options, we
+      // select the toolchain by its identifier if "toolchain_identifier" attribute is present.
+      // Otherwise, we fall back to going through the CROSSTOOL file to select the toolchain using
+      // the legacy selection mechanism.
+      String identifier =
+          NonconfigurableAttributeMapper.of((Rule) ccToolchain)
+              .get("toolchain_identifier", Type.STRING);
+      toolchain =
+          identifier.isEmpty()
+              ? CrosstoolConfigurationLoader.selectToolchain(
+                  file.getProto(), options, cpuTransformer.getTransformer())
+              : CrosstoolConfigurationLoader.getToolchainByIdentifier(
+                  file.getProto(), identifier, desiredCpu, cppOptions.cppCompiler);
+    }
+
     Label sysrootLabel = getSysrootLabel(toolchain, cppOptions.libcTopLabel);
 
     return new CppConfigurationParameters(
diff --git a/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java b/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java
index caed2d37b6..a475aeb1a2 100644
--- a/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java
+++ b/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java
@@ -18,6 +18,7 @@ import static java.nio.charset.StandardCharsets.UTF_8;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Function;
+import com.google.common.base.Strings;
 import com.google.common.cache.Cache;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.io.BaseEncoding;
@@ -375,7 +376,7 @@ public class CrosstoolConfigurationLoader {
               + cpuBuilder
               + "]");
     }
-    checkToolChain(selectedIdentifier, desiredCpu);
+    checkToolchain(selectedIdentifier, desiredCpu);
 
     for (CrosstoolConfig.CToolchain toolchain : release.getToolchainList()) {
       if (toolchain.getToolchainIdentifier().equals(selectedIdentifier)) {
@@ -406,15 +407,14 @@ public class CrosstoolConfigurationLoader {
   }
 
   /**
-   * Makes sure that {@code selectedIdentifier} is a valid identifier for a toolchain,
-   * i.e. it starts with a letter or an underscore and continues with only dots, dashes,
-   * spaces, letters, digits or underscores (i.e. matches the following regular expression:
-   * "[a-zA-Z_][\.\- \w]*").
+   * Makes sure that {@code selectedIdentifier} is a valid identifier for a toolchain, i.e. it
+   * starts with a letter or an underscore and continues with only dots, dashes, spaces, letters,
+   * digits or underscores (i.e. matches the following regular expression: "[a-zA-Z_][\.\- \w]*").
    *
-   * @throws InvalidConfigurationException if selectedIdentifier does not match the
-   *         aforementioned regular expression.
+   * @throws InvalidConfigurationException if selectedIdentifier does not match the aforementioned
+   *     regular expression.
    */
-  private static void checkToolChain(String selectedIdentifier, String cpu)
+  private static void checkToolchain(String selectedIdentifier, String cpu)
       throws InvalidConfigurationException {
     // If you update this regex, please do so in the javadoc comment too, and also in the
     // crosstool_config.proto file.
@@ -438,4 +438,47 @@ public class CrosstoolConfigurationLoader {
     selectToolchain(file.getProto(), options, cpuTransformer);
     return file.getProto();
   }
+
+  /**
+   * Selects a crosstool toolchain based on the toolchain identifier.
+   *
+   * @throws InvalidConfigurationException if no matching toolchain can be found, if multiple
+   *     toolchains with the same identifier are found, or if the target_cpu or compiler of the
+   *     selected toolchain are not the same as --cpu and --compiler options.
+   */
+  public static CrosstoolConfig.CToolchain getToolchainByIdentifier(
+      CrosstoolConfig.CrosstoolRelease proto,
+      String toolchainIdentifier,
+      String cpu,
+      @Nullable String compiler)
+      throws InvalidConfigurationException {
+    checkToolchain(toolchainIdentifier, cpu);
+    CrosstoolConfig.CToolchain selectedToolchain = null;
+    for (CrosstoolConfig.CToolchain toolchain : proto.getToolchainList()) {
+      if (toolchain.getToolchainIdentifier().equals(toolchainIdentifier)) {
+        if (selectedToolchain != null) {
+          throw new InvalidConfigurationException(
+              String.format("Multiple toolchains with '%s' identifier", toolchainIdentifier));
+        }
+        selectedToolchain = toolchain;
+      }
+    }
+    if (selectedToolchain == null) {
+      throw new InvalidConfigurationException(
+          String.format("Toolchain identifier '%s' was not found", toolchainIdentifier));
+    }
+    if ((compiler != null && !selectedToolchain.getCompiler().equals(compiler))
+        || !selectedToolchain.getTargetCpu().equals(cpu)) {
+      throw new InvalidConfigurationException(
+          String.format(
+              "The selected toolchain's cpu and compiler must match the command line options:\n"
+                  + "  --cpu: %s, toolchain.target_cpu: %s\n"
+                  + "  --compiler: %s, toolchain.compiler: %s",
+              cpu,
+              selectedToolchain.getTargetCpu(),
+              Strings.nullToEmpty(compiler),
+              selectedToolchain.getCompiler()));
+    }
+    return selectedToolchain;
+  }
 }