aboutsummaryrefslogtreecommitdiffhomepage
path: root/tools/test
Commit message (Collapse)AuthorAge
* test-wrapper: add empty impl + Bazel flagGravatar Laszlo Csomor2018-08-14
| | | | | | | | | | | | | | | | | | | | | | | This commit adds: - the skeleton implementation of the Windows native test wrapper - a depenency on the native test wrapper from test rules, through the new $test_wrapper rule attribute - the --windows_native_test_wrapper Bazel flag, which is currently a no-op See https://github.com/bazelbuild/bazel/issues/5508 Change-Id: I8df95c8ce8bab53c51c257698ec95416065a836e Closes #5854. Change-Id: I2ffc78bceec5dd867af775b5878f105fa87c3dba PiperOrigin-RevId: 208650699
* Add --filter_sources flag to LcovMerger.Gravatar iirina2018-08-14
| | | | | | | | | | | | When the `--filter_sources` flag is used `LcovMerger` excludes from the merged coverage the sources whose names match any of the regex in the flag. The value of `--filter_sources` is a list of regex separated by `,`. This flag comes as a preparation for using `gcov` instead of `lcov` for collecting coverage data for C++. The flag was not needed before because `lcov` has some functionality for excluding some files (e.g. `--no-external` to ignore coverage data for system files). Closes #5834. PiperOrigin-RevId: 208606867
* Do not hardcode LCOV pathGravatar Vladimir Zaytsev2018-08-10
| | | | | | | | fixes #5644 Closes #5646. PiperOrigin-RevId: 208175002
* Windows: add //tools/... tests to test suiteGravatar Laszlo Csomor2018-08-09
| | | | | | | | | | | | Add all tests under //tools/... to //src:all_windows_tests, to run them on Bazel CI. See https://github.com/bazelbuild/bazel/issues/4292 Closes #5839. Change-Id: Iae647f561683cc50ef62416c4b834576ae918486 PiperOrigin-RevId: 208167743
* Support 2 more valid gcov info lines in GcovParser.Gravatar iirina2018-08-09
| | | | | | Closes #5843. PiperOrigin-RevId: 208041110
* Create BUILD.tools for //tools/test package.Gravatar Laszlo Csomor2018-08-09
| | | | | | | | | | | | | | | | | | | Motivation: as part of fixing issue #5508, I'll add a cc_binary to this package. The binary will be embedded in Bazel as a prebuilt binary (to avoid requiring a C++ compiler to use it). Therefore //tools/test/BUILD and @bazel_tools//tools/test/BUILD will contain different rules, mandating the BUILD file split. See https://github.com/bazelbuild/bazel/issues/5508 Change-Id: If21bafbc3d83d59e52de498cf3efd2b1b79fa7b6 Closes #5787. Change-Id: If21bafbc3d83d59e52de498cf3efd2b1b79fa7b6 PiperOrigin-RevId: 208005382
* Add a parser for gcov intermediate file format.Gravatar Irina Iancu2018-08-01
| | | | | | | | | | | This change comes as a preparation for using gcov for Bazel C++ coverage instead of lcov. See documentation of gcov intermediate format at https://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html#Invoking-Gcov under --intermediate-format. Change-Id: I6cd2df8b3a6611b187a2b0c161b14388413bc670 PiperOrigin-RevId: 206940660
* Automatic code cleanup.Gravatar Googler2018-07-30
| | | | PiperOrigin-RevId: 206567309
* Add a flag to split test.xml generation into a separate SpawnGravatar ulfjack2018-07-27
| | | | | | | | | | At this time, this is only implemented for the StandaloneTestStrategy. This solves a race condition on Posix-like systems, where we cannot guarantee that the pipes are actually fully flushed to disk when the test process exits, and this can cause the test.xml to be empty, which makes it hard to debug issues. (The test.log files do not show up in normal CI systems, only the test.xml files.) Progress on #4608. PiperOrigin-RevId: 206292179
* Create Bazel coverage report action.Gravatar elenairina2018-07-06
| | | | | | | Fixes #5246 RELNOTES: None. PiperOrigin-RevId: 203453340
* [LcovMerger] Make int fields of SourceFileCoverage be dynamically computed.Gravatar elenairina2018-07-04
| | | | | RELNOTES: None. PiperOrigin-RevId: 203264765
* [LcovMerger] Make gcc-branch related fields String instead of int.Gravatar elenairina2018-07-04
| | | | | | These fields can have very large values that don't fit into int. There's also no added value from storing an int instead of a String. PiperOrigin-RevId: 203260628
* Pull in the right deps into the embedded tools.Gravatar Irina Iancu2018-06-14
| | | | | | | | | | | | | | | | | Fixes #5378. $ find $(bazel-bin/src/bazel info install_base) -type f | wc -l 508 $ stat -c %s bazel-bin/src/bazel 115014201 The number of embedded files is now 508 and the binary size is ~115Mb. RELNOTES: None. Change-Id: I75e43a959e1201cc36718a9acd0af96e9506c7d4 PiperOrigin-RevId: 200539940
* Fix some missing cases from ↵Gravatar Googler2018-06-12
| | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/bbf3e421ed8b2b431a72cd3ab4ba591dc8833634. For languages that use launchers, there are some requirements about files other than the main executable being co-located with the executable: 1. For Python, the .zip file must be alongside the executable. 2. For Shell, the shell file must be alongside the exe. In addition, a .exe suffix is needed for launchers. Tested that cc_test (non-launcher), java_test, py_test, sh_test (all launchers) work remotely when using the TEST_SHORT_EXEC_PATH env var. RELNOTES: N/A PiperOrigin-RevId: 200216308
* Implement LcovMerger.Gravatar elenairina2018-06-11
| | | | | | | | | | | | | | LcovMerger is a tool that merges all the intermediate lcov tracefiles (with .dat extension) found under a coverage directory and prints the merged tracefile to a given output file. A custom implementation for merging lcov tracefiles is needed because the merging functionality of lcov itself is very slow. LcovMerger is required to get a single coverage report (lcov tracefile) from a bazel coverage command that executes multiple tests. ATM LcovMerger is only invoked by tools/test/collect_coverage.sh that collects and merges the tracefiles from a single test invocation. It will also be used from a CoverageReportAction. Progress on #5246. PiperOrigin-RevId: 200054506
* Add the ability to force the path to the test binary to a fixed length by ↵Gravatar Googler2018-06-11
| | | | | | | | | | | using a symlink. This is enabled with --test_env=TEST_SHORT_EXEC_PATH=true. Passing CI run: https://buildkite.com/bazel/google-bazel-presubmit/builds/3576 RELNOTES: N/A PiperOrigin-RevId: 200050318
* Portability with BSD stat in collect_coverage.shGravatar Siddhartha Bagaria2018-05-17
| | | | | | | | BSD stat does not have --printf option. Closes #5162. PiperOrigin-RevId: 196965302
* Rewrite lcov path munging to make the paths all relativeGravatar Ulf Adams2018-05-16
| | | | | | | | | | | | | This makes coverage work with sandboxing, and gets rid of a previous workaround. The only downside is that it requires running genhtml in the exec root in order to be able to resolve relative pahts to source files, like so: (GENHTML_CMD="genhtml -o $(pwd)/report/ -p "$(readlink -f bazel-project)" $(readlink -f bazel-testlogs)/path/to/coverage.dat" && cd bazel-project && $GENHTML_CMD) Closes #5043. PiperOrigin-RevId: 196814568
* Make runfiles usage on Windows more flexible to support remote execution.Gravatar Googler2018-04-30
| | | | | | | | | | | | | | | When trying to find a runfile on Windows: 1. First look for the runfiles MANIFEST and find runfile locations using this if it exists (current behavior). 2. If no MANIFEST file exists, look for runfiles in the runfiles directory (new behavior). As part of this, remove setting RUNFILES_MANIFEST_ONLY for the benefit of test-setup.sh. Instead of telling it what to do, it decides what to do based on the observed state of the world. Launchers still set RUNFILES_MANIFEST_ONLY for the benefit of launched programs, since some may depend on this. Fixes https://github.com/bazelbuild/bazel/issues/4962. RELNOTES: Remote execution works for Windows binaries with launchers. PiperOrigin-RevId: 194785440
* Prototype support for llvm coverageGravatar Ulf Adams2018-04-26
| | | | | | | | | | | | | | | https://clang.llvm.org/docs/SourceBasedCodeCoverage.html Closes #5036. Usage with statically linked tests (works with clang 3.9 or later): BAZEL_USE_LLVM_NATIVE_COVERAGE=1 GCOV=llvm-profdata-3.9 CC=clang-3.9 /tmp/bazel coverage --instrumentation_filter=src/main/cpp --dynamic_mode=off src/test/cpp:option_processor_test (LLVM_COV_CMD="llvm-cov-3.9 show -format=html -output-dir=$(pwd)/report/ -instr-profile bazel-out/k8-fastbuild/testlogs/src/test/cpp/option_processor_test/coverage.dat bazel-out/k8-fastbuild/bin/src/test/cpp/option_processor_test" && cd bazel-os-bazel && $LLVM_COV_CMD) Usage with dynamically linked tests (requires clang 4.0 or later): BAZEL_USE_LLVM_NATIVE_COVERAGE=1 GCOV=llvm-profdata-4.0 CC=clang-4.0 bazel coverage --instrumentation_filter=src/main/cpp src/test/cpp:option_processor_test (LLVM_COV_CMD="llvm-cov-4.0 show -format=html -output-dir=$(pwd)/report/ -instr-profile bazel-out/k8-fastbuild/testlogs/src/test/cpp/option_processor_test/coverage.dat bazel-out/k8-fastbuild/bin/src/test/cpp/option_processor_test $(cat bazel-out/k8-fastbuild/bin/src/test/cpp/option_processor_test.runfiles_manifest | cut -d' ' -f 2 | egrep "\.so$" | xargs -n 1 -I xxx echo -n "-object xxx ")" && cd bazel-os-bazel && $LLVM_COV_CMD) PiperOrigin-RevId: 194357292
* test-setup: remove leading "./" from test nameGravatar Laszlo Csomor2018-04-24
| | | | | | | | | | | Fixes release-blocking regression https://github.com/bazelbuild/bazel/issues/5079 Change-Id: I26202d571d53cd843d559b86d849d29173df4015 Closes #5083. Change-Id: I26202d571d53cd843d559b86d849d29173df4015 PiperOrigin-RevId: 194068956
* Add a workaround for gcov-based coverage with clangGravatar Ulf Adams2018-04-19
| | | | | | | | | | | | | | | | | | - Allow overriding the gcov tool with the GCOV env variable in cc_configure - Symlink the GCOV tool in collect-coverage.sh to a temporary location with the name "gcov" This allows the user to specify GCOV=llvm-cov in the environment of a bazel build, which then leads to cc_configure picking up llvm-cov in the crosstool, which the collect-coverage.sh script then uses as "gcov". On linux distributions, the gcov tool does not generally work with clang coverage output, so this provides at least a workaround for running coverage with clang (by setting the GCOV env variable). Closes #5040. PiperOrigin-RevId: 193487773
* Ignore missing .gcno files in coverage collectionGravatar Ulf Adams2018-04-16
| | | | | | | | | | Bazel uses an instrumentation filter to collect coverage from a subset of source files. Without this flag, lcov fails if the instrumentation filter does not cover everything in the source tree (i.e., --instrumentation_filter=.*). Some progress on #1118. PiperOrigin-RevId: 193023189
* Add a new test environment variable, TEST_TARGET.Gravatar John Cater2018-04-06
| | | | | | | | | | | | | | | | | | | | Also include the test target in the test log for easier debugging of tests. Fixes #4960. Change-Id: Ic79bf21e8eeba765b607755c1d290ee4e910d01f Closes #4972. Sample output: ``` exec ${PAGER:-/usr/bin/less} "$0" || exit 1 Executing tests from //src/test/shell/bazel:bazel_example_test ----------------------------------------------------------------------------- ``` Change-Id: I433cd9251a564b269da5566bb8401151ae4c99b7 PiperOrigin-RevId: 191891278
* report test duration in XML using attribute "time"Gravatar Shimin Guo2018-03-07
| | | | | | | | Fixes #4738 Closes #4764. PiperOrigin-RevId: 188167356
* Fix test name in fallback xml when --run_under is used.Gravatar Benjamin Peterson2018-02-22
| | | | | | | | | | | When --run_under is used, the run-under wrapper will be injected in front of the normal test command line. So, it's incorrect to use the beginning of the command line to infer the test name. However, we can always get the underlying test executable with the TEST_BINARY environmental variable. Rename TEST_NAME to EXE to avoid future confusion. Fixes https://github.com/bazelbuild/bazel/issues/4588. Change-Id: I6fd05cca5e5441c13ee16290c4028ec84adec983 PiperOrigin-RevId: 186590311
* Roll forward of ↵Gravatar olaola2018-02-20
| | | | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/3904ac33a983fd8faebba1b52bcac5a3ff942029 (https://github.com/bazelbuild/bazel/commit/3904ac33a983fd8faebba1b52bcac5a3ff942029). Fix #4625 by running the test process in a sub-shell. Apparently, nested background processes interfere with SIGINT handling in bash. I don't 100% understand why and how, but I do have a small bash script that demonstrates the problem: script A that spawns a background process, sends it a SIGINT, and verifies it was received. The script works, *unless* run in the background by a process B; this extra layer of backgrounding cause process A's logic to stop working. See experimental/users/olaola/shell/ for examples. See also https://stackoverflow.com/questions/48847722/nested-background-processes-and-sigint-handling *** Original change description *** Fixing test-setup.sh occasionally missing stdout/stderr, on systems where "tail --pid" is supported. The solutions aren't mine, the new test was taken from Ola's unknown commit and the way to avoid race condition courtesy of sethkoehler@ Mitigates #4608 for compatible Linux systems. TESTED=presubmits, manual shell tests on new bazel RELNOTES: None PiperOrigin-RevId: 186312008
* Automated rollback of commit 28bd997c1c8793973f63dcae4c22bbae49e7d8b7.Gravatar Googler2018-02-12
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** https://github.com/bazelbuild/bazel/issues/4625 What I thought was a short fix is turning into a long hunt, so I better roll this back to get the build green again. I'm not yet 100% certain what the interactions are, but there's a chance that it's back to the drawing board. *** Original change description *** Fixing test-setup.sh occasionally missing stdout/stderr, on systems where "tail --pid" is supported. The solutions aren't mine, the new test was taken from Ola's unknown commit and the way to avoid race condition courtesy of sethkoehler@ Mitigates #4608 for compatible Linux systems. TESTED=manual scripts and new test case. RELNOTES: None PiperOrigin-RevId: 185482604
* Fixing test-setup.sh occasionally missing stdout/stderr, on systems where ↵Gravatar Googler2018-02-12
| | | | | | | | | | | | "tail --pid" is supported. The solutions aren't mine, the new test was taken from Ola's unknown commit and the way to avoid race condition courtesy of sethkoehler@ Mitigates #4608 for compatible Linux systems. TESTED=manual scripts and new test case. RELNOTES: None PiperOrigin-RevId: 185374273
* Remove hard-wired basenames for the test setup and the coverage collection ↵Gravatar lberki2018-02-09
| | | | | | | | | script. Accomplished by creating an explicit attribute for both of them on test actions. RELNOTES: None. PiperOrigin-RevId: 185132460
* windows,test: add logging to bootstrap testGravatar Laszlo Csomor2018-01-29
| | | | | | | | | | | | | | | Add logging to bazel_bootstrap_distfile_test in case it is running on Windows. The logging will help collect basic performance stats. Also remove the %N placeholder from `date` format string in the log messages, because macOS doesn't support it. See https://github.com/bazelbuild/bazel/issues/4503 Change-Id: Idf00bf1512d02a793b27e1cc761fbcd630e79618 PiperOrigin-RevId: 183642578
* HOME missing from the action env (despite mentioned in the docs)Gravatar Gursewak2018-01-16
| | | | | | | | Issue #4401 Closes #4437. PiperOrigin-RevId: 182067976
* Fix some broken targets and failing tests.Gravatar ajmichael2017-12-06
| | | | | RELNOTES: None PiperOrigin-RevId: 178099410
* Revert name of coverage_suppot to coverage_support.Gravatar cparsons2017-11-29
| | | | | | | The previous change was accidental. RELNOTES: None. PiperOrigin-RevId: 177330228
* Create the output source jar in java_common.compileGravatar elenairina2017-11-29
| | | | | | | | | | | | | | | | | and expose transitive source jars to Skylark. Initial change was rolled back (unknown commit). java_common.compile wouldn't fill the JavaRuleOutputJarsProvider when there was only one input source jar. Fixed that and added test (there was one before, but didn't check this particular provider). Slightly refactor java classes to take in specific host javabase inputs and host java executable for creating the source jar, instead of always relying on fetching them from native java rules specific attributes. Creating the output source jar in java_common.compile makes the behavior more similar to java_library. Exposing the transitive_source_jars to Skylark helps with the Skylark migration from the old 'java' provider to JavaInfo. Progress on #2614. RELNOTES: transitive_source_jars is now exposed on JavaInfo. PiperOrigin-RevId: 177311138
* Handle spaces in absolute paths to tests.Gravatar Seth Greenstein2017-11-29
| | | | | | | | | | | | | | | | | | | | | | This allows for output_base paths that contain spaces, fixing #3897 The test path is set by grepping the MANIFEST, in which each line is a space-separated 2-tuple of relative path and absolute path to the test. Replace the awk command that extracts the second item in the space-separated list with a sed command that extracts everything after the first space. Also, quote the manifest path arg to the grep command. There will still be problems if there are spaces in the relative path, below the Bazel project root. Test: Built and tested a test with an output_base containing a space. Change-Id: I5dba285bb50c27eedace17a5e20efac7392a9a6e Closes #3899. Change-Id: I5dba285bb50c27eedace17a5e20efac7392a9a6e PiperOrigin-RevId: 177283695
* Automated rollback of commit 0ebb3e54fc890946ae6b3d059ecbd50e4b5ec840.Gravatar dmarting2017-11-22
| | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Cause rare case of test hanging. To cherry-pick for #3772 *** Original change description *** Attempting to fix an occasionally missing stdout from test.xml. In hello-world_test, when executed inside a docker container, for about 2% of the runs the test.xml has an empty CDATA, instead of the expected "Hello, world!". I'm not sure still what exactly was the bug, but in any case this change simplifies the test execution code line, so if this doesn't fix it, at least further debugging will be easier. I ran the test in a loop 1360 times, and the error did not reproduce once, which hints to... *** PiperOrigin-RevId: 176662654
* Removing the USER variable from test environment, unless explicitly provided.Gravatar olaola2017-10-16
| | | | | | TESTED=added remote test RELNOTES: Fixes #2574 PiperOrigin-RevId: 172294781
* Attempting to fix an occasionally missing stdout from test.xml.Gravatar olaola2017-10-06
| | | | | | | | | | In hello-world_test, when executed inside a docker container, for about 2% of the runs the test.xml has an empty CDATA, instead of the expected "Hello, world!". I'm not sure still what exactly was the bug, but in any case this change simplifies the test execution code line, so if this doesn't fix it, at least further debugging will be easier. I ran the test in a loop 1360 times, and the error did not reproduce once, which hints towards this fixing it, since for an unpatched Bazel it reproduced 7 times out of 293 runs. TESTED=ran Bazel test in a loop RELNOTES: None PiperOrigin-RevId: 170702015
* Don't emit an undeclared outputs MANIFEST if there are no files.Gravatar Googler2017-09-25
| | | | | | | Tested manually (with a test that doesn't emit undeclared outputs), and added a test that failed before and passes after the test-setup.sh change. RELNOTES: N/A PiperOrigin-RevId: 169687782
* Add support for writing undeclared outputs to TEST_UNDECLARED_OUTPUTS_DIR ↵Gravatar Googler2017-09-20
| | | | | | | | | | | | | | | | | | | and TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR to Bazel. After this change: - Any files written to the TEST_UNDECLARED_OUTPUTS_DIR directory will be zipped up and added to an outputs.zip file under bazel-testlogs. - Files will be listed in a MANIFEST file under bazel-testlogs. - Any files written to TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR will be concatenated together into an ANNOTATIONS file under bazel-testlogs. This provides a channel for tests to provide extra information outside of the test output itself. This is useful for things like verbose server logs. Note: The //src/test/shell/bazel:bazel_test_test target has a pre-existing breakage (see https://github.com/bazelbuild/bazel/issues/3727). But the new tests pass. RELNOTES: Tests can now write files to TEST_UNDECLARED_OUTPUTS_DIR and TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR and these will be reflected under bazel-testlogs. PiperOrigin-RevId: 169282528
* Set correct PWD in coverage ↵Gravatar Mansheng Yang2017-09-15
| | | | | | | | (https://docs.bazel.build/versions/master/test-encyclopedia.html) mentioned that when running tests the `PWD` should be `$TEST_SRCDIR/workspace-name` Closes #3616. PiperOrigin-RevId: 168806570
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* test-setup.sh: don't cat the log file if it does not existsGravatar dmarting2017-08-08
| | | | | | This should fix flakiness of test_flaky_test PiperOrigin-RevId: 164467764
* Rollforward of commit b8514f533d4546d3bfbec3700012f2bbeffd1c37Gravatar Damien Martin-Guillerez2017-07-27
| | | | | | | | | | | Add stdout to default XML file and generate XML file on timeout This time corretly quote XML output: - Remove invalid UTF-8 characters - Wrap stdout in a CDATA Change-Id: I70bf38641d1946418e7b8e3b4e44ba0820e7b3c6 PiperOrigin-RevId: 163317477
* Automated rollback of commit b8514f533d4546d3bfbec3700012f2bbeffd1c37.Gravatar dmarting2017-07-20
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks XML format when there is weird characters in the output (Jenkins fails to read XML output for re2 test) *** Original change description *** Add stdout to default XML file and generate XML file on timeout This should fix #1027 and get better error result on Jenkins. Change-Id: I5ce30b64f634e01dd350af10748c4a9455a6bea8 PiperOrigin-RevId: 162598130
* Add stdout to default XML file and generate XML file on timeoutGravatar Damien Martin-Guillerez2017-07-19
| | | | | | | This should fix #1027 and get better error result on Jenkins. Change-Id: I5ce30b64f634e01dd350af10748c4a9455a6bea8 PiperOrigin-RevId: 162474168
* Ensure shards of the same test have unique names in the fallback xmlGravatar Benjamin Peterson2017-06-28
| | | | | | | | | | | | | test-setup.sh currently uses the test target name as the "name" attribute in the xml output's "testcase" element. This ensures names in the xml are unique across the workspace for unsharded tests. However, sharded tests all end up with the same "name" attribute. This angers some junitxml processing tools, which can be quite persnickity about uniqueness. Let's generate unique test names of the form "mytest_shard_N/M" for sharded tests. Change-Id: I6ea82f56846b88449da977dc6629a73b695a485e PiperOrigin-RevId: 160397341
* give the duration of test in the fallback xml fileGravatar Benjamin Peterson2017-06-27
| | | | | | | | | A common extension to junitxml is to provide the duration of each test case in seconds. Here, we add the duration to the xml file that test-setup.sh provides if the test driver doesn't generate one. Change-Id: I25f94f10bb99db7078919f3e0bcdf7bd3cbb637d PiperOrigin-RevId: 160252993
* Use the correct JVM for LcovMerger instead of whatever "java" gets us.Gravatar lberki2017-06-09
| | | | | | | Fixes #2904. RELNOTES: None. PiperOrigin-RevId: 158516169
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-02 21:54:34 +0000