| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
toolchain API
RELNOTES: None.
PiperOrigin-RevId: 204291210
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
contain a large set of files required for a particular toolchain that are added
to every single compile action. Some exeutors in turn sort all action inputs in
order to properly cache, deduplicate and serialize them.
Sorting the expansion of these middlemen needs to be done only once and a later
sorting together with a bunch of other action input files is much faster (as
Java's TimSort algorithm is effectively linear with mostly sorted inputs).
RELNOTES: None.
PiperOrigin-RevId: 204285426
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change allows local files referenced by the BEP/BES protocol
to be uploaded to a ByteStream gRPC service.
The ByteStreamUploader is now implicitly also used by the BES
module which has a different lifecycle than the remote module.
We introduce reference counting to ensure that the channel is
closed after its no longer needed. This also fixes a bug where
we currently leak one socket per remote build until the Bazel
server is shut down.
RELNOTES: None
PiperOrigin-RevId: 204275316
|
|
|
|
|
|
|
|
|
|
|
| |
- Don't duplicate usedModules into additionalInputs (this shouldn't be
necessary).
- Use ImmutableLists instead of ImmutableSets where possible to reduce memory
consumption.
- Use set operations to make the code more readable.
RELNOTES: None.
PiperOrigin-RevId: 204268489
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First, make the function that wastes user time keep track of how much
processor time has been used so far, and make the function that wastes
system time track overall wall time. This is to minimize the number
of "overhead" system calls, which can tamper with our measurements.
Second, make the loop of the function that wastes user time more costly
so that modern processors don't run it too fast. Otherwise, the overhead
to track used time becomes significant in system time and makes our
tests fail.
This should fix the flakiness observed in our process-wrapper tests
when run on the modern CPUs that the iMac Pros carry.
Tested: I haven't been able to reproduce the flakiness. Running the
spend_cpu_time binary by hand reports very accurate timings for both
system and user time, even when pausing the program while it runs.
RELNOTES: None.
PiperOrigin-RevId: 204264910
|
|
|
|
|
|
|
|
| |
They're different anyway and we will need this to set a different default value
for aquery.
RELNOTES: None
PiperOrigin-RevId: 204259933
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //devtools/blaze/integration:{[]_test_test,gdp_validation_test} and at leats //contentads/supermixer/server:supermixer .
*** Original change description ***
Refactor handling of API generation in JavaPluginInfoProvider
Instead of keeping two copies of state for the API-generating and
non-API-generating cases, create a 'JavaPluginInfo' abstraction to contain all
state for each case, and then keep two copies in the top-level
JavaPluginInfoProvider provider.
This will make it easier and less error-prone to add additional state to the
provider.
PiperOrigin-RevId: 204258844
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
//src/test/shell/bazel:bazel_rules_test now runs
on Windows.
See https://github.com/bazelbuild/bazel/issues/4292
Change-Id: If88daa9c65689ae8363ee83cfd401c3200215c74
Closes #5571.
Change-Id: If88daa9c65689ae8363ee83cfd401c3200215c74
PiperOrigin-RevId: 204256609
|
|
|
|
| |
PiperOrigin-RevId: 204254234
|
|
|
|
|
|
|
|
| |
delay the string conversion till we actually write the manifest file.
This might get some memory savings after adding some in unknown commit.
RELNOTES: None
PiperOrigin-RevId: 204216582
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
than the graph version when that is feasible.
* It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below.
* It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status.
Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied.
A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1.
All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code.
Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse.
PiperOrigin-RevId: 204210719
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
FilesetOutputSymlink whenever available.
In this change I'm simply plumbing the FileArtifactValue we requested within RecursiveFilesystemTraversalFunction to the FilesetOutputSymlink.
This does not work when the targets are output directories (or symlink to output dirs). The main scenarios this happens is when:
1. Fileset depends on the output dir created by a genrule.
2. Fileset depends on a GoAppengineBinary which creates an output dir.
3. Fileset depends on another Fileset in the non-recommended way (Fileset.entry.files = [<another_fileset>]) instead of the recommended way (FilesetEntry.srcdir = <another_fileset>).
RELNOTES: None
PiperOrigin-RevId: 204209612
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 204181375
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 204173447
|
|
|
|
|
| |
RELNOTES:None
PiperOrigin-RevId: 204170706
|
|
|
|
|
|
|
| |
uploader cannot upload a particular file.
RELNOTES: None
PiperOrigin-RevId: 204167372
|
|
|
|
|
|
|
| |
Example workspace: https://github.com/bazelbuild/examples/pull/66/files
RELNOTES: None.
PiperOrigin-RevId: 204162234
|
|
|
|
|
|
|
|
| |
- Change Skydoc to only document exported symbols in the target file, instead of all exported symbols in the transitive dependencies of the target file. This circumvents a prior error scenario where main.bzl could depend on dep.bzl, and both export a rule named ?my_rule?, which would result in a conflict.
- Offer the option to specify whitelisted symbols for which to output documentation, allowing a user to only request documentation for a subset of rules defined in a given file. This allows more granular control of documentation layout.
RELNOTES: None.
PiperOrigin-RevId: 204161197
|
|
|
|
| |
PiperOrigin-RevId: 204160729
|
|
|
|
| |
PiperOrigin-RevId: 204154609
|
|
|
|
|
|
|
| |
The artifact uploaders may need command-level options.
RELNOTES: None
PiperOrigin-RevId: 204151808
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of keeping two copies of state for the API-generating and
non-API-generating cases, create a 'JavaPluginInfo' abstraction to contain all
state for each case, and then keep two copies in the top-level
JavaPluginInfoProvider provider.
This will make it easier and less error-prone to add additional state to the
provider.
PiperOrigin-RevId: 204151605
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 204147228
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Previously we would zip the output of dsymutil and then proceed to unzip it.
Due to the size of dSYM files, this could add a few seconds to all builds
with dSYMs.
- There's no need to have a DsymOutputType or even Info.plist as all we need
is the DWARF symbol file; the dSYM is repackaged later on by the bundler.
This change is synchronized with the CROSSTOOL and wrapped_clang via the
`no_dsym_create_zip` feature, which Bazel sets on the CROSSTOOL to inform
wrapped_clang that no zip file should be created for the dSYM.
PiperOrigin-RevId: 204134986
|
|
|
|
|
| |
RELNOTES:
PiperOrigin-RevId: 204126150
|
|
|
|
| |
PiperOrigin-RevId: 204121958
|
|
|
|
|
|
|
|
|
|
| |
Calculating the throughput of a digest operation and using it to assess whether
I/O is slow or not only makes sense when we're doing sequential I/O. With
parallel hashing, it's expected that individual operations are slow due to how
the scheduler works, but it doesn't mean that the overall progress is slow.
RELNOTES: None.
PiperOrigin-RevId: 204115311
|
|
|
|
|
|
|
|
|
| |
the result of include validation if dotd file scanning (and in turn input
validation) is disabled. Fingerprinting these data structures is costly as they
are large NestedSets.
RELNOTES: None.
PiperOrigin-RevId: 204112075
|
|
|
|
|
|
|
|
|
|
|
| |
This adds support for Unix sockets to Bazel for the
remote http cache. See corresponding issue #5098
for discussion.
RELNOTES: Introduce the --remote_cache_proxy flag,
which allows for remote http caching to connect
via a unix domain socket.
PiperOrigin-RevId: 204111667
|
|
|
|
|
|
|
|
| |
The page https://docs.bazel.build/versions/master/experimental.html is not useful.
The only link is also available via the `Bazel and Apple` section.
RELNOTES: None.
PiperOrigin-RevId: 204110632
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes #5534.
Skylint was operating under assumption that all lvalues have a single
bound identifier, which is not true for `IndexExpression`s like
```
d["foo"] += "bar"
```
As a result, Skylint would crash. This change makes it handle cases without
bound identifiers gracefully.
Ideally `IndexExpression` assigments should be analyzed too, but it's a more
involved change.
Closes #5535.
PiperOrigin-RevId: 204109060
|
|
|
|
|
|
|
|
| |
Until we have more complete support for Python runtime/toolchain configurations, it's useful for the hash tool to support `--action_env`, so the user's PATH can optionally be searched by the `py_binary` wrapper.
Closes #5451.
PiperOrigin-RevId: 204108228
|
|
|
|
|
|
|
| |
seem to be used.
RELNOTES: None.
PiperOrigin-RevId: 204084726
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks non-Rabbit builds, see b/111275650.
*** Original change description ***
Avoid long, duplicated directory structures. In the common case, generated
files are going to be beneath the target that generates them. In this case,
don't duplicated the package's path.
RELNOTES: None.
PiperOrigin-RevId: 204084475
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- MSVC does not have `errx` functions, so use `diag_errx` etc. instead.
- Fix format when trying to print `size_t`, use `%zu` so that the function will handle 32/64-bit `size_t` according to target system automatically.
- Adding/guarding a few includes for MSVC.
- MSVC does not have `ssize_t`, so replace it with `ptrdiff_t`
#2241
/cc @laszlocsomor
Closes #5499.
PiperOrigin-RevId: 204074420
|
|
|
|
|
|
|
|
| |
This observably removes any ill effect of CAS transience.
Closes #5229.
PiperOrigin-RevId: 204010317
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 203977219
|
|
|
|
|
|
|
| |
This will be needed by py_wrap_cc in a follow up CL.
RELNOTES:none
PiperOrigin-RevId: 203964457
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Remove `/D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS`: it disables warnings for deprecated C++ headers (`<stdext/*>`, `<hash_map>` etc.). Only useful for legacy codebase.
- Remove `/J`: `char` is signed according to ISO C++. There is no reason to make `char` unsigned by default.
- Remove default `/Gy`: moved to feature (`smaller_binary`).
- Remove default `/GF`: string pooling, enabled by default with `/O2`.
- Remove `/DDEBUG` in `dbg` mode: conflict with macros defined in some OSS projects such as LLVM. User should use standard `NDEBUG` to detect debug mode.
Implements a few general `feature` to abstract away details about the compiler:
- `treat_warnings_as_errors`: aka `-Werror`.
- `frame_pointer`: aka `-fno-omit-frame-pointer'.
- `determinism`: redact `__TIME__` macros etc.
- `disable_assertions`: allow user to have `assert/DCHECK` back even in `opt` mode.
- `smaller_binary`: common optimization enabled in `opt` mode. Still allow user to disable it in `opt` (such as when using ASAN).
- `ignore_noisy_warnings`: suppress a bunch of warnings most people don't care. If users want to be pedantic, they can still get those warnings back with `--features=-ignore_noisy_warnings`.
Closes #5519.
PiperOrigin-RevId: 203959555
|
|
|
|
|
|
|
| |
This is an intermediate step in migrating to the new Skylark API for the C++ toolchain.
RELNOTES: None.
PiperOrigin-RevId: 203955015
|
|
|
|
|
|
|
|
| |
Currently dSYM information is propagated using a zip file emitted by the linking action which contains a dwarf file and plist. A second action then unzips said file, discards the plist and propagates the dwarf file. This CL is part of a change which makes the link action emit the dwarf file directly.
This change adds logic to the CROSSTOOL and wrapped_clang to temporarily support both the old and new approaches using the presence of DSYM_HINT_DSYM_BUNDLE_ZIP as a signal as to which to use. After this change the new (and temporary) CROSSTOOL feature no_dsym_create_zip will allow suppressing DSYM_HINT_DSYM_BUNDLE_ZIP from being passed. Once the migration is complete we will remove both the signal variable and feature.
PiperOrigin-RevId: 203954795
|
|
|
|
|
| |
Change-Id: Iefeab0de85541b963c6cc262339e73b79f630f8b
PiperOrigin-RevId: 203952823
|
|
|
|
|
|
|
| |
'_cc_toolchain' exists on the ctx being passed to this function. (I broke TAP once because I forgot to add the attribute. Perhaps not that helpful this late in the migration but doesn't hurt.)
RELNOTES: None.
PiperOrigin-RevId: 203947282
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Refocus synchronization mechanism to cope with file descriptor set fork-
induced races to more tightly constrain concurrent fork/exec pairs. This
problem has been observed in bazel proper repeatedly, exhibiting as the
iconic ETXTBSY - Text file busy in wide worker pool builds and tests.
Evidence that this was discovered by @buchgr is in the comment and change
to the embedded ExecutionService implementation, and the description of
the race and the need for the synchronization was lifted from that scope
to the JavaSubprocessFactory. This factory is a singleton and represents
the gateway to all worker process execution, and serves as the correct
lock primitive to ensure that file descriptor sets are not duplicated
across forks, which gave rise to this issue.
To test this, I demonstrated a reproducer presented at
https://bugs.java.com/view_bug.do?bug_id=8068370
with 2.4% of invocations in that pathological case exhibiting the issue.
With a functionally equivalent change - synchronizing around a
processBuilder.start() call - as the only modification to the reproducer,
no further failures of any kind were observed, over several hundred runs.
Closes #5556.
PiperOrigin-RevId: 203947224
|
|
|
|
|
|
|
|
|
| |
Add to the documentation that version_file and release_file cannot be used to generate an output with the N-V-R.A naming scheme.
Fixes #5522.
Closes #5555.
PiperOrigin-RevId: 203946762
|
|
|
|
|
|
|
|
|
|
|
|
| |
//src/test/shell/bazel:bazel_build_event_stream_test
now runs on Windows.
See https://github.com/bazelbuild/bazel/issues/4292
Closes #5560.
Change-Id: Ica3d59fb6f557c85d2fd646cf005ffd831046119
PiperOrigin-RevId: 203945734
|
|
|
|
|
|
| |
RELNOTES:
--experimental_shortened_obj_file_path is removed.
PiperOrigin-RevId: 203944948
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
compose a nested set encompassing both). The current prunableHeaders were used
in various places, at least two of which let to a duplicate iteration through
the comparatively large declaredIncludeSrcs: CppCompileAction.computeKey() and
CppCompileAction.getAllowedDerivedInputs().
Also do some other minor optimizations:
- CcCompileActionBuilder.buildAllInputs() now just returns a NestedSet with the
same order as its parameter. As inputsForValidation is almost always empty,
this makes this NestedSet pass the underlying one through.
- CcCompilationContext.directModuleMaps is just a list of the module maps of
direct dependencies. As such, there is no use for keeping them in a
NestedSet.
RELNOTES: None.
PiperOrigin-RevId: 203938011
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede updated files to use
try-with-resources when dealing with streams.
The change also got rid of asByteSource, replacing
it with throw-away ByteSource instances wrapping a
try-with-resources-guarded InputStream. Doing so
was unnecessary though, because all ByteSource
methods (except for open[Buffered]InputStream)
close the stream. Thanks to @jbduncan to point
that out and explain in detail [1].
[1] see comment thread on https://github.com/bazelbuild/bazel/commit/59f17d6e0550bf63a0b6ef182e2d63474e058ede under `FilesetManifest.java`
RELNOTES: none
PiperOrigin-RevId: 203934830
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 203934582
|