aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/shell
Commit message (Collapse)AuthorAge
* Fix StackOverflowError on Windows. Fixes #5730Gravatar buchgr2018-08-09
| | | | | | | | | | | | | | | | | | | We found that with JDK9 and up Bazel would sometimes crash with a StackOverflowError in one of the Command-Accumulator-Thread-* threads. We experimentally found that this error was due to these threads being constrained to a 32KiB stack size. The default stack size for JVM threads on most 64-bit systems is 1MiB (So that's 3% of the default). The purpose of the Command-Accumulator-Threads is to read stdout/stderr from processes that Bazel launches locally. The proposed fix is to just use the system default stack size for these threads. The alternative is to increase the size limit to some arbitrary number that happens to work, but this is likely premature optimization and I'd like to avoid that if possible. We further found that this code even predates Blaze/Bazel and is from 2005. PiperOrigin-RevId: 208009940
* Synchronize on process factory to inhibit ETXTBSYGravatar George Gensure2018-07-10
| | | | | | | | | | | | | | | | | | | | | | | | | | Refocus synchronization mechanism to cope with file descriptor set fork- induced races to more tightly constrain concurrent fork/exec pairs. This problem has been observed in bazel proper repeatedly, exhibiting as the iconic ETXTBSY - Text file busy in wide worker pool builds and tests. Evidence that this was discovered by @buchgr is in the comment and change to the embedded ExecutionService implementation, and the description of the race and the need for the synchronization was lifted from that scope to the JavaSubprocessFactory. This factory is a singleton and represents the gateway to all worker process execution, and serves as the correct lock primitive to ensure that file descriptor sets are not duplicated across forks, which gave rise to this issue. To test this, I demonstrated a reproducer presented at https://bugs.java.com/view_bug.do?bug_id=8068370 with 2.4% of invocations in that pathological case exhibiting the issue. With a functionally equivalent change - synchronizing around a processBuilder.start() call - as the only modification to the reproducer, no further failures of any kind were observed, over several hundred runs. Closes #5556. PiperOrigin-RevId: 203947224
* Roll forward of ↵Gravatar philwo2018-04-16
| | | | | | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix. Big round of sandbox fixes / performance improvements. - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 193031017
* Internal changeGravatar dannark2018-04-04
| | | | PiperOrigin-RevId: 191642942
* Big round of sandbox fixes / performance improvements.Gravatar Philipp Wollermann2018-03-26
| | | | | | | | | | | | | | | | - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 190472170
* Refactor and cleanup the sandboxing code.Gravatar Philipp Wollermann2018-03-23
| | | | | | | | | | | | | - Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty). - Use Bazel's own Path class when dealing with paths, not String or java.io.File. - Move LinuxSandboxUtil into the "sandbox" package. - Remove dead code and unused fields. - Migrate deprecated VFS method calls to their replacements. - Fix a bug in ExecutionStatistics where a FileInputStream was not closed. Closes #4868. PiperOrigin-RevId: 190217476
* Add a "direct" mode to "blaze run" that makes the process being run a directGravatar lberki2018-02-05
| | | | | | | | | | | | | | child of the process where the Blaze client itself was run. Limitations: - Untested on Windows; it should work because ExecuteProgram() is implemented there, too, but since Windows doesn't support exec(), there is at least one process in between Progress towards #2815. RELNOTES[NEW]: The new "--direct_run" flag on "blaze run" lets one run interactive binaries. PiperOrigin-RevId: 184528845
* Provide more execution statistics in the ExecutionStatistics class.Gravatar ruperts2017-12-06
| | | | | | | In particular, provide all of the statistics available from the getrusage() system call. RELNOTES: None. PiperOrigin-RevId: 178162687
* Make ProcessWrapperUtil aware of the execution statistics file, and add new ↵Gravatar ruperts2017-12-05
| | | | | | | ExecutionStatisticsProvider. RELNOTES: None. PiperOrigin-RevId: 178056182
* Add user and system time to CommandResults, and plumb them into SpawnResults.Gravatar ruperts2017-11-29
| | | | | RELNOTES: None PiperOrigin-RevId: 177290508
* Add optional user execution time and system execution time fields to ↵Gravatar ruperts2017-11-06
| | | | | | | TerminationStatus, and also add a TerminationStatus.Builder and tests. RELNOTES: None. PiperOrigin-RevId: 174557303
* Windows, jni: Don't close stdout/stderr in nativeWaitFor functionGravatar Yun Peng2017-10-24
| | | | | | | | | | | | | | | | | | | | | | | | These two close operations were added to work around #1708, but caused #2675. We found the root cause of the hanging problem in #1708 is a race condition when creating Windows processes: When Bazel trys to create two processes, one for a local command execution, one for starting the worker process. The worker process might accidentally inherits handles opened when creating the local command process, and it holds those handles as long as it lives. Therefore, ReadFile function hangs when handles for the write end of stdout/stderr pipes are released by the worker. The solution is to make Bazel native createProcess JNI function explicitly inheirts handles as needed, and use this function to start worker process. Related: http://support.microsoft.com/kb/315939 Fixed https://github.com/bazelbuild/bazel/issues/2675 Change-Id: I1c9b1ac3c9383ed2fd28ea92f528f19649693275 PiperOrigin-RevId: 173244832
* Automatic code cleanup.Gravatar cushon2017-09-15
| | | | PiperOrigin-RevId: 168802886
* Move Subprocess.Factory to a top-level classGravatar ulfjack2017-09-08
| | | | | | | | | | | Also move the implementation of FutureCommandResult to a top-level class. This is in preparation for significantly simplifying the shell library. The plan is to remove the Subprocess abstraction, and have lower-level implementations implement the much simpler FutureCommandResult interface instead. PiperOrigin-RevId: 167844736
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Fix assorted ErrorProne warnings.Gravatar lberki2017-09-06
| | | | | RELNOTES: None. PiperOrigin-RevId: 167574104
* Rename all logger instances to "logger" (instead "LOG" or "log").Gravatar lberki2017-09-05
| | | | | RELNOTES: None. PiperOrigin-RevId: 167505493
* Extract authandtls, buildeventservice, buildeventstream into package-level ↵Gravatar philwo2017-09-04
| | | | | | | | BUILD files. Replace all ":relative" labels with "//absolute:path" labels for easier search & replace. PiperOrigin-RevId: 167500985
* Split "shell", "graph" into their own packages.Gravatar philwo2017-08-29
| | | | PiperOrigin-RevId: 166849610
* Implement timeouts on top of Java ProcessGravatar ulfjack2017-08-10
| | | | PiperOrigin-RevId: 164827022
* Rewrite the Command APIGravatar ulfjack2017-08-10
| | | | | | | | | | | | | | | | | | | | | | | | Important: the simplified API now defaults to forwarding interrupts to subprocesses. I did audit all the call sites, and I think this is a safe change to make. - Properly support timeouts with all implementations - Simplify the API - only provide two flavours of blocking calls, which require no input and forward interrupts; this is the most common usage - provide a number of async calls, which optionally takes input, and a flag whether to forward interrupts - only support input streams, no byte arrays or other 'convenience features' that are rarely needed and unnecessarily increase the surface area - use java.time.Duration to specify timeout; for consistency, interpret a timeout of <= 0 as no timeout (i.e., including rather than excluding 0) - KillableObserver and subclasses are no longer part of the public API, but still used to implement timeouts if the Subprocess.Factory does not support them - Update the documentation for Command - Update all callers; most callers now use the simplified API PiperOrigin-RevId: 164716782
* Simplify LocalSpawnRunnerGravatar ulfjack2017-04-18
| | | | PiperOrigin-RevId: 153444516
* Remove an unused throws clause, and make one more specificGravatar Ulf Adams2017-02-16
| | | | | | -- PiperOrigin-RevId: 147716435 MOS_MIGRATED_REVID=147716435
* SubprocessBuilder: document argv contractGravatar Laszlo Csomor2017-01-16
| | | | | | | | | | | | Make it part of the interface contract that the argv[0] must be either absolute or just a file name. The only caller has been adhering to this contract already, but document it explicitly still. -- PiperOrigin-RevId: 144447641 MOS_MIGRATED_REVID=144447641
* Expose killSubprocessOnInterrupt on Command#executeAsynchronouslyGravatar Michajlo Matijkiw2016-08-30
| | | | | -- MOS_MIGRATED_REVID=131620345
* Use guava Uninterruptibles#joinUninterruptibly utility method instead of ↵Gravatar Janak Ramakrishnan2016-08-22
| | | | | | | rolling our own in Consumers. -- MOS_MIGRATED_REVID=130936036
* Implement timeouts on Windows.Gravatar Lukacs Berki2016-08-22
| | | | | | | Makes #1664 much less acute. -- MOS_MIGRATED_REVID=130750731
* Fix capturing stdin/stdout on Windows.Gravatar Dmitry Lomov2016-07-11
| | | | | | | | | | | 1. Return EOF for streams representing Windows process pipes. 2. Fix the timing of process.close() 3. Un-synchronized reading of stderr and stdout. -- Change-Id: Iec98f45db9984be2c2b066962801cbd3ca60da3f Reviewed-on: https://bazel-review.googlesource.com/#/c/4000/ MOS_MIGRATED_REVID=126910063
* Rollback of commit 1e37a5375f918376c132fa537e25695f673f41b8.Gravatar Dmitry Lomov2016-07-07
| | | | | | | | | | | | | *** Reason for rollback *** Apparently we now try to open output files for the process twice: once when we are constructing the output streams, and the second time when we tell the process to redirect its outputs. This causes the outputs to be empty on Windows *** Original change description *** Do redirection of stdout / stderr in Java instead of reimplementing it in every process wrapper again. -- MOS_MIGRATED_REVID=126801016
* Make Windows process management nicer.Gravatar Lukacs Berki2016-07-06
| | | | | | | | | Subprocesses now get killed if the Bazel server itself is killed and so do their subprocesses. Also implemented Subprocess#close() so that we get a little more control over when the native structures are cleaned up. -- MOS_MIGRATED_REVID=126628000
* Use the subprocess factory that uses the Win32 API on Windows.Gravatar Lukacs Berki2016-07-04
| | | | | | | We should really do something about the mess that is loading our JNI libraries -- io.bazel.EnableJNI is mentioned eight times in the code in various diverse contexts. This change is not the right place to do it, though. -- MOS_MIGRATED_REVID=126570481
* Add an implementation for SubprocessFactory based on the new JNI interface ↵Gravatar Lukacs Berki2016-07-04
| | | | | | | | | | | to Windows process management. With this change, Bazel can build itself using native Windows process management and Ctrl-C works in server mode as expected. Yay! Flipping the flag will come in a separate change that's easy to roll back if need be. -- MOS_MIGRATED_REVID=126408264
* Implement an abstraction layer over java.lang.Process so that the Windows ↵Gravatar Lukacs Berki2016-07-04
| | | | | | | implementation can eventually be plugged in. -- MOS_MIGRATED_REVID=126404913
* Do redirection of stdout / stderr in Java instead of reimplementing it in ↵Gravatar Philipp Wollermann2016-06-30
| | | | | | | every process wrapper again. -- MOS_MIGRATED_REVID=126279021
* Make "bazel info server_pid" work on Windows.Gravatar Lukacs Berki2016-06-29
| | | | | | | | | This is the first actual use of Windows JNI! Also a cleanup of ProcessUtils. Injecting a mock implementation was never used. -- MOS_MIGRATED_REVID=126068832
* Use Thread#isInterrupted() instead of Thread#interrupted() followed by ↵Gravatar Lukacs Berki2016-06-27
| | | | | | | Thread#interrupt() . -- MOS_MIGRATED_REVID=125933542
* Do not try to read from the stdout/stderr streams of destroyed processes.Gravatar Lukacs Berki2016-06-24
| | | | | | | | | According to https://bugs.openjdk.java.net/browse/JDK-4311711 , the behavior is undefined and it reproducibly results in hangs on Windows. This makes Ctrl-C on be able to interrupt Bazel during the execution phase, too. There are no guarantees about actually killing the child processes, though: Process.destroy() apparently leaves some child processes running. Unfortunately, Java doesn't offer a solution to this, so we'll have to resort to native code in some way. -- MOS_MIGRATED_REVID=125758911
* Allow async command invocations to set the "closeOutput" bit.Gravatar Eric Fellheimer2016-02-22
| | | | | -- MOS_MIGRATED_REVID=115063418
* Use absolute paths in ProcessBuilder invocations.Gravatar Dmitry Lomov2016-02-17
| | | | | | | Needed for #276. -- MOS_MIGRATED_REVID=114838538
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* workers: Make sure to wait for worker processes to exit so that they don't ↵Gravatar Philipp Wollermann2015-09-21
| | | | | | | become zombies. -- MOS_MIGRATED_REVID=103541217
* Fix some warnings.Gravatar Ulf Adams2015-09-04
| | | | | -- MOS_MIGRATED_REVID=102332437
* Get rid of unnecessary synchronization. This is a contention point when we ↵Gravatar Nathan Harmata2015-08-26
| | | | | | | have lots of threads. -- MOS_MIGRATED_REVID=101485204
* Some cleanup changes.Gravatar Ulf Adams2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87821306
* Remove shell.Shell. The windows support is unused.Gravatar Han-Wen Nienhuys2015-02-16
| | | | | -- MOS_MIGRATED_REVID=86427416
* Drop deprecated CommandResult.getExitStatusGravatar Han-Wen Nienhuys2015-02-09
| | | | | -- MOS_MIGRATED_REVID=85874067
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-13 14:58:58 +0000