aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/util/io
Commit message (Collapse)AuthorAge
* Console output: switch to file line endingGravatar Klaus Aehlig2018-08-06
| | | | | | | | | | | As nobody uses bazel on a traditional teletyper terminal, we can as well use the line ending of native files instead of terminal/network line endings. While the console output is not meant to be handled by anything but a terminal, this still seems to be done. Change-Id: Ied745eeedaec70572ef4b3a3c716a0cf57dfdfd1 PiperOrigin-RevId: 207511649
* Reimplement AsynchronousFileOutputStream to use a writer threadGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | | | | Background: the original code is implementing the OutputStream interface. Given a sequence of write calls, the code puts each of these writes into a queue. On the other side of the queue is an unbounded thread pool, which takes the writes off the queue one by one, and then does individual blocking writes with a fixed file offset. There are three problems with the original code: 1. Writes are sent to the Kernel one-by-one. Imagine if the incoming writes a single-byte writes, then we do one kernel call for every single byte. This is the worst case. 2. Due to multithreading, the writes can get reordered. In the worst case, the order is reversed, and the kernel flushes each write to disk individually. Since the writes are not aligned to disk block boundaries, each write has to first *read* the block from disk, overwrite a few bytes, and then flush the block back to disk. On a spinning platter, this is the worst possible sequence of operations: write a block, read the same block back, write the same block again, read the same block back, etc., with each operation having to wait for a full disk rotation. Note that this gets worse if there's high thread and / or disk contention, e.g., when running a build system in the background. 3. Due to the unbounded thread pool, it may end up creating a new thread for every single write (possibly as many as one per byte written). This is also the worst case, although it's probably negligible compared to 1+2. Compared to that, this change uses an in-memory buffer before sending writes to the kernel so non-block sized writes are batched, it writes sequentially, and it uses a single thread created at the start. A single thread should be more than able to fully saturate local disk I/O, so multi-threading should ~never be a improvement here. This might look different if we had perfectly aligned writes of an integer multiple of the disk block size to a distributed network file system or a local SSD raid. If you look at the clients of this class, that's definitely not the case: this code is primarily used for local file BEP transports - we wouldn't use local file BEP transports to write to a network file system, we'd use the BES instead. It's also used to write a couple of log files, also all local - otherwise we'd add the data to BEP. These are all unaligned, ~random-sized writes. If we created a lot of these files, then using a thread pool with fewer threads than files and using non-blocking I/O might be an improvement due to the reduction in thread count, but I think it's very unlikely that we'll ever need that complexity. PiperOrigin-RevId: 200694423
* Logging the spawn graph on demand. First step on #4891.Gravatar olaola2018-04-19
| | | | | | | | This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds. TESTED: manually on various rules, including directories. RELNOTES: None PiperOrigin-RevId: 193539034
* tests: add TimestampGranularityUtilsGravatar laszlocsomor2018-03-13
| | | | | | | | | | | | | | Add a new test utility class TimestampGranularityUtils which offers methods to wait for the filesystem timestamp granularity to pass. Such waiting is necessary if a test asserts changes in a file's mtime or ctime. This is a follow-up to comments on https://github.com/bazelbuild/bazel/commit/a31e035fb42aa8db9fd248ef2ebd665a411053b8. PiperOrigin-RevId: 188847729
* tests: fix flaky RecursiveFSTraversalFunctionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | | | | | | Fix the testTraversalOfGeneratedDirectory method in RecursiveFilesystemTraversalFunctionTest that was flaky on Windows. The fix is to wait so that changes to files in the InMemoryFileSystem will have observable effects on the file ctimes. Depends on https://github.com/bazelbuild/bazel/pull/4787 Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4788. PiperOrigin-RevId: 188470080
* tests: fix flaky SkyframeAwareActionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | Fix testCacheBypassingActionWithMtimeChangingInput in SkyframeAwareActionTest by ensuring that enough time elapses between file updates so their effects are observable on the file's ctime. Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4787. PiperOrigin-RevId: 188458542
* Adding async proto or text logging utility class.Gravatar olaola2018-03-06
| | | | | | | WANT_LGTM=buchgr TESTED=unit tests, 500 runs per test RELNOTES: None PiperOrigin-RevId: 188093043
* A couple quality-of-life improvements for Bazel devs, in response to an ↵Gravatar nharmata2018-02-23
| | | | | | | | | | email from philwo@. (i) Only have TimestampGranularityMonitor log the first file of relevance. This reduces log spam, especially in tests, while still maintaining useful information in the logs. (ii) Don't have ExternalFilesHelper log the fact that it encountered an external file when we're in a unit test or an integration test. Tests, especially bazel tests that use external repositories, tend to involve lots of "external" files. RELNOTES: None PiperOrigin-RevId: 186799176
* Pass ctime to the TimestampGranularityMonitorGravatar ulfjack2018-02-05
| | | | | | | | We're now using ctime to detect file changes, so the timestamp granularity monitor should as well. Unfortunately, we currently get nanosecond ctime from Linux, but then only return millis from FileStatus, so this doesn't change the accuracy of the monitor at all. PiperOrigin-RevId: 184536539
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Rename all logger instances to "logger" (instead "LOG" or "log").Gravatar lberki2017-09-05
| | | | | RELNOTES: None. PiperOrigin-RevId: 167505493
* Make the print function output debug messagesGravatar vladmos2017-08-11
| | | | | RELNOTES: The `print` function now prints debug messages instead of warnings. PiperOrigin-RevId: 164880003
* Merge handleOptions into beforeCommandGravatar ulfjack2017-07-17
| | | | | | | Now that we have the options before calling beforeCommand, there's no need for a separate handleOptions method in the BlazeModule API. Remove it. PiperOrigin-RevId: 162002300
* Extract the ansi color escape sequences to an enumGravatar Ulf Adams2017-06-19
| | | | | | | | | | I want to make the color scheme configurable, which requires an abstraction to represent color, so it can be looked up / stored in a map / etc. Closes #2487. Change-Id: I2f8bd0dd19ecd6a243ac9b7acc7be52e59c90021 PiperOrigin-RevId: 159426774
* Also filter stderr when parse_showincludes feature is enabledGravatar Yun Peng2017-06-13
| | | | | | | | | | On Windows, cl.exe generates include information to stdout. But when using the cuda compiler(nvcc.exe) to invoke cl.exe, it's redirected to stderr. So we need to parse the output of stderr as well. Change-Id: I9a763a6eefa531932bf1077ef4155f6bda934240 PiperOrigin-RevId: 158842448
* Internally track changes to stdout / stderr files, so we can avoid ↵Gravatar felly2017-06-07
| | | | | | filesystem operations to check whether they've been written. In the common case, they haven't, and we can avoid checking that. PiperOrigin-RevId: 158172182
* Parse /showIncludes output for MSVC compilerGravatar Yun Peng2017-04-25
| | | | | | | | Instead of parsing .d file generated by wrapper script, we directly parse the output of /showIncludes option. Change-Id: Id94e20a5cb05a494a793fd6a43756d44d27cea8a PiperOrigin-RevId: 154161939
* Add logging to TimestampGranularityMonitor.Gravatar Julio Merino2016-11-02
| | | | | | | | These new log statements help in understanding what files trigger the TimestampGranularityMonitor's wait logic and when the wait is performed. -- MOS_MIGRATED_REVID=137868235
* Do not write terminal output in the experimental UI character-by-character ↵Gravatar Lukacs Berki2016-09-30
| | | | | | | | | (which results in single-character gRPC messages). Fixes #1861. -- MOS_MIGRATED_REVID=134775680
* Rollback of commit 1e37a5375f918376c132fa537e25695f673f41b8.Gravatar Dmitry Lomov2016-07-07
| | | | | | | | | | | | | *** Reason for rollback *** Apparently we now try to open output files for the process twice: once when we are constructing the output streams, and the second time when we tell the process to redirect its outputs. This causes the outputs to be empty on Windows *** Original change description *** Do redirection of stdout / stderr in Java instead of reimplementing it in every process wrapper again. -- MOS_MIGRATED_REVID=126801016
* Do redirection of stdout / stderr in Java instead of reimplementing it in ↵Gravatar Philipp Wollermann2016-06-30
| | | | | | | every process wrapper again. -- MOS_MIGRATED_REVID=126279021
* Rename FileOutErr.get{Output,Error}File to get{Output,Error}Path, because ↵Gravatar Philipp Wollermann2016-06-29
| | | | | | | they actually return a Path and not a File. -- MOS_MIGRATED_REVID=126102820
* Move LoggingTerminalWriter out of testGravatar Klaus Aehlig2016-06-27
| | | | | | | | | ...so that it can be used by the main code base. -- Change-Id: Ibf59d74ab1b0cca9bd9406bb76ee6e71189653bf Reviewed-on: https://bazel-review.googlesource.com/#/c/3903 MOS_MIGRATED_REVID=125950656
* Make LineWrappingAnsiTerminalWriter more efficientGravatar Klaus Aehlig2016-04-15
| | | | | | | | | | ...by only starting the new line, if needed and not already if the last usable character of the line is written. -- Change-Id: I86519389fe64fe74ba9045be07483ce5f55d5e9a Reviewed-on: https://bazel-review.googlesource.com/#/c/3384 MOS_MIGRATED_REVID=119949042
* Add a position-aware wrapper around AnsiTerminalWriterGravatar Klaus Aehlig2016-04-15
| | | | | | | | | | | | | When generating output targeted for a specific terminal width, it is important to know the current position in order to appropriately shorten the message still to be added to the current line. So make it possible to add this functionality to the terminal writer itself, to avoid too many lengthy position computations at call site. -- Change-Id: I03400b9544c32567fc6ea7ab35e742c4ccd7b610 Reviewed-on: https://bazel-review.googlesource.com/#/c/3373 MOS_MIGRATED_REVID=119946982
* Revamp the client/server communication protocol so that it is portable to ↵Gravatar Lukacs Berki2016-03-22
| | | | | | | | | Windows. Progress towards #930. -- MOS_MIGRATED_REVID=117799006
* Make LineWrappingAnsiTerminalWriter a public classGravatar Klaus Aehlig2016-03-01
| | | | | | | | | ...so that it can be used by other event handlers as well. -- Change-Id: Ic7b37f9041aab63611d05bdb5f18e72bb0c8f1d9 Reviewed-on: https://bazel-review.googlesource.com/#/c/3040 MOS_MIGRATED_REVID=116012068
* Add wrapper class around AnsiTerminalWriter that breaks linesGravatar Klaus Aehlig2016-02-29
| | | | | | | | | | | | | | | In order to update Bazel's progress bar, the old one has to be removed first; this requires knowledge about the number of lines it currently uses. For small terminals, this requires us to take line breaks into account. While this normally works well, there are situations where our believe about the terminal width is smaller that the actual width of the terminal, causing the deletion of too many lines. This wrapper class provides a solution for this case by explicitly breaking lines at the given width. -- Change-Id: I9d33e389730568ab8c15ee082594de9b35abb71c Reviewed-on: https://bazel-review.googlesource.com/#/c/3023 MOS_MIGRATED_REVID=115827354
* Add an interface for append-only access to an AnsiTerminalGravatar Klaus Aehlig2016-02-25
| | | | | | | | | | | | Providing only access to this interface rather than handing out a full AnsiTerminal allows a more controlled access. In particular, it will be easier to track the state the terminal will end up in. -- Change-Id: Ic47b14a045ece2a84113698632a61daf7948c3b4 Reviewed-on: https://bazel-review.googlesource.com/#/c/3013 MOS_MIGRATED_REVID=115533760
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Clean up the TestFileOutErr to be less brittle, avoiding the "brutal ↵Gravatar Eric Fellheimer2015-04-27
| | | | | | | overloading" it had been doing previously. -- MOS_MIGRATED_REVID=91979641
* Add a clear() method to FileOutErr.Gravatar Eric Fellheimer2015-04-21
| | | | | -- MOS_MIGRATED_REVID=91605508
* Add OutErr#close, and call that in test strategies.Gravatar Han-Wen Nienhuys2015-02-18
| | | | | | | | This fixes "Text file is busy" when trying to execute log files for local tests just after blaze finishes. -- MOS_MIGRATED_REVID=86597766
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-30 06:27:16 +0000