aboutsummaryrefslogtreecommitdiffhomepage
Commit message (Collapse)AuthorAge
* Implement the first stage of Bazel's "Sandbox 2.0" for Linux.Gravatar Philipp Wollermann2016-08-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has the following improvements upon the older one: - Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for further process isolation and 100% reliable killing of child processes. - Uses clone() instead of unshare() to work around a Linux kernel bug that made creating a sandbox unreliable. - Instead of mounting a hardcoded list of paths + whatever you add with --sandbox_add_path, this sandbox instead mounts all of /, except for what you make inaccessible via --sandbox_block_path. This should solve the majority of "Sandboxing breaks my build, because my compiler is installed in /opt or /usr/local" issues that users have seen. - Instead of doing magic with bind mounts, we create a separate execroot for each process containing symlinks to the input files. This is simpler and gives more predictable performance. - Actually makes everything except the working directory read-only (fixes #1364). This means that a running process can no longer accidentally modify your source code (yay!). - Prevents a number of additional "attacks" or leaks, like accidentally inheriting file handles from the parent. - Simpler command-line interface. - We can provide the same semantics in a Mac OS X sandbox, which will come in a separate code review from yueg@. It has the following caveats / known issues: - The "fallback to /bin/bash on error" feature is gone, but now that the sandbox mounts everything by default, the main use-case for this is no longer needed. The following improvements are planned: - Use a FUSE filesystem if possible for the new execroot, instead of creating symlinks. - Mount a base image instead of "/". FAQ: Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox useless? A: This is still a reasonable behavior, because the sandbox never tried to isolate your build from the operating system it runs in. Instead it is supposed to protect your data from a test running "rm -rf $HOME" and to make it difficult / impossible for actions to use input files that are not declared dependencies. For even more isolation the sandbox will support mounting a base image as its root in a future version (similar to Docker images). Q: Let's say my process-specific execroot contains a symlink to an input file "good.h", can't the process just resolve the symlink, strip off the file name and then look around in the workspace? A: Yes. Unfortunately we could not find any way on Linux to make a file appear in a different directory with *all* of the semantics we would like. The options investigated were: 1) Copying input files, which is much too slow. 2) Hard linking input files, which is fast, but doesn't work cross- filesystems and it's also not possible to make them read-only. 3) Bind mounts, which don't scale once you're up in the thousands of input files (across all actions) - it seems like the kernel has some non-linear performance behavior when the mount table grows too much, resulting in the mount syscall taking more time the more mounts you have. 4) FUSE filesystem, good in theory, but wasn't ready for the first iteration. RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system. -- Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063 Reviewed-on: https://bazel-review.googlesource.com/#/c/4053 MOS_MIGRATED_REVID=130638204
* Fix tmpdir with standalone test strategyGravatar Kristina Chodorow2016-08-18
| | | | | | | | | | | | | | | | | If the tmpdir wasn't below the execroot, Bazel would crash and print a stack trace. Also, the test was incorrect because the EOF wasn't quoted, so it was actually just executing "echo TEST_TMPDIR=/path/to/tmpdir" which (surprise surprise) matched TEST_TMPDIR=/path/to/tmpdir. Finally, the test was _also_ incorrect because it was using the cached test result for the second case, since changing the tmpdir doesn't invalidate the test result. So it not only was comparing a constant string to a constant string, but it wasn't even re-evaluating the constant string. -- MOS_MIGRATED_REVID=130637221
* Fixes order of keyword arguments in the error message printed when a JavaGravatar Pedro Liberal Fernandez2016-08-18
| | | | | | | | | method cannot be found when called from Skylark. PAIR=laurentlb -- MOS_MIGRATED_REVID=130636387
* Use -fno-canonical-system-headers when supportedGravatar Brian Silverman2016-08-18
| | | | | | | | | | This makes the same change to cc_configure.bzl made to the hard-coded CROSSTOOL in 763f1397155fc7c12e1f1071a1bc942f91b867c4 to fix #1642 -- Change-Id: Ia05788243d0128ea4921f12c437ef6cebd83a4ff Reviewed-on: https://bazel-review.git.corp.google.com/#/c/5450/ MOS_MIGRATED_REVID=130636065
* Add a test for sandboxing when nobody's UID is differentGravatar Brian Silverman2016-08-18
| | | | | | | | | | | | | | | | Please don't merge before I8b8c3596fd2a4c125071406eefd407ca402099dd. The test will fail then. I've seen two issues with this functionality breaking: #481 and #1637. Seems like it's time to add a test which catches the issue on the Debian/Ubuntu machines people commonly test on. The test fails on my Debian system before I8b8c3596fd2a4c125071406eefd407ca402099dd, and passes with that change applied. -- Change-Id: Ib785c874cdb9192920f9935b696bfd6c9c0e5f4f Reviewed-on: https://bazel-review.git.corp.google.com/#/c/5451/ MOS_MIGRATED_REVID=130635565
* call SetupDirectories after uids are mapped in the user namespaceGravatar Jason Lunz2016-08-18
| | | | | | | | | fixes #1637 (https://github.com/bazelbuild/bazel/issues/1637) -- Change-Id: I8b8c3596fd2a4c125071406eefd407ca402099dd Reviewed-on: https://bazel-review.git.corp.google.com/#/c/5410/2 MOS_MIGRATED_REVID=130633667
* Fix typos: min_rank -> minrank, max_rank -> maxrankGravatar Googler2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130631646
* Stop catching InterruptedException in remote execution.Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | | | | | | If the user interrupts the build, they probably don't want local fallback to happen. Also clean up an unused method and parameter. -- Change-Id: I6bf80fa110bbba911b0743f24c25240c208c98d1 Reviewed-on: https://bazel-review.googlesource.com/5470 MOS_MIGRATED_REVID=130612791
* Log gRPC interrupt events.Gravatar Lukacs Berki2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130609319
* Use array comparison and not reference comparison between digests.Gravatar Shreya Bhattarai2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130576075
* Delete unused empty file.Gravatar Adam Michael2016-08-18
| | | | | | | The filegroup using this file was deleted in commit c9d4fae25137cf480e83566d992bef813fcad7ee -- MOS_MIGRATED_REVID=130553843
* Fix NPE when aspect propagates over an alias.Gravatar Googler2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130553300
* Replaces sh_binary redirects with aliases in Bazel.Gravatar Adam Michael2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130549870
* Implement the fully link action in the OSX crosstool.Gravatar Cal Peyser2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130547971
* Restore objc -> cc_inc_library deps, add regression test.Gravatar Cal Peyser2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130546999
* java_(lite_)?proto_library to get Java proto runtimes from //external: ↵Gravatar Carmi Grushko2016-08-18
| | | | | | | instead of hard-coded values. -- MOS_MIGRATED_REVID=130543727
* Implement --compression and --dont_change_compression.Gravatar Sasha Smundak2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130541462
* RELNOTES: The string list returned by the skylark 'glob' function is now ↵Gravatar Nathan Harmata2016-08-18
| | | | | | | | | | | | | | sorted. Previously, it would return a list formed by concatenating the sorted results of each pattern in the 'includes' list. A bunch of cleanups and one bug fix: -Remove the unused-except-for tests GlobCache#globsUpToDate. This code has been dead for a very very long time, ever since we switched to using Skyframe. -Change the semantics of the 'glob' function as described above. -Change UnixGlob to return unsorted results. Document this in UnixGlob and GlobCache. -Change LegacyGlobber to conditionally return sorted results. Have users other than PackageFunction get sorted results (as described above). Have PackageFunction's use case get completely unsorted results, and have PackageFunction do the sorting itself. -Have PackageFunction's HybridGlobber unconditionally sort the glob result list. This ensure deterministic glob results, fixing a bug where the order of the elements of the result depended on the contents of the Skyframe graph, which of course depends on the sequence of incremental Blaze commands. -- MOS_MIGRATED_REVID=130540152
* Remove unnecessary wrapping of InterruptedException in SkyQueryEnvironment.Gravatar Janak Ramakrishnan2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130535008
* Check that dependencies are non-null upon insertion.Gravatar Carmi Grushko2016-08-18
| | | | | | | Otherwise, we'll get an NPE in build(), which doesn't help in finding the place where the null was added. -- MOS_MIGRATED_REVID=130531765
* Propagate DEVELOPER_DIR environment variable to the apple release bundling ↵Gravatar Chris Parsons2016-08-18
| | | | | | | codesign action. -- MOS_MIGRATED_REVID=130530871
* Allow SkyframePackageRootResolver's methods to throw interrupts, now that ↵Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | the interface allows it. -- MOS_MIGRATED_REVID=130530262
* Set the LibraryIdentifier before building the LTO Indexing link actionGravatar Googler2016-08-18
| | | | | | | | | | Otherwise we fail the Precondition check added at the start of CppLinkAction build() for a shared library target's indexing action. Added a test which reproduces the issue without the fix. -- MOS_MIGRATED_REVID=130513291
* Improve BlazeModule documentation.Gravatar Ulf Adams2016-08-18
| | | | | | | Also allow AbruptExitException from all server startup hooks. -- MOS_MIGRATED_REVID=130513167
* Remove the awkward logic that used to look at the string form at a command ↵Gravatar Lukacs Berki2016-08-18
| | | | | | | line option to determine if it should be a whole archive one and use the artifact category in LinkerInput to make that decision instead. -- MOS_MIGRATED_REVID=130508699
* Make native deps with embedded runtimes work again. commit ↵Gravatar Lukacs Berki2016-08-17
| | | | | | | 0978a8ea1593ef204ea97155014f76baba2508c5 broke it. -- MOS_MIGRATED_REVID=130496094
* Remove SourcesJavaCompilationArgsProviderGravatar Liam Miller-Cushon2016-08-17
| | | | | -- MOS_MIGRATED_REVID=130469884
* Fix slow hostname lookupGravatar Marcel Hlopko2016-08-17
| | | | | | | | | | | InetAddress.getLocalHost().getHostName() can take seconds to complete as it performs reverse DNS lookup. Prior to this cl hostname lookup was performed on every build, noticeably slowing down null builds especially. This cl caches computed hostname so null builds are faster for the lifetime of the server. -- Reviewed-on: https://bazel-review.googlesource.com/#/c/5432/ MOS_MIGRATED_REVID=130441617
* Share source paths for more compact res proto serializationGravatar Googler2016-08-17
| | | | | | | | | | | | | | | | | | | | | | | Source abs paths can be pretty long. If a value file like colors.xml has N resources then we serialize the path N times. Instead, make a table and just serialize the index. Can reduce resource proto sizes from X to 0.65*X. in some experiments. CPU instructions executed is slightly lower, but critical path impact is pretty minimal since parsing happens in parallel anyway. This doesn't help with drawables (path only shows up once) but doesn't really hurt (an extra index number). I tried sharing the root (a table of the res dirs). That can be another 10%, and helps with the drawable case. However, a naive enumeration of roots (src.getParent().getParent()) added *much* more overhead to the writing stage, so I didn't go on with that. -- MOS_MIGRATED_REVID=130440810
* Open-source java_lite_proto_library.Gravatar Carmi Grushko2016-08-17
| | | | | | | The new rule doesn't currently function - it requires adding a JavaLite protoc plugin to Bazel, and an additional change to this code to load and use it. -- MOS_MIGRATED_REVID=130440693
* Rollback of commit cf00c82a394e5a392a55c058ed01483d2e39c3db.Gravatar Liam Miller-Cushon2016-08-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke exoblaze build This is a partial rollback: changes to mocks, and to allow running java tests with the launcher enabled and disabled were left in. *** Original change description *** Roll forward commit 94c86135d05a1844263c59f3ce6b1c1917e0f4c8 And don't provide a default value for :java_launcher -- MOS_MIGRATED_REVID=130429620
* Make retrieval of an already loaded target interruptible. There is no reason ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | now to forbid it, since Skyframe lookups are interruptible. -- MOS_MIGRATED_REVID=130429286
* Add defines to experimental_objc_library.Gravatar Cal Peyser2016-08-17
| | | | | -- MOS_MIGRATED_REVID=130424806
* Simplify SkyframeExecutor#evaluateSkyKeyForCodeMigration and allow it to ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | throw InterruptedException. -- MOS_MIGRATED_REVID=130424634
* Preparation for enabling --compression/--dont_change_compression options:Gravatar Sasha Smundak2016-08-17
| | | | | | | Allow combiners to copy the payload out instead of compressing it. -- MOS_MIGRATED_REVID=130419840
* Add a more robust test for cc_library deps that is not confused by anGravatar Cal Peyser2016-08-17
| | | | | | | experimental_objc_library, which may export cc providers. -- MOS_MIGRATED_REVID=130415669
* Test for META-INF/services/* and META-INF/spring.handlers and ↵Gravatar Sasha Smundak2016-08-17
| | | | | | | META-INF/spring.schemas. -- MOS_MIGRATED_REVID=130414929
* Make the proto bundling behavior the default when using the new library. Take 2Gravatar Sergio Campama2016-08-17
| | | | | -- MOS_MIGRATED_REVID=130406840
* Let Bazel use sandboxing during bootstrap, if it's available.Gravatar Philipp Wollermann2016-08-17
| | | | | -- MOS_MIGRATED_REVID=130406085
* In BlazeQueryEnvironment, stop wrapping InterruptedExceptions in ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | | | QueryExceptions now that all relevant methods declare that they throw InterruptedException. Small side benefit of commit 3c0adb26bac6d756fb97e4bcc6d4e5b2cefa5eeb. -- MOS_MIGRATED_REVID=130402917
* Differentiate between regular and alwayslink static libraries in ↵Gravatar Lukacs Berki2016-08-17
| | | | | | | LinkerInput#getArtifactCategory() and use this information to remove one use of LINK_LIBRARY_FILETYPES. -- MOS_MIGRATED_REVID=130400793
* Remove references to ARCHIVE_FILETYPES and ARCHIVE_LIBRARY_FILETYPES from ↵Gravatar Lukacs Berki2016-08-17
| | | | | | | our C++ rules (except from precondition checks) -- MOS_MIGRATED_REVID=130396421
* Handle public tags in merger -> R.class writerGravatar Googler2016-08-17
| | | | | | | | | | | | | | | Developers do use public tags, and before this change it would assert in the AndroidResourceClassWriter as an unhandled ResourceType. We probably didn't want to write out a R.public.field anyway. Also, handle public tags with the same name, but different type. They get mapped to the same FQN, so use the combining mechanism to keep track of the different types and ids. -- MOS_MIGRATED_REVID=130395089
* Use the the artifact category in LinkerInput to determine if it's a dynamic ↵Gravatar Lukacs Berki2016-08-17
| | | | | | | library. -- MOS_MIGRATED_REVID=130394540
* Deprecate the ${link page.heading} syntax for referencing BE static pages.Gravatar David Chen2016-08-16
| | | | | -- MOS_MIGRATED_REVID=130374987
* Use Options for BE docgen command line arguments.Gravatar David Chen2016-08-16
| | | | | -- MOS_MIGRATED_REVID=130371455
* Only call StreamObserver.onNext() from a single thread in order to avoid a ↵Gravatar Lukacs Berki2016-08-16
| | | | | | | memory leak in gRPC (or Netty). -- MOS_MIGRATED_REVID=130369785
* Adds proto grouping behavior so that generation and compilation actions are ↵Gravatar Sergio Campama2016-08-16
| | | | | | | much smaller. This adds more granularity and helps prevent excessive compilation by minimizing the number of inputs for each action. -- MOS_MIGRATED_REVID=130359288
* Fix binary attribute documentation for apple1 and apple2 extension rules.Gravatar Chris Parsons2016-08-16
| | | | | -- MOS_MIGRATED_REVID=130330900
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 09:28:25 +0000