aboutsummaryrefslogtreecommitdiffhomepage
path: root/infra
Commit message (Collapse)AuthorAge
* [CIFuzz] Give testcases unique names. (#5424)Gravatar jonathanmetzman2021-03-17
|
* Go coverage : use gofuzz tag by default (#5425)Gravatar Catena cyber2021-03-17
| | | as is done by gofuzz build
* Golang coverage summary updated to version 2.0.1 (#5346)Gravatar Catena cyber2021-03-17
|
* update afl++ commit id (#5402)Gravatar van Hauser2021-03-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * output afl++ setup * update commit id * update afl++ commit id * asan + cmplog fix * update commit id * update and enhance afl++ * update afl++ commit id, better run asan options * fix linter * add debug_afl script * Update debug_afl * Update compile_afl * fix for karchive * put debug_afl in the docker container * asan poison fix * fix asan settings for zeek * update afl++ commit id * fix * update afl++ commit id * final touches Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
* Golang coverage: use absolute paths (#5407)Gravatar Catena cyber2021-03-16
|
* [CIFuzz] Fix issue deleting corpus (#5391)Gravatar jonathanmetzman2021-03-15
| | | | | Sometimes deleting the corpus in free_disk_if_needed exceptions. Pass ignore_errors=True to fix this. Related: #5383
* Rust coverage : do not trigger debug assertions (#5385)Gravatar Catena cyber2021-03-14
|
* Golang coverage: build binary instead of bash script (#5377)Gravatar Catena cyber2021-03-14
|
* More generic target directory for rust coverage (#5366)Gravatar Catena cyber2021-03-12
| | | | | * More generic target directory for rust coverage * Rust coverage cargo does not change PATH
* [CIFuzz] Add functionality to save diskspace (#5342)Gravatar jonathanmetzman2021-03-12
| | | | | | | | | | | * [CIFuzz] Add functionality to save diskspace. Add a LOW_DISK_SPACE env/config var. When this is specified (always true for Github actions) run_fuzzers will delete base-builder and the project builder image before fuzzing. After it finishes fuzzing with a target, it will also delete the targets, its seed corpus and its corpus. Related: #4879
* Don't rpath patch llvm-symbolizer. (#5357)Gravatar Abhishek Arya2021-03-10
| | | Should fix https://github.com/google/oss-fuzz/issues/5349
* Fix unzip regression on base-runner. (#5356)Gravatar Abhishek Arya2021-03-10
| | | | | | | | | | | * Fix unzip regression on base-runner. * Dont rpath patch llvm-symbolizer. Should fix https://github.com/google/oss-fuzz/issues/5349 * Revert "Dont rpath patch llvm-symbolizer." This reverts commit fdd881dea252fa2da655eb8c88dd4057db6f0215.
* [base-builder] Remove unnecessary python files (58 MB) (#5348)Gravatar jonathanmetzman2021-03-09
| | | Related: #5170
* Rust coverage : source remap after changing dir (#5345)Gravatar Catena cyber2021-03-09
|
* [base-builder] Delete unneeded files from atheris install (#5344)Gravatar jonathanmetzman2021-03-09
|
* [infra] Remove minijail. (#5343)Gravatar jonathanmetzman2021-03-09
| | | | We don't need minijail tooling anymore since we aren't using it on OSS-Fuzz.
* [base-runner] Reduce size by ~200 MB by using apt better (#5339)Gravatar jonathanmetzman2021-03-09
| | | | Don't install recommended packages if it means installing an entire gcc toolchain. We don't need it in the runner.
* [CIFuzz][cifuzz-base] Slim down cifuzz-base (#5288)Gravatar jonathanmetzman2021-03-09
| | | | | | Reduce cifuzz-base size from 846MB to 444MB. 1. Don't install parts of docker that aren't necessary. 2. Use .dockerignore properly.
* Golang coverage ok with package in $SRC (#5325)Gravatar Catena cyber2021-03-09
|
* fix afl++ asan options (#5336)Gravatar van Hauser2021-03-09
|
* Golang coverage with go 1.16 and modules (#5321)Gravatar Catena cyber2021-03-08
| | | | | | | | * Fixes go coverage with modules * Golang coverage html report turning off modules Otherwise, we get the error working directory is not part of a module
* [base-runner] Fix go issue and remove uneeded rust stuff. (#5319)Gravatar jonathanmetzman2021-03-08
| | | | | Go needs the toolchain for now, so add it back to base-runner. We don't actually need the rust toolchain so get rid of it (saving about 1GB).
* Rust coverage report (for Suricata) (#4697)Gravatar Catena cyber2021-03-08
| | | | | * Rust coverage test * Workaround to get rust coverage for Suricata
* add asan poison check to afl++ cmplog (#5311)Gravatar van Hauser2021-03-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * output afl++ setup * update commit id * update afl++ commit id * asan + cmplog fix * update commit id * update and enhance afl++ * update afl++ commit id, better run asan options * fix linter * add debug_afl script * Update debug_afl * Update compile_afl * fix for karchive * put debug_afl in the docker container * asan poison fix Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
* [infra] Avoid adding fuzzing language for shell subcommand. (#5300) (#5301)Gravatar Sebastian Rasmussen2021-03-05
|
* fix karchive build for afl++ (#5305)Gravatar van Hauser2021-03-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * output afl++ setup * update commit id * update afl++ commit id * asan + cmplog fix * update commit id * update and enhance afl++ * update afl++ commit id, better run asan options * fix linter * add debug_afl script * Update debug_afl * Update compile_afl * fix for karchive * put debug_afl in the docker container Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
* [base-runner] Reduce image size by 45% (645 MB). (#5283)Gravatar jonathanmetzman2021-03-04
| | | | | | Reduce image size by: 1. Not installing go toolchain in final image. Build go tools in seperate image that doesn't become base-runner. 2. Download the JVM zip in the same step we remove it.
* update afl++ commit id plus add afl++ debug script (#5297)Gravatar van Hauser2021-03-04
|
* [AFL++][base-builder] Precompile afl++ (#5290)Gravatar jonathanmetzman2021-03-03
| | | | | Precompile AFL like we already do for honggfuzz. This saves about a minute in compilation time of AFL targets by doing it in base-builder It only adds about 30 MB to the image size.
* [base-clang] Reduce build time by ~65% (#5280)Gravatar jonathanmetzman2021-03-03
| | | | | | | | | | | | | | | | Reduce build time by doing the following: 1. Building the second stage clang build with a clang binary we download from chromium. 2. Changing NPROC to be half of the cores instead of assuming it's 16 cores. This still addresses the OOM when building on GCB but speeds up local building. 3. Don't install recommended packages and use --depth 1 when possible (very minor improvements compared to the above). In all this reduces local build time of base-clang from 32 minutes to 11 minutes. Because build times are reduced, it will be easier to iteratively develop changes needed for #5170
* [CIFuzz][coverage] Fix bug in getting coverage reports (#5284)Gravatar jonathanmetzman2021-03-03
| | | Also add tests.
* [TSAN] Support TSAN (#5266)Gravatar jonathanmetzman2021-03-02
| | | | We aren't planning on suppoting TSAN on ClusterFuzz. But we can support in CIFuzz since Skia wants it.
* update afl++ commit id (#5273)Gravatar van Hauser2021-03-02
| | | | | | | | | | | * output afl++ setup * update commit id * update afl++ commit id * asan + cmplog fix * update commit id
* afl++ commit id update (#5271)Gravatar van Hauser2021-03-01
| | | | | | | | | * output afl++ setup * update commit id * update afl++ commit id * asan + cmplog fix
* afl++ commit id update (#5258)Gravatar van Hauser2021-02-26
| | | | | | | * output afl++ setup * update commit id * update afl++ commit id
* [base-builder] Use builtin libFuzzer instead of recompling. (#4682)Gravatar jonathanmetzman2021-02-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of recompiling libFuzzer each time we do a libFuzzer build of a project, always use Clang's builtin version of libFuzzer. Do this by copying the builtin libFuzzer to /usr/local/lib/FuzzingEngine.a. This means that the projects that aren't using -fsanitize=fuzzer now also use the builtin libFuzzer. And we no longer need to compile a sanitized libFuzzer for them. This change improves fuzzing performance and developer experience. 1. It improves developer experience by saving time spent compiling libFuzzer when recompiling fuzzers. The time saved is about 25 seconds on my machine. This will make iterating on fuzzer integration much easier. 2. It improves fuzzer performance. The builtin libFuzzer isn't sanitized so it is faster. In some cases (see [here](https://bugs.chromium.org/p/chromium/issues/detail?id=934639)) sanitized libFuzzers can waste 37% of the time running non-performant implementations of code that the builtin-libFuzzer can do almost instantaneously (assembly vs C code). The consequences of improving developer experience and fuzzer performance aren't so easy to measure (though we will look for perf consequences on ClusterFuzz). But some of the consequences of saving time compiling libFuzzer are easy to figure out and quite important. They are: 1. Saving $14646 a year on build costs. Based on the following: build time saved (on GCB): ~38 seconds libFuzzer builds per day: 990 builds per year: >365 price per build-minute (32 core instance, https://cloud.google.com/build/pricing): 0.064 38/60*.064*990*365 = 14,646 2. Speeding up infra-tests. Many of the integration tests build fuzzers and so building libFuzzer was a considerable bottleneck. On my many-core machine the savings were good and noticeable (and are probably larger on the less performant CI machines). | | With compiling libfuzzer | Without compiling libfuzzer | | ---------------------- | ------------------------------- | ----------------------------------- | | Parallel tests | 45 | 34 | | Sequential tests | 276 | 190 | 3. Speeding up CIFuzz. CIFuzz needs to be fast but it spends about 40 seconds compiling libFuzzer. In a run where no bugs are discovered which is intended to take about 20 minutes compiling libFuzzer takes about 3% of the time (40/(20*60)*100). Now we don't need to waste that time. See https://github.com/google/oss-fuzz/issues/5180, which this partially fixes. This bug fixes https://github.com/google/oss-fuzz/issues/2312 and https://github.com/google/oss-fuzz/issues/4677.
* Go 1.16 local builds (#5241)Gravatar Catena cyber2021-02-23
| | | | | | | | | | | | | | | | | * Makes vitess build local As it uses vitess.io instead of github * Completes minify project * Completes quic-go * Local build for nats project * Completes ipfs * run go mod tidy after adding go module * Right bash sequence for go mod tidy
* Go 1.16 more fixes (#5239)Gravatar Catena cyber2021-02-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fixes cilium build * Fixes dragonfly build * Fixes fasthttp build * Fixes fastjson build * golang build: change directory only temporary * Fixes gitea build * Fixes grpc-gateway build * Fixes hugo project build * Fixes ipfs build * Fixes jsonparser build * Fixes kubernetes build * Fixes loki build * Fixes minify build * Fixes nats build * Removes go get from the docs * Fixes quic-go build * Fixes radon build * Fixes syzkaller build * Fixes tidb build * Fixes vitess build
* Fix spuriously failing test (#5218)Gravatar jonathanmetzman2021-02-22
|
* Fixes go coverage build with go 1.16 (#5237)Gravatar Catena cyber2021-02-22
|
* Fixes for afl++ (#5235)Gravatar van Hauser2021-02-22
| | | | | * output afl++ setup * update commit id
* infra: update the timeout of AFL (#5223)Gravatar DavidKorczynski2021-02-21
| | | | | * Updated the timeout for AFL. * Updated the AFL timeout.
* Go 1.16 build fix with right parentheses (#5228)Gravatar Catena cyber2021-02-21
| | | | | | | | | | | | | * go: right bash condition for changing directory * go-json-iterator: uses git clone So as to copy fuzz target in right directory * go: uses tags when running go list * go-redis: uses git clone and builds local fuzz target * cascadia: uses git clone instead of go get
* [CIFuzz] Support languages non-C++ projects (e.g. Python projects) (#5222)Gravatar jonathanmetzman2021-02-19
| | | | Allow use of non-C++ projects by specifying the language in the workflow file. Fixes #5195
* golang: compatibility with go 1.16 modules (#5221)Gravatar Catena cyber2021-02-19
|
* Final afl++ integration (#5191)Gravatar van Hauser2021-02-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | * final afl++ integration * remove afl++ cmplog tests * update afl++ commit id * support rebuild * llvm 13 workaround * apply fix for llvm 13 * fix nits * Fix nits. * Fix name nit. * update commit id * update commit id * update commit id to stable Co-authored-by: Abhishek Arya <inferno@chromium.org>
* [CIFuzz] Fix handling of sanitizer artifacts (#5182)Gravatar jonathanmetzman2021-02-18
| | | | | Fixes #5175 1. Put sanitizer in artifact name. 2. Fix parsing of non-ASAN stacks.
* [infra] Remove unneeded binaries, docs from CMake install (#5161)Gravatar jonathanmetzman2021-02-18
| | | This should reduce the image layer size from 127 MB to 57 MB
* [jazzer] Adapt infra scripting to JVM fuzz targets (#5176)Gravatar Fabian Meumertzheim2021-02-16
| | | | | | | | | | | | | compile, bad_build_check, and presubmit.py require small tweaks to support JVM fuzz targets, most of which are similar to those required for Python. The following additional changes are required: * Since the Jazzer driver binary already links in libFuzzer, it should not be built as a static library. * It is not clear how to do architecture checks as JVM fuzz targets can load their native dependencies dynamically at runtime. For now, the check is disabled. * The Jazzer binaries are moved into $OUT and need to be skipped over in find_fuzz_targets.
* Add Jazzer to base-builder and base-runner (#5166)Gravatar Fabian Meumertzheim2021-02-16
| | | | | | | | | | | | | | | | | | | | | | | | | | Jazzer is built from HEAD using Bazel and the clang toolchain provided by base-clang. While it could be built with OpenJDK 8, which is available as a package, JVM fuzz targets should not be forced to be compatible with Java 8. For this reason, the official binary release of OpenJDK 15 is pulled into both base-builder and base-runner and set as JAVA_HOME. It is trimmed down in size by removing src.zip and the jmods directory. Jazzer consists of the following four components: * The API (`jazzer_api_deploy.jar`), which is required for fuzz targets that use FuzzedDataProvider or custom method hooks, is made available in /usr/local/lib in base-builder. * The driver (`jazzer_driver`), which links in libFuzzer and is reused across fuzz targets. Since it is used to run fuzz targets, it is included into base-runner. * The ASanified driver (`jazzer_driver_asan`), which is obtained from `jazzer_driver` by linking in ASan. * The agent (`jazzer_agent_deploy.jar`), which bundles the runtime instrumentation agent with the Jazzer API. It is loaded by the driver and thus also included into base-runner. The changes to the infra scripts required by JVM fuzz targets will be submitted as a separate PR.
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-07 07:00:53 +0000