aboutsummaryrefslogtreecommitdiffhomepage
path: root/projects/libra
diff options
context:
space:
mode:
authorGravatar David Wong <mimoo@users.noreply.github.com>2020-04-02 14:28:09 -0700
committerGravatar GitHub <noreply@github.com>2020-04-02 14:28:09 -0700
commitf5098035eb1a14aa966c8651d88ea3d64323823d (patch)
treeeaf2babcc0c190a9c3650fc60b0fb87f7bbd47d5 /projects/libra
parentfa2d9f50172d0d362a626e0bc1893c83b326d9b0 (diff)
[libra] Fix building failure (#3566)
After many days banging my head on FFI issues in rust, I hereby present a fix to the issue. Note that I've got some help, and I'm not sure I understand everything here. But this is my understanding of what was not working, and how we fixed it. The **problem** is that on Ubuntu 16 with llvm/clang 10, we were **statically linking libc++** in [rocksdb][1]: ```rust let stdlib = if tool.is_like_gnu() { "libstdc++.a" } else if tool.is_like_clang() { "libc++.a" } else { // Don't link to c++ statically on windows. return; }; // ... // remove lib prefix and .a postfix. println!( "cargo:rustc-link-lib=static={}", &stdlib[3..stdlib.len() - 2] ); ``` This means that during building, when we reach building of rocksdb, we import a number of symbols from libc++ (like [__muloti4][2]) that end up in the associated `.rlib` (rust obj file). These symbols interestingly do not exist in libstdc++ which is used by gcc. This is important because on linux (unlike mac), the rust toolchain is compiled with gcc. So these intrinsics are not present in the linux rust toolchain, and have been redeclared in the [compiler-builtins][3] crate. So here is the problem: * rust toolchain's defines these intrinsics functions * libc++ defines these intrinsics functions And the recipe for disaster: * libc++ is statically linked in rocksdb, which means all the symbols are imported * symbols in rocksdb's produced `.rlib` are colliding with the symbols from the rust toolchain `.rlib` To fix this. Maybe we could have compiled the stuff with libstdc++? But instead we: 1. removed the static linking 2. we linked libc++ at the very last moment via: ```rust RUSTFLAGS="-C link-arg=-L/usr/local/lib -C link-arg=-lc++" ``` At final linking time, the linker sees that the intrinsics are already defined in one of the `.rlib` (produced by compiler-builtins) and so does not import these functions from libc++. Actually, at this point it only statically link the functions that need to be linked. It seems to work. [1]: https://github.com/tikv/rust-rocksdb/blob/c79d2c2ac68f9ec40fa03faac3882baac38db314/librocksdb_sys/build.rs#L115 [2]: https://github.com/llvm-mirror/libcxx/blob/master/src/filesystem/int128_builtins.cpp [3]: https://github.com/rust-lang/compiler-builtins/blob/e578d47247470935ebcc2bbfc81430b94f31ffd8/src/int/mul.rs#L107
Diffstat (limited to 'projects/libra')
-rw-r--r--projects/libra/build.sh63
1 files changed, 49 insertions, 14 deletions
diff --git a/projects/libra/build.sh b/projects/libra/build.sh
index ea07ced6..c778169e 100644
--- a/projects/libra/build.sh
+++ b/projects/libra/build.sh
@@ -17,23 +17,33 @@
# Note: This project creates Rust fuzz targets exclusively
-# make OSS-Fuzz work with Rust
-export CUSTOM_LIBFUZZER_PATH="$LIB_FUZZING_ENGINE_DEPRECATED"
-export CUSTOM_LIBFUZZER_STD_CXX=c++
+# recipe:
+# -------
+# 1. we list all the fuzzers and save to a file fuzzer_list
+# 2. we build the corpus for each fuzzer
+# 3. we build all the fuzzers
+
+# reset flags of OSS-Fuzz
export CFLAGS="-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION"
export CXXFLAGS_EXTRA="-stdlib=libc++"
export CXXFLAGS="$CFLAGS $CXXFLAGS_EXTRA"
-# RUSTC_BOOTSTRAP: to get some nightly features like ASAN
-export RUSTC_BOOTSTRAP=1
-
-#
+# correct workdir
cd $SRC/libra/testsuite/libra-fuzzer
-# list fuzzers
+# fetch all dependencies (needed for patching rocksdb)
+cargo fetch
+
+# patch rocksdb to not link libc++ statically
+sed -i "s/link_cpp(&mut build)/build.cpp_link_stdlib(None)/" \
+ /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/72e45c3/librocksdb_sys/build.rs
+# so now we need to link libc++ at the end
+export RUSTFLAGS="-C link-arg=-L/usr/local/lib -C link-arg=-lc++"
+
+# 1. list fuzzers
cargo run --bin libra-fuzzer list --no-desc > fuzzer_list
-# build corpus and move to $OUT
+# 2. build corpus and move to $OUT
cat fuzzer_list | while read -r line
do
cargo run --bin libra-fuzzer generate -n 128 $line
@@ -41,11 +51,36 @@ do
rm -r fuzz/corpus/$line
done
-# build fuzzers
-# --cfg fuzzing -> used to change code logic
-# -Cdebug-assertions -> to get debug_assert in rust
-# other flags -> taken from cargo fuzz
-export RUSTFLAGS="--cfg fuzzing -Cdebug-assertions -Cpasses=sancov -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cllvm-args=-sanitizer-coverage-inline-8bit-counters -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Cllvm-args=-sanitizer-coverage-pc-table -Clink-dead-code -Cllvm-args=-sanitizer-coverage-stack-depth -Zsanitizer=address -Ccodegen-units=1"
+# rust libfuzzer flags (https://github.com/rust-fuzz/libfuzzer/blob/master/build.rs#L12)
+export CUSTOM_LIBFUZZER_PATH="$LIB_FUZZING_ENGINE_DEPRECATED"
+export CUSTOM_LIBFUZZER_STD_CXX=c++
+# export CUSTOM_LIBFUZZER_STD_CXX=none
+
+# RUSTC_BOOTSTRAP: to get some nightly features like ASAN
+export RUSTC_BOOTSTRAP=1
+
+# export fuzzing flags
+RUSTFLAGS="$RUSTFLAGS --cfg fuzzing" # used to change code logic
+RUSTFLAGS="$RUSTFLAGS -Cdebug-assertions" # to get debug_assert in rust
+RUSTFLAGS="$RUSTFLAGS -Zsanitizer=address" # address sanitizer (ASAN)
+
+RUSTFLAGS="$RUSTFLAGS -Cdebuginfo=1"
+RUSTFLAGS="$RUSTFLAGS -Cforce-frame-pointers"
+
+RUSTFLAGS="$RUSTFLAGS -Cpasses=sancov"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-level=4"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-trace-compares"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-inline-8bit-counters"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-trace-geps"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-prune-blocks=0"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-pc-table"
+RUSTFLAGS="$RUSTFLAGS -Clink-dead-code"
+RUSTFLAGS="$RUSTFLAGS -Cllvm-args=-sanitizer-coverage-stack-depth"
+RUSTFLAGS="$RUSTFLAGS -Ccodegen-units=1"
+
+export RUSTFLAGS
+
+# 3. build all the fuzzers!
cat fuzzer_list | while read -r line
do
# build