gsrsup
gsrsup implements resumable streaming uploads to Google Cloud Storage. It’s
useful for uploading a reproducible stream of data too large to fit on disk. For
example, you can upload a ZFS snapshot by running
zfs send tank/fs@snapshot | gsrsup gs://bucket/object
This pipeline can be interrupted at any time; if rerun, gsrsup will seek to the
appropriate place in the input stream and continue uploading from there.
gsrsup performs integrity checking on the data it uploads and will report a
problem if the input stream changes between two invocations. However, the design
of the Google Cloud Storage API prevents gsrsup from detecting such a change
until the entire stream has been uploaded.
For safety, gsrsup treats Google Cloud Storage objects as append-only – it
will never overwrite or delete them. Furthermore, it will never append to an
object once its upload has completed. If you run
gsrsup gs://bucket/bash </bin/bash
twice, the second invocation will skip uploading altogether and only run the integrity-checking code.
Building
gsrsup builds using a handwritten Ninja file.
Simply run ninja to create a development build. You can also run Ninja with a
custom build file to request a different profile:
ninja -f buildconf/asan.ninja: Build with AddressSanitizer.ninja -f buildconf/dbg.ninja: Build for debugging.ninja -f buildconf/release.ninja: Build with optimizations.