README.md


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# gsrsup

`gsrsup` implements _resumable streaming uploads_ to Google Cloud Storage. It’s
useful for uploading a reproducible stream of data too large to fit on disk. For
example, you can upload a ZFS snapshot by running

    zfs send tank/fs@snapshot | gsrsup gs://bucket/object

This pipeline can be interrupted at any time; if rerun, `gsrsup` will seek to the
appropriate place in the input stream and continue uploading from there.

`gsrsup` performs integrity checking on the data it uploads and will report a
problem if the input stream changes between two invocations. However, the design
of the Google Cloud Storage API prevents `gsrsup` from detecting such a change
until the entire stream has been uploaded.

For safety, `gsrsup` treats Google Cloud Storage objects as append-only – it
will never overwrite or delete them. Furthermore, it will never append to an
object once its upload has completed. If you run

    gsrsup gs://bucket/bash </bin/bash

twice, the second invocation will skip uploading altogether and only run the
integrity-checking code.

## Building

`gsrsup` builds using a handwritten [Ninja](https://ninja-build.org/) file.
Simply run `ninja` to create a development build. You can also run Ninja with a
custom build file to request a different profile:

  - `ninja -f buildconf/asan.ninja`: Build with AddressSanitizer.
  - `ninja -f buildconf/dbg.ninja`: Build for debugging.
  - `ninja -f buildconf/release.ninja`: Build with optimizations.