diff options
| author |  A. Unique TensorFlower <nobody@tensorflow.org> | 2016-01-18 19:37:50 -0800 |
|---|---|---|
| committer |  Manjunath Kudlur <keveman@gmail.com> | 2016-01-19 07:31:45 -0800 |
| commit | 8a1c308450b64a7d3c2ca520109c4909acbf13ff (patch) | |
| tree | 3bee1c9e4e9cb69c83abf0ce010c19537dfc14be /tensorflow/core/lib/strings/str_util.h | |
| parent | 8a445e54b893f313ca44c7ef124b1884f4994e95 (diff) | |
Remove word2vec's dependency on RE2. This is primarily to allow inputs
over 2GiB in size.
As part of this, add ConsumeNonWhitespace() to str_util.
Change: 112436569
Diffstat (limited to 'tensorflow/core/lib/strings/str_util.h')
| -rw-r--r-- | tensorflow/core/lib/strings/str_util.h | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/tensorflow/core/lib/strings/str_util.h b/tensorflow/core/lib/strings/str_util.h index 134a49e631..a485d3e4a9 100644 --- a/tensorflow/core/lib/strings/str_util.h +++ b/tensorflow/core/lib/strings/str_util.h @@ -65,6 +65,11 @@ size_t RemoveWhitespaceContext(StringPiece* text); // overflow occurred, returns false. Otherwise, returns false. bool ConsumeLeadingDigits(StringPiece* s, uint64* val); +// Consume a leading token composed of non-whitespace characters only. +// If *s starts with a non-zero number of non-whitespace characters, store +// them in *val, advance *s past them, and return true. Else return false. +bool ConsumeNonWhitespace(StringPiece* s, StringPiece* val); + // If "*s" starts with "expected", consume it and return true. // Otherwise, return false. bool ConsumePrefix(StringPiece* s, StringPiece expected); |