| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
to replace it.
This change prepares `tf.data` for TensorFlow 2.0, where `tf.contrib` will no longer exist. It retains the pre-existing endpoints in `tf.contrib.data` with deprecation warnings.
Note there are some exceptions to the move:
* Deprecated symbols in `tf.contrib.data` have not been moved to `tf.data.experimental`, because replacements already exist.
* `tf.contrib.data.LMDBDataset` has not been moved, because we plan to move it to a SIG-maintained repository.
* `tf.contrib.data.assert_element_shape()` has not yet been moved, because it depends on functionality in `tf.contrib`, and it will move in a later change.
* `tf.contrib.data.AUTOTUNE` has not yet been moved, because we have not yet determined how to `tf_export()` a Python integer.
* The stats-related API endpoints have not yet appeared in a released version of TensorFlow, so these are moved to `tf.data.experimental` without retaining an endpoint in `tf.contrib.data`.
In addition, this change includes some build rule and ApiDef refactoring:
* Some of the "//third_party/tensorflow/python:training" dependencies had to be split in order to avoid a circular dependency.
* The `tf.contrib.stateless` ops now have a private core library for the generated wrappers (and accordingly are hidden in their ApiDef) so that `tf.data.experimental.sample_from_datasets()` can depend on them.
PiperOrigin-RevId: 215304249
|
|
|
|
| |
PiperOrigin-RevId: 214653279
|
|
|
|
| |
PiperOrigin-RevId: 214495925
|
|
|
|
|
|
|
|
| |
self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about:
* the fact that the session may be reused.
* the session is not closed even when doing a "with self.test_session()" statement.
PiperOrigin-RevId: 213944932
|
|
|
|
| |
PiperOrigin-RevId: 211733735
|
|
|
|
|
|
|
|
| |
environment
variable, to avoid having to copy a particular file to `/usr/share`.
PiperOrigin-RevId: 210550389
|
|
|
|
|
|
|
|
|
|
| |
- replacing `OpKernelContext` with newly introduced `DatasetContext` in `DatasetBase` constructor to make it possible to instantiate `DatasetBase` in places where an instance of `OpKernelContext` is not available
- replacing `dataset::MakeIteratorContext(OpKernelContext* ctx)` factory with `IteratorContext(OpKernelContext *ctx)` constructor.
- folding `GraphDatasetBase` into `DataseBase` and removing the default implementation of `AsGraphDefInternal`, making it the responsibility of the derived class to implement it to encourage/hint developers to provide serialization logic
PiperOrigin-RevId: 208560010
|
|
|
|
| |
PiperOrigin-RevId: 208352025
|
|
|
|
| |
PiperOrigin-RevId: 208263100
|
|
|
|
| |
PiperOrigin-RevId: 208219138
|
|
|
|
| |
PiperOrigin-RevId: 208146858
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Back-ticks are now converted to links in the api_docs generator. With the new docs repo we're moving to simplify the docs pipeline, and make everything more readable.
By doing this we no longer get test failures for symbols that don't exist (`tf.does_not_exist` will not get a link).
There is also no way, not to set custom link text. That's okay.
This is the result of the following regex replacement (+ a couple of manual edits.):
re: @\{([^$].*?)(\$.+?)?}
sub: `\1`
Which does the following replacements:
"@{tf.symbol}" --> "`tf.symbol`"
"@{tf.symbol$link_text}" --> "`tf.symbol`"
PiperOrigin-RevId: 208042358
|
|\
| |
| |
| | |
PiperOrigin-RevId: 207988541
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Add link to updating scope on a running VM
* Add code formatting and Python syntax highlighting
* Clarify kwargs argument formatting
* Fix method name in docstring
PiperOrigin-RevId: 207204628
|
|/ |
|
|
|
|
|
|
|
| |
This removes the confusion between BigTable and Bigtable naming. Also cleaned
up all other uses of BigTable in error messages.
PiperOrigin-RevId: 205586899
|
|
|
|
| |
PiperOrigin-RevId: 205530581
|
|
|
|
|
|
| |
Because the tf.data integration currently doesn't support setting client-side timestamps, using the AlwaysRetryMutationPolicy can make data loading more reliable. (This is safe-ish to do, because when reading TF always read Latest(1), so duplicate writes will not be visible to user programs.)
PiperOrigin-RevId: 204331133
|
|
|
|
| |
PiperOrigin-RevId: 204140674
|
|
|
|
|
|
|
|
| |
In order to stream data from Cloud Bigtable at high speed, it's important to use multiple connections to stream simultaneously from multiple tablet servers. This change adds two new methods to the BigTable object to set up a dataset based on the SampleKeys method, and tf.contrib.data.parallel_interleave. Because the keys returned from SampleKeys is not guaranteed to be deterministic (in fact, it can change over time without any new data added to the table), the resulting datasets are not deterministic. (In order to further boost performance, we enable sloppy interleaving.)
When comparing the table.scan_* methods vs the table.parallel_scan_* methods for a test workload (based on ImageNet), we see performance gains of over 15x, and over 10x compared to a reasonably tuned GCS input pipeline.
PiperOrigin-RevId: 203945580
|
|
|
|
|
|
| |
When storing images in Cloud Bigtable, the resulting gRPC messages are often larger than the default receive message max size value. This change makes the maximum receive message sizes configurable, and sets a more reasonable default for general TensorFlow use.
PiperOrigin-RevId: 203569796
|
|
|
|
| |
PiperOrigin-RevId: 203405436
|
|
|
|
|
|
| |
When writing high performance input pipelines, you often need to read from multiple servers in parallel. Being able to sample row keys from a table allows one to easily construct high performance parallel input pipelines from Cloud Bigtable.
PiperOrigin-RevId: 203389017
|
|
|
|
|
|
| |
In order to test parallel scanning strategies, we rely on SampleKeys returning a series of values. This change augments the test client to return every other row to facilitate easy testing of the parallel scan code.
PiperOrigin-RevId: 203383247
|
|
|
|
| |
PiperOrigin-RevId: 203319567
|
|
|
|
|
|
| |
Some errors returned from the Bigtable client are not proper utf8 strings. As a result, when we construct the errors::Unknown error to return to the client, the resulting error message gets suppressed. In order to ensure that some useful context is always available to users, we log the error message in addition to returning it.
PiperOrigin-RevId: 203203543
|
|
|
|
|
|
| |
Use SetCell with no timestamp arg when using server-set timestamps instead of using std::chrono::milliseconds timestamp(-1), because this results in a timestamp_micros value of -1000 instead of -1, which causes the server to become unhappy.
PiperOrigin-RevId: 203165785
|
|
|
|
|
|
|
| |
If left unset, the default should be far larger than the default 4 the
client library uses.
PiperOrigin-RevId: 203139678
|
|
|
|
| |
PiperOrigin-RevId: 202986386
|
|
|
|
| |
PiperOrigin-RevId: 202673820
|
|
This change allows TensorFlow users to stream data directly from
Cloud Bigtable into the TensorFlow runtime using tf.data.
PiperOrigin-RevId: 202664219
|