aboutsummaryrefslogtreecommitdiffhomepage
path: root/tensorflow/contrib/kinesis
Commit message (Collapse)AuthorAge
* [tf.data] Adding a private method for (recursively) tracking dataset inputs.Gravatar Jiri Simsa2018-09-25
| | | | PiperOrigin-RevId: 214495925
* Move from deprecated self.test_session() to self.cached_session().Gravatar A. Unique TensorFlower2018-09-21
| | | | | | | | self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement. PiperOrigin-RevId: 213944932
* [tf.data] Internal refactoring of C++ classes and APIs.Gravatar Jiri Simsa2018-08-13
| | | | | | | | | | - replacing `OpKernelContext` with newly introduced `DatasetContext` in `DatasetBase` constructor to make it possible to instantiate `DatasetBase` in places where an instance of `OpKernelContext` is not available - replacing `dataset::MakeIteratorContext(OpKernelContext* ctx)` factory with `IteratorContext(OpKernelContext *ctx)` constructor. - folding `GraphDatasetBase` into `DataseBase` and removing the default implementation of `AsGraphDefInternal`, making it the responsibility of the derived class to implement it to encourage/hint developers to provide serialization logic PiperOrigin-RevId: 208560010
* [tf.data] Optimization checkpointing improvements.Gravatar Jiri Simsa2018-08-10
| | | | | | | | This CL: - changes the `OptimizeDataset` checkpointing logic to checkpoint the optimized dataset (as opposed to the original dataset + the optimizations, re-running optimization every time a checkpoint is restored) - replaces `OpKernelContext` with newly introduced `SerializationContext` in the signature of `AsGraphDefInternal` to reduce the scope of the context and also simplify the logic for overriding the `FunctionLibraryDefinition` when optimizations take place PiperOrigin-RevId: 208282562
* Add KinesisDataset support for tensorflow Dataset (#19712)Gravatar Yong Tang2018-07-02
* Add KinesisDataset support for tensorflow Dataset This fix is an attempt to add Kinesis support for tensorflow's Dataset. Kinesis is provided by AWS as a managed data streaming service. It is similiar to Apache Kafka, often used in places where maintaining a independent Kafka cluster on AWS is not desirable or possible. This fix adds the Kinesis support for tensorflow Dataset. Similiar to the Kafka integration in tensorflow, KinesisDataset outputs tf.string for records. Test cases have also been added, which could be invoked manually. Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Expose KinesisDataset in dataset_ops.cc Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Expose KinesisDataset in python wrapper Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Add test cases for KinesisDataset Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Update AWS library include files Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Add Bazel BUILD files Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Rename s3_crypto to aws_crypto Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Rename with_s3_support to with_aws_support Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Selectively add kinesis to tensorflow/contrib/BUILD Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Set different partition key and pylint fix. Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Add missing modules in cmake's python_modules.txt Signed-off-by: Yong Tang <yong.tang.github@outlook.com> * Address review feedback Signed-off-by: Yong Tang <yong.tang.github@outlook.com>