aboutsummaryrefslogtreecommitdiffhomepage
path: root/tensorflow/python/data
Commit message (Collapse)AuthorAge
* [tf.data] Adding specialization for `MapDataset`, `ParallelMapDataset`, and ↵Gravatar Jiri Simsa2018-10-08
| | | | | | `MapAndBatchDataset` whose user-provided functions have the property that each output argument take its value directly from an input argument (e.g. `lambda x, y: y, x`). This specialization can produce the result without having to schedule the function using the executor. PiperOrigin-RevId: 216206232
* Fix typoGravatar Makoto Uchida2018-10-08
| | | | PiperOrigin-RevId: 216203408
* Automated rollback of commit ae0bc6f006497cc04a2ee75166d4ec71c7154fd8Gravatar Jiri Simsa2018-10-05
| | | | PiperOrigin-RevId: 215969360
* [tf.data] Adding specialization for `MapDataset`, `ParallelMapDataset`, and ↵Gravatar Jiri Simsa2018-10-05
| | | | | | `MapAndBatchDataset` whose user-provided functions have the property that each output argument take its value directly from an input argument (e.g. `lambda x, y: y, x`). This specialization can produce the result without having to schedule the function using the executor. PiperOrigin-RevId: 215957592
* Automated rollback of commit 6b538d9ce54e878576131cde0c76e43a893180c2Gravatar Smit Hinsu2018-10-04
| | | | PiperOrigin-RevId: 215808649
* [tf.data] Add a notion of `captured args` to MapDefunGravatar Rachel Lim2018-10-04
| | | | PiperOrigin-RevId: 215788485
* [tf.data] Clean up tests for `tf.data.experimental`.Gravatar Derek Murray2018-10-04
| | | | | | This change splits up large test files into smaller ones, and re-enables tests that were disabled for obsolete reasons. PiperOrigin-RevId: 215785396
* Add ability to vectorize nodes that do not derive from function arguments. ↵Gravatar Rachel Lim2018-10-04
| | | | | | (This indirectly handles "Const" outputs automagically, since they are always unstacked.) PiperOrigin-RevId: 215749824
* Automated rollback of commit 70a395f9795a48c21bc35cdf1dc44778f73a7bbaGravatar A. Unique TensorFlower2018-10-04
| | | | PiperOrigin-RevId: 215710849
* [tf.data] Fix bug in `tf.data.experimental.unbatch()`.Gravatar Derek Murray2018-10-03
| | | | | | | | | Previously, if the rank of the input to this transformation was statically unknown, we would erroneously report that the output is a scalar, and violate downstream shape integrity checks. Instead, in that case the output shape should be unknown. PiperOrigin-RevId: 215683027
* Update size of multi_device_iterator_test to medium to fix timeoutsGravatar Smit Hinsu2018-10-03
| | | | PiperOrigin-RevId: 215637785
* [data-stats] Sets user given `tag` and `counter_prefix` with ↵Gravatar Shivani Agrawal2018-10-03
| | | | | | | | `set_stats_aggregator`. `tag` would get prep-end with all the statistics recorded as summary and `counter_prefix` would set the prefix for the statistics recorded as counter. Note: `counter` defaults to `\tensorflow`, and `tag` and `prefix` gets associated with the dataset (not the stats_aggregator). PiperOrigin-RevId: 215609159
* [tf.data] Fix noisy warning.Gravatar Jiri Simsa2018-10-03
| | | | PiperOrigin-RevId: 215607171
* Automated rollback of commit 51b266fba181dffb6b3f9207280cde6b7670dd90Gravatar Jiri Simsa2018-10-03
| | | | PiperOrigin-RevId: 215593867
* [tf.data] Fix noisy warning.Gravatar Jiri Simsa2018-10-03
| | | | PiperOrigin-RevId: 215592456
* Automated rollback of commit b7e9cbab27c893283acc4a6154d7a59dffb23758Gravatar Derek Murray2018-10-02
| | | | PiperOrigin-RevId: 215503549
* Use `defun` instead of `Defun` for `tf.data`, except for ↵Gravatar Shivani Agrawal2018-10-02
| | | | | | `make_one_shot_iterator` which is to be deprecated in future. PiperOrigin-RevId: 215491729
* Internal change.Gravatar Revan Sopher2018-10-02
| | | | PiperOrigin-RevId: 215473351
* [tf.data] Adding `tf.data.Options()`, `tf.data.Dataset.options()`, and ↵Gravatar Jiri Simsa2018-10-01
| | | | | | `tf.data.Dataset.with_options()` to make it possible to respectively represent, get, and set options, such as optimization configuration, of a tf.data input pipeline. PiperOrigin-RevId: 215310764
* [tf.data] Deprecate `tf.contrib.data` and introduce `tf.data.experimental` ↵Gravatar Derek Murray2018-10-01
| | | | | | | | | | | | | | | | | | | | to replace it. This change prepares `tf.data` for TensorFlow 2.0, where `tf.contrib` will no longer exist. It retains the pre-existing endpoints in `tf.contrib.data` with deprecation warnings. Note there are some exceptions to the move: * Deprecated symbols in `tf.contrib.data` have not been moved to `tf.data.experimental`, because replacements already exist. * `tf.contrib.data.LMDBDataset` has not been moved, because we plan to move it to a SIG-maintained repository. * `tf.contrib.data.assert_element_shape()` has not yet been moved, because it depends on functionality in `tf.contrib`, and it will move in a later change. * `tf.contrib.data.AUTOTUNE` has not yet been moved, because we have not yet determined how to `tf_export()` a Python integer. * The stats-related API endpoints have not yet appeared in a released version of TensorFlow, so these are moved to `tf.data.experimental` without retaining an endpoint in `tf.contrib.data`. In addition, this change includes some build rule and ApiDef refactoring: * Some of the "//third_party/tensorflow/python:training" dependencies had to be split in order to avoid a circular dependency. * The `tf.contrib.stateless` ops now have a private core library for the generated wrappers (and accordingly are hidden in their ApiDef) so that `tf.data.experimental.sample_from_datasets()` can depend on them. PiperOrigin-RevId: 215304249
* Automated rollback of commit d78595d333c9b5c8a0705ba6852c08b107d6c462Gravatar A. Unique TensorFlower2018-09-29
| | | | PiperOrigin-RevId: 215073584
* Make cuda_py_test create a gpu and cpu target.Gravatar A. Unique TensorFlower2018-09-29
| | | | | | | | Currently, we run tests on machines with GPUs based on the "gpu" tag, and the tests automatically adapt to whether a GPU is available. Creating two targets, one tagged with "gpu" and one not, will make us run the tests in both modes. PiperOrigin-RevId: 215045035
* [tf.data] Merged contrib.data's DatasetTestBase with the DatasetTestBase in ↵Gravatar Rachel Lim2018-09-28
| | | | | | core (and added that as a base class for all the contrib tests). Also changed the assertDatasetsEqual functions so they are both graph and eager compatible (took the code from CSVDatasetTest) :) PiperOrigin-RevId: 215004892
* [tf.data Introducing tf.data.Dataset.reduce() which reduces elements of a ↵Gravatar Jiri Simsa2018-09-27
| | | | | | (finite) dataset to a single element. PiperOrigin-RevId: 214852364
* [tf.data] Minor refactoring of tf.data tests.Gravatar Jiri Simsa2018-09-27
| | | | PiperOrigin-RevId: 214781794
* [tf.data] Adding a private method for (recursively) tracking dataset inputs.Gravatar Jiri Simsa2018-09-25
| | | | PiperOrigin-RevId: 214495925
* Disabling MultiDeviceIterator in Eager mode. Support is coming soon.Gravatar Rohan Jain2018-09-24
| | | | PiperOrigin-RevId: 214296771
* [tf.data] Add `tf.contrib.data.Optional` support to `Structure`.Gravatar Derek Murray2018-09-23
| | | | | | | | | | | This change switches `tf.contrib.data.Optional` to use a `Structure` class to represent the structure of its value, instead of `output_types`, `output_shapes`, and `output_classes` properties. It adds support for nesting `Optional` objects and representing their structure. This change also makes a modification to the `Structure` class: `Structure.is_compatible_with(x)` now takes another `Structure` as the `x` argument, instead of a value. This makes it easier to work with nested structures (where we might not have a value readily available), and better matches the interface of other `is_compatible_with()` methods (e.g. in `tf.TensorShape` and `tf.DType`). Finally, in the process of making this change, I observed possible crash-failures when a DT_VARIANT tensor containing another DT_VARIANT tensor is copied between CPU and GPU. This change "fixes" the immediate problem by raising an UnimplementedError, but more work will be necessary to support the full range of use cases. PiperOrigin-RevId: 214198993
* Moving MultiDeviceIterator from contrib to core.Gravatar Rohan Jain2018-09-23
| | | | PiperOrigin-RevId: 214173896
* Merge pull request #22170 from Smokrow:patch-1Gravatar TensorFlower Gardener2018-09-21
|\ | | | | | | PiperOrigin-RevId: 214058098
| * Fix lint errorsGravatar Martin Wicke2018-09-21
| |
* | [tf.data] Add a test for state persistence between iterators over the same ↵Gravatar Derek Murray2018-09-18
| | | | | | | | | | | | MapDataset. PiperOrigin-RevId: 213555982
| * Moved example and changed wordingGravatar Moritz Kröger2018-09-18
| |
* | [tf.data] Introducing `tf.data.Dataset.window(size, shift, stride, ↵Gravatar Jiri Simsa2018-09-17
| | | | | | | | | | | | | | | | drop_remainder)`, which can be used for combining elements of input dataset into "windows". A window is itself a finite dataset and, among other things, can be used for generalized batching (see https://github.com/tensorflow/community/pull/5 for details). PiperOrigin-RevId: 213360134
* | Move from deprecated self.test_session() to self.cached_session().Gravatar A. Unique TensorFlower2018-09-17
| | | | | | | | | | | | | | | | self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement. PiperOrigin-RevId: 213326167
* | Automated rollback of commit d31f360e1574553ed23b8d483512a2065ac426ebGravatar A. Unique TensorFlower2018-09-11
| | | | | | | | PiperOrigin-RevId: 212551965
| * Update of flat_mapGravatar Smokrow2018-09-11
| | | | | | Rework based on Marks review
| * added example for flat_mapGravatar Smokrow2018-09-11
| |
* | Move from deprecated self.test_session() to self.cached_session().Gravatar A. Unique TensorFlower2018-09-10
| | | | | | | | | | | | | | | | self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement. PiperOrigin-RevId: 212336464
* | Move from deprecated self.test_session() to self.cached_session().Gravatar A. Unique TensorFlower2018-09-10
| | | | | | | | | | | | | | | | self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement. PiperOrigin-RevId: 212336258
* | Automated rollback of commit 39b2fb7cfef489424fead18ec5174d8e8b2a9a1aGravatar Akshay Modi2018-09-09
| | | | | | | | PiperOrigin-RevId: 212177437
* | Remove unnecessary function calls from data/util/nest.pyGravatar Akshay Modi2018-09-07
| | | | | | | | PiperOrigin-RevId: 212054927
* | [tf.data] Adding `use_inter_op_parallelism` attr to `ParallelMapDataset` and ↵Gravatar Jiri Simsa2018-09-07
| | | | | | | | | | | | removing unused `graph_def_version` field PiperOrigin-RevId: 212054031
| * edited flat_map description and removed typoGravatar Smokrow2018-09-07
|/ | | The examples in interleave are quite helpful. I just added a reference to this example
* [tf.data] Adding support for `num_parallel_calls` to ↵Gravatar Jiri Simsa2018-09-06
| | | | | | | | `tf.data.Dataset.interleave`. Unlike the `tf.data.contrib.parallel_interleave` whose parallelism is tied to the `cycle_length` argument, the newly introduced `num_parallel_calls` argument of `tf.data.Dataset.interleave` is decoupled from the `cycle_length` argument and identifies the degree of parallelism to use for fetching output elements. PiperOrigin-RevId: 211886816
* Test cleanupsGravatar Asim Shankar2018-09-04
| | | | | | | | | | | | | | - Remove unnecessary use of test_session() in tests that run with eager execution enabled. - Use cached_session() instead of test_session() (self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement.) PiperOrigin-RevId: 211562969
* [tf.data] Add internal optimizations for executing simple functions in ↵Gravatar Derek Murray2018-09-04
| | | | | | `MapDataset`. PiperOrigin-RevId: 211520001
* [tf.data] Fixes error in `dataset.concatenate()`, now throws error if two ↵Gravatar Shivani Agrawal2018-08-22
| | | | | | | | | input datasets are of type dictionaries with different set of key(s). FIXES #20626 REL_NOTES: bug fix in `tf.data.Dataset.concatenate()`, now throws error if trying to concatenate two datasets of type dictionaries with different set of key(s). PiperOrigin-RevId: 209845337
* Move from deprecated self.test_session() to self.session() when a graph is set.Gravatar A. Unique TensorFlower2018-08-21
| | | | | | self.test_session() has been deprecated in cl/208545396 as its behavior confuses readers of the test. Moving to self.session() instead. PiperOrigin-RevId: 209696110
* [tf.data] Move captured function instantiation to iterator initialization time.Gravatar Derek Murray2018-08-21
| | | | | | | | | | Previously, a function instantiation error (e.g. in `Dataset.map()`) would lead to an error in each GetNext() call that attempted to use the function. Moving this to iterator instantiation time has the benefit that the error will be reported once when the initialization op is executed, which has a more helpful stack trace, since it should not be conflated with other potential op failures. PiperOrigin-RevId: 209633511
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-27 20:59:51 +0000