aboutsummaryrefslogtreecommitdiffhomepage
path: root/tensorflow/python/data
Commit message (Collapse)AuthorAge
...
* | Move from deprecated self.test_session() to self.cached_session().Gravatar A. Unique TensorFlower2018-09-10
| | | | | | | | | | | | | | | | self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement. PiperOrigin-RevId: 212336258
* | Automated rollback of commit 39b2fb7cfef489424fead18ec5174d8e8b2a9a1aGravatar Akshay Modi2018-09-09
| | | | | | | | PiperOrigin-RevId: 212177437
* | Remove unnecessary function calls from data/util/nest.pyGravatar Akshay Modi2018-09-07
| | | | | | | | PiperOrigin-RevId: 212054927
* | [tf.data] Adding `use_inter_op_parallelism` attr to `ParallelMapDataset` and ↵Gravatar Jiri Simsa2018-09-07
| | | | | | | | | | | | removing unused `graph_def_version` field PiperOrigin-RevId: 212054031
| * edited flat_map description and removed typoGravatar Smokrow2018-09-07
|/ | | The examples in interleave are quite helpful. I just added a reference to this example
* [tf.data] Adding support for `num_parallel_calls` to ↵Gravatar Jiri Simsa2018-09-06
| | | | | | | | `tf.data.Dataset.interleave`. Unlike the `tf.data.contrib.parallel_interleave` whose parallelism is tied to the `cycle_length` argument, the newly introduced `num_parallel_calls` argument of `tf.data.Dataset.interleave` is decoupled from the `cycle_length` argument and identifies the degree of parallelism to use for fetching output elements. PiperOrigin-RevId: 211886816
* Test cleanupsGravatar Asim Shankar2018-09-04
| | | | | | | | | | | | | | - Remove unnecessary use of test_session() in tests that run with eager execution enabled. - Use cached_session() instead of test_session() (self.test_session() has been deprecated in 9962eb5e84b15e309410071b06c2ed2d6148ed44 as its name confuses readers of the test. Moving to cached_session() instead which is more explicit about: * the fact that the session may be reused. * the session is not closed even when doing a "with self.test_session()" statement.) PiperOrigin-RevId: 211562969
* [tf.data] Add internal optimizations for executing simple functions in ↵Gravatar Derek Murray2018-09-04
| | | | | | `MapDataset`. PiperOrigin-RevId: 211520001
* [tf.data] Fixes error in `dataset.concatenate()`, now throws error if two ↵Gravatar Shivani Agrawal2018-08-22
| | | | | | | | | input datasets are of type dictionaries with different set of key(s). FIXES #20626 REL_NOTES: bug fix in `tf.data.Dataset.concatenate()`, now throws error if trying to concatenate two datasets of type dictionaries with different set of key(s). PiperOrigin-RevId: 209845337
* Move from deprecated self.test_session() to self.session() when a graph is set.Gravatar A. Unique TensorFlower2018-08-21
| | | | | | self.test_session() has been deprecated in cl/208545396 as its behavior confuses readers of the test. Moving to self.session() instead. PiperOrigin-RevId: 209696110
* [tf.data] Move captured function instantiation to iterator initialization time.Gravatar Derek Murray2018-08-21
| | | | | | | | | | Previously, a function instantiation error (e.g. in `Dataset.map()`) would lead to an error in each GetNext() call that attempted to use the function. Moving this to iterator instantiation time has the benefit that the error will be reported once when the initialization op is executed, which has a more helpful stack trace, since it should not be conflated with other potential op failures. PiperOrigin-RevId: 209633511
* [tf.data] Add a `Structure` class that represents output_{types,shapes,classes}.Gravatar Derek Murray2018-08-18
| | | | | | | | | | This change adds the skeleton implementation of `Structure` for tensors, sparse tensors, and nests. Future changes will (i) add structure representations for `Dataset` and `Optional`, (ii) add support for registering custom types, and (iii) switch `Dataset`, `Iterator`, and `Optional` to use `Structure` internally. PiperOrigin-RevId: 209281967
* Remove magic-doc-links from code.Gravatar Mark Daoust2018-08-16
| | | | | | | | | | This change contains no code changes. Only doc-strings. We can't use relative links in code files, so we don't have much choice but to link to tensorflow.org/ The deleted links were to docs that no longer exist. PiperOrigin-RevId: 209019572
* BEGIN_PUBLICGravatar Akshay Agrawal2018-08-10
| | | | | | | | | Rollback breaking change. END_PUBLIC Automated rollback of commit 20622daa9db9d403a66111f5682367566e2bb8db. Revert #21038. PiperOrigin-RevId: 208256026
* Merge pull request #21038 from recogni:dev-recogni/bug_20983Gravatar TensorFlower Gardener2018-08-10
|\ | | | | | | PiperOrigin-RevId: 208232199
* | Remove usage of magic-api-link syntax from source files.Gravatar Mark Daoust2018-08-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Back-ticks are now converted to links in the api_docs generator. With the new docs repo we're moving to simplify the docs pipeline, and make everything more readable. By doing this we no longer get test failures for symbols that don't exist (`tf.does_not_exist` will not get a link). There is also no way, not to set custom link text. That's okay. This is the result of the following regex replacement (+ a couple of manual edits.): re: @\{([^$].*?)(\$.+?)?} sub: `\1` Which does the following replacements: "@{tf.symbol}" --> "`tf.symbol`" "@{tf.symbol$link_text}" --> "`tf.symbol`" PiperOrigin-RevId: 208042358
* | Merge pull request #20876 from elielhojman:masterGravatar TensorFlower Gardener2018-08-08
|\ \ | | | | | | | | | PiperOrigin-RevId: 207984120
* | | [tf.data] Add support for copying `Optional` variants to/from GPU.Gravatar Derek Murray2018-08-05
| | | | | | | | | | | | PiperOrigin-RevId: 207490563
* | | [tf.data] Raise an InvalidArgumentError if the argument to ↵Gravatar Derek Murray2018-08-03
| | | | | | | | | | | | | | | | | | | | | | | | Dataset.list_files() matches no files. The main effect of this change is to change a late `OutOfRangeError` at iteration time into an earlier `InvalidArgumentError` at initialization time, which will improve error reporting in Eager mode and high-level APIs that control the training loop (Estimator, Keras, etc.). This change will break some advanced uses that concatenate many potentially empty file listings, but it is possible to work around this using `tf.data.Dataset.from_tensor_slices(tf.matching_files(file_pattern))`. We expect that the improved productivity from an earlier, more actionable error message will outweigh the inconvenience of modifying a small number of existing programs. PiperOrigin-RevId: 207344116
* | | [tf.data] Add checkpointing for memory-based `cache()`.Gravatar Jiri Simsa2018-08-03
| | | | | | | | | | | | PiperOrigin-RevId: 207320100
* | | Split checkpoint management utility functions out of saver.pyGravatar Allen Lavoie2018-08-02
| | | | | | | | | | | | | | | | | | Pure refactor, in preparation for adding a higher level checkpoint management utility. This utility will also need to work with the Checkpoint proto, and globbing it on to saver.py seems dirty. PiperOrigin-RevId: 207179646
* | | [tf.data] Add new `Optional` type to represent missing data.Gravatar Derek Murray2018-08-01
| | | | | | | | | | | | | | | | | | | | | The `Optional` type makes it possible to represent missing values (e.g. an attempt to run `Iterator.get_next()` after the sequence has ended) without raising an error. NOTE: The `Optional` type is currently only supported on CPU, and a follow-up change will add support for other devices. After then, we will add this to the `tf.contrib.data` API, with a view to eventually migrating it to core. PiperOrigin-RevId: 207049979
* | | [Checkpointable] Make Iterator checkpointable.Gravatar Shivani Agrawal2018-08-01
| | | | | | | | | | | | | | | | | | Use object-based save/restore to make dataset/iterator checkpointable in both graph as well as eager mode. PiperOrigin-RevId: 206998349
* | | [tf.data] Mark //tensorflow/python/data/kernel_tests:iterator_ops_test as a ↵Gravatar Derek Murray2018-07-30
| | | | | | | | | | | | | | | | | | | | | | | | CUDA test. The GPU-using unit test was not being executed as a presubmit. PiperOrigin-RevId: 206704667
* | | Convert device function stack into TraceableStack for use in error message ↵Gravatar A. Unique TensorFlower2018-07-26
| | | | | | | | | | | | | | | | | | interpolation. PiperOrigin-RevId: 206120307
| | * [BugFix] Fixes #20983 - Store py_func dtypes for correct conversionGravatar Shaba Abhiram2018-07-23
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Implement wrapper class `PythonFunc` to store Tout list for appropriate dtype conversion from tensorflow types to numpy types using the `dtype.as_numpy_dtype` member. 2. Adjust BUILD file to include py_test of the script_ops_test. 3. Add script_ops_test to illustrate broken behavior (see `test_convert_empty_list`). Run the specific test suite with: bazel test //tensorflow/python:script_ops_test 4. `_convert` was being called from dataset_ops, update this to refer to the `PythonFunc` instead of the `FuncRegistry`. 5. [Review Feedback] Move script_ops tests to py_func_test.py. Also removed the BUILD rules for the deleted test file.
| * Fixed pylint errorGravatar Eliel Hojman2018-07-22
| | | | | | Line was too long
* | Fixing bug where in Eager mode datasets not intended to be on the GPU end up ↵Gravatar Rohan Jain2018-07-19
| | | | | | | | | | | | being there. PiperOrigin-RevId: 205331171
* | Making the Eager iterator use the new copy_to_device.Gravatar Rohan Jain2018-07-19
| | | | | | | | | | | | This CL gets rid of the forced placement of all eager datasets / iterators on the CPU since now we can have some datasets on the GPU. PiperOrigin-RevId: 205264791
* | Removing redundant comment.Gravatar Shivani Agrawal2018-07-17
| | | | | | | | PiperOrigin-RevId: 205011204
| * Small typo fixGravatar Eliel Hojman2018-07-17
|/
* Creating a CopyToDeviceDataset that uses GeneratorDataset and remote ↵Gravatar Rohan Jain2018-07-07
| | | | | | | | function calls to copy data from one device to another. Used in combination with PrefetchDataset, this can serve as a replacement for PrefetchToDeviceDataset. This also adds GPU kernel registrations for some iterator ops and for GeneratorDataset and PrefetchDataset PiperOrigin-RevId: 203538112
* [tf.data] Improve error message when the function return type is unsupported.Gravatar Derek Murray2018-07-07
| | | | PiperOrigin-RevId: 203525056
* [tf.data] Experimental transformations for windowing and batching of windows.Gravatar Jiri Simsa2018-07-03
| | | | | | | | This change introduces the `window` tf.data transformation, which can be used to create windows of elements (represented as a dataset) from a dataset. This transformation enables applying different batching logic to different components of a dataset. To illustrate the benefits of the transformation, this CL also introduces transformations for batching and padded batching of windows of both dense and sparse tensors. Notably, padded batching of sparse tensors was previously not possible. PiperOrigin-RevId: 203179522
* [tf.data] Add examples of `map_func` signatures to the `Dataset.map()` ↵Gravatar Derek Murray2018-06-29
| | | | | | | | documentation. Fixes #20265. PiperOrigin-RevId: 202695249
* [tf.data] In `Dataset.padded_batch()` test, handle randomly created empty ↵Gravatar Derek Murray2018-06-29
| | | | | | batches correctly. PiperOrigin-RevId: 202688283
* [tf.data] Add benchmark for batching sparse tensors.Gravatar Derek Murray2018-06-25
| | | | PiperOrigin-RevId: 201994101
* Rename programmers_guide/ directory to guide/.Gravatar Billy Lamberta2018-06-22
| | | | | | Update references in source files and docs in tensorflow and related projects. PiperOrigin-RevId: 201766994
* Replace unnecessary `()` in `run_in_graph_and_eager_modes()`.Gravatar Tom Hennigan2018-06-22
| | | | PiperOrigin-RevId: 201652888
* [tf.data] Internal change to nested `Dataset` handling in functions. No ↵Gravatar Derek Murray2018-06-21
| | | | | | | | | | functional change. This change adds experimental support in `StructuredFunctionWrapper` for taking datasets as function arguments and returning them, which is a stepping stone to creating datasets of datasets. However, the public API for such datasets (in particular how introspection using `Dataset.output_classes` etc. would work) is not settled, so this change only uses the new support in the implementation of `Dataset.flat_map()` and `tf.contrib.data.group_by_window()`. The change also folds some code into its sole point of use in `StructuredFunctionWrapper`. PiperOrigin-RevId: 201613301
* [tf.data] Updating outdated documentation for `tf.data.Dataset.batch` and ↵Gravatar Jiri Simsa2018-06-21
| | | | | | `tf.data.Dataset.padded_batch`. PiperOrigin-RevId: 201544952
* [tf.data] Adding support for tf.data.Dataset.prefetch(buffer_size=0).Gravatar Jiri Simsa2018-06-14
| | | | PiperOrigin-RevId: 200642171
* [tf.data] Convert GeneratorDataset to use StructuredFunctionWrapper.Gravatar Derek Murray2018-06-14
| | | | PiperOrigin-RevId: 200639895
* [tf.data] Add `StructuredFunctionWrapper` to encapsulate tf.data's ↵Gravatar Derek Murray2018-06-14
| | | | | | | | enhancements to Defun. This cuts down further on the boilerplate in functional tf.data transformations. PiperOrigin-RevId: 200571420
* [tf.data] Factor out a helper for creating flat args to `function.Defun`.Gravatar Derek Murray2018-06-13
| | | | | | The `defun_args()` helper flattens a nested structure down into the flat tuple of tensor types expected by `Defun`. PiperOrigin-RevId: 200441074
* [tf.data] Factor out function argument restructuring into a helper.Gravatar Derek Murray2018-06-13
| | | | | | This cuts down on the amount of repeated (or near-repeated) code in Dataset wrappers. PiperOrigin-RevId: 200424152
* [tf.data] Factor out `output_shapes` and `output_types` attr-setting code ↵Gravatar Derek Murray2018-06-13
| | | | | | into a helper. PiperOrigin-RevId: 200386950
* [Documentation]: Fix #19657Gravatar Asim Shankar2018-06-12
| | | | PiperOrigin-RevId: 200213440
* [tf.data] Improve the error messages for `Dataset.from_generator()`.Gravatar Derek Murray2018-06-11
| | | | | | | | | | In particular: * Improve the error message when the generator yields something with the wrong structure. * Improve the error message when the generator yields something with the wrong element type. PiperOrigin-RevId: 200124096
* [tf.data] Adding `drop_remainder` argument to `tf.data.Dataset.batch()` and ↵Gravatar Jiri Simsa2018-06-08
| | | | | | `tf.data.Dataset.padded_batch()`, deprecating tf.contrib.data.batch_and_drop_remainder()` and `tf.contrib.data.padded_batch_and_drop_remainder()`. PiperOrigin-RevId: 199871303
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-18 12:21:33 +0000