diff options
author | 2018-01-03 07:54:54 -0800 | |
---|---|---|
committer | 2018-01-03 07:58:09 -0800 | |
commit | ca19540ebdb827c9ac9a237bde97065e787dbe4f (patch) | |
tree | b54019c962d8ee95fefe6165d58a01dcc4cb2de5 /tensorflow/core/ops/checkpoint_ops.cc | |
parent | 961be409bbb0d3febf8a1005e67cb6750b75806d (diff) |
Removing doc strings from REGISTER_OP calls in core/ops.
PiperOrigin-RevId: 180670333
Diffstat (limited to 'tensorflow/core/ops/checkpoint_ops.cc')
-rw-r--r-- | tensorflow/core/ops/checkpoint_ops.cc | 104 |
1 files changed, 2 insertions, 102 deletions
diff --git a/tensorflow/core/ops/checkpoint_ops.cc b/tensorflow/core/ops/checkpoint_ops.cc index 08b00c8255..5fe82e1653 100644 --- a/tensorflow/core/ops/checkpoint_ops.cc +++ b/tensorflow/core/ops/checkpoint_ops.cc @@ -38,49 +38,7 @@ REGISTER_OP("GenerateVocabRemapping") c->set_output(0, c->Vector(num_new_vocab)); c->set_output(1, c->Scalar()); return Status::OK(); - }) - .Doc(R"doc( -Given a path to new and old vocabulary files, returns a remapping Tensor of -length `num_new_vocab`, where `remapping[i]` contains the row number in the old -vocabulary that corresponds to row `i` in the new vocabulary (starting at line -`new_vocab_offset` and up to `num_new_vocab` entities), or `-1` if entry `i` -in the new vocabulary is not in the old vocabulary. The old vocabulary is -constrained to the first `old_vocab_size` entries if `old_vocab_size` is not the -default value of -1. - -`num_vocab_offset` enables -use in the partitioned variable case, and should generally be set through -examining partitioning info. The format of the files should be a text file, -with each line containing a single entity within the vocabulary. - -For example, with `new_vocab_file` a text file containing each of the following -elements on a single line: `[f0, f1, f2, f3]`, old_vocab_file = [f1, f0, f3], -`num_new_vocab = 3, new_vocab_offset = 1`, the returned remapping would be -`[0, -1, 2]`. - -The op also returns a count of how many entries in the new vocabulary -were present in the old vocabulary, which is used to calculate the number of -values to initialize in a weight matrix remapping - -This functionality can be used to remap both row vocabularies (typically, -features) and column vocabularies (typically, classes) from TensorFlow -checkpoints. Note that the partitioning logic relies on contiguous vocabularies -corresponding to div-partitioned variables. Moreover, the underlying remapping -uses an IndexTable (as opposed to an inexact CuckooTable), so client code should -use the corresponding index_table_from_file() as the FeatureColumn framework -does (as opposed to tf.feature_to_id(), which uses a CuckooTable). - -new_vocab_file: Path to the new vocab file. -old_vocab_file: Path to the old vocab file. -new_vocab_offset: How many entries into the new vocab file to start reading. -num_new_vocab: Number of entries in the new vocab file to remap. -old_vocab_size: Number of entries in the old vocab file to consider. If -1, - use the entire old vocabulary. -remapping: A Tensor of length num_new_vocab where the element at index i - is equal to the old ID that maps to the new ID i. This element is -1 for any - new ID that is not found in the old vocabulary. -num_present: Number of new vocab entries found in old vocab. -)doc"); + }); REGISTER_OP("LoadAndRemapMatrix") .Input("ckpt_path: string") @@ -109,63 +67,5 @@ REGISTER_OP("LoadAndRemapMatrix") c->set_output(0, c->Matrix(num_rows, num_cols)); return Status::OK(); - }) - .Doc(R"doc( -Loads a 2-D (matrix) `Tensor` with name `old_tensor_name` from the checkpoint -at `ckpt_path` and potentially reorders its rows and columns using the -specified remappings. - -Most users should use one of the wrapper initializers (such as -`tf.contrib.framework.load_and_remap_matrix_initializer`) instead of this -function directly. - -The remappings are 1-D tensors with the following properties: - -* `row_remapping` must have exactly `num_rows` entries. Row `i` of the output - matrix will be initialized from the row corresponding to index - `row_remapping[i]` in the old `Tensor` from the checkpoint. -* `col_remapping` must have either 0 entries (indicating that no column - reordering is needed) or `num_cols` entries. If specified, column `j` of the - output matrix will be initialized from the column corresponding to index - `col_remapping[j]` in the old `Tensor` from the checkpoint. -* A value of -1 in either of the remappings signifies a "missing" entry. In that - case, values from the `initializing_values` tensor will be used to fill that - missing row or column. If `row_remapping` has `r` missing entries and - `col_remapping` has `c` missing entries, then the following condition must be - true: - -`(r * num_cols) + (c * num_rows) - (r * c) == len(initializing_values)` - -The remapping tensors can be generated using the GenerateVocabRemapping op. - -As an example, with row_remapping = [1, 0, -1], col_remapping = [0, 2, -1], -initializing_values = [0.5, -0.5, 0.25, -0.25, 42], and w(i, j) representing -the value from row i, column j of the old tensor in the checkpoint, the output -matrix will look like the following: - -[[w(1, 0), w(1, 2), 0.5], - [w(0, 0), w(0, 2), -0.5], - [0.25, -0.25, 42]] - -ckpt_path: Path to the TensorFlow checkpoint (version 2, `TensorBundle`) from - which the old matrix `Tensor` will be loaded. -old_tensor_name: Name of the 2-D `Tensor` to load from checkpoint. -row_remapping: An int `Tensor` of row remappings (generally created by - `generate_vocab_remapping`). Even if no row remapping is needed, this must - still be an index-valued Tensor (e.g. [0, 1, 2, ...]), or a shifted - index-valued `Tensor` (e.g. [8, 9, 10, ...], for partitioned `Variables`). -col_remapping: An int `Tensor` of column remappings (generally created by - `generate_vocab_remapping`). May be a size-0 `Tensor` if only row remapping - is to be done (e.g. column ordering is the same). -initializing_values: A float `Tensor` containing values to fill in for cells - in the output matrix that are not loaded from the checkpoint. Length must be - exactly the same as the number of missing / new cells. -num_rows: Number of rows (length of the 1st dimension) in the output matrix. -num_cols: Number of columns (length of the 2nd dimension) in the output matrix. -max_rows_in_memory: The maximum number of rows to load from the checkpoint at - once. If less than or equal to 0, the entire matrix will be loaded into - memory. Setting this arg trades increased disk reads for lower memory usage. -output_matrix: Output matrix containing existing values loaded from the - checkpoint, and with any missing values filled in from initializing_values. -)doc"); + }); } // namespace tensorflow |