aboutsummaryrefslogtreecommitdiffhomepage
path: root/tensorflow/core/platform/cloud
Commit message (Collapse)AuthorAge
* Make GCS filesystem/metadata lookup retries configurableGravatar Russell Power2018-10-01
| | | | PiperOrigin-RevId: 215243030
* Added ABSL_DEPRECATED annotations to various deprecated TensorFlow functions.Gravatar A. Unique TensorFlower2018-09-19
| | | | PiperOrigin-RevId: 213693027
* Modify signature of WritableFile::Append to accept StringPiece data by value.Gravatar Eugene Brevdo2018-09-12
| | | | PiperOrigin-RevId: 212736286
* Removed redundant std::string -> string conversions.Gravatar A. Unique TensorFlower2018-09-04
| | | | PiperOrigin-RevId: 211487989
* Replaced calls to tensorflow::StringPiece::ToString with std::string ↵Gravatar A. Unique TensorFlower2018-08-27
| | | | | | | | | | conversions. That is, instances of sp.ToString() are replaced with string(sp). This will allow tensorflow::StringPiece::ToString to be removed, which is necessary before it can be replaced with absl::string_view. PiperOrigin-RevId: 210394878
* fix C++ header guards.Gravatar A. Unique TensorFlower2018-08-21
| | | | PiperOrigin-RevId: 209679086
* Make allowed bucket locations case insensitive.Gravatar Michael Banfield2018-08-09
| | | | PiperOrigin-RevId: 208104156
* Add a optional bucket location check to the GCS Filesystem.Gravatar Michael Banfield2018-08-02
| | | | PiperOrigin-RevId: 207152562
* Add ZoneProvider interface and GoogleZoneProvider class to detect which ↵Gravatar Michael Banfield2018-07-31
| | | | | | Google Cloud Engine zone tensorflow is running in. PiperOrigin-RevId: 206817614
* Adding `NowNanos()` to `tensorflow::Env`.Gravatar Jiri Simsa2018-07-31
| | | | PiperOrigin-RevId: 206812942
* Permit filesystem registration to be declared outside of namespace declaration.Gravatar A. Unique TensorFlower2018-07-09
| | | | PiperOrigin-RevId: 203791186
* Log final GCS cache attributes immediately prior to cache creation.Gravatar A. Unique TensorFlower2018-07-03
| | | | | | | | The GCS caching attributes can be set via several environment variables, and in some cases interact with each other. This log statement helps developers verify the final effect of these settings on the cache in use. PiperOrigin-RevId: 203181905
* Merge changes from github.Gravatar Mingxing Tan2018-06-28
| | | | PiperOrigin-RevId: 202585094
* Correctly build and link in the GCS control opsGravatar Brennan Saeta2018-06-14
| | | | PiperOrigin-RevId: 200651761
* Add GCS_READ_CACHE_DISABLED explicit env var to GcsFileSystemGravatar Nick Felt2018-05-30
| | | | PiperOrigin-RevId: 198658074
* Remove code returning bad status when the input pointer is nullptr in internalGravatar Ruoxin Sang2018-05-30
| | | | | | | functions. That should be a programmatic error and we have full control of internal functions, so it is OK to crash if error happens. PiperOrigin-RevId: 198651749
* Regard a path as a directory if it ends with "/" in GCS. This implies the ↵Gravatar Ruoxin Sang2018-05-30
| | | | | | assumption that if a real GCS object has file name ending with "/", it is always a directory mark rather than an object carrying actual contents. PiperOrigin-RevId: 198640604
* Add GCS configure ops.Gravatar Brennan Saeta2018-05-30
| | | | PiperOrigin-RevId: 198624285
* Internal changeGravatar A. Unique TensorFlower2018-05-30
| | | | PiperOrigin-RevId: 198582954
* Merge changes from github.Gravatar Yifei Feng2018-05-24
| | | | | | | Revert #18413. Too many internal test failures due to the name scope change caused by this change. Revert #18192. Cannot use re2::StringPiece internally. Need alternative for set call. Will pull and clean this up in a separate change. PiperOrigin-RevId: 197991247
* Fix CurlHttpRequest handling unexpectedly large responsesGravatar Nick Felt2018-05-23
| | | | | | | | | | | | | | This fixes a few issues with CurlHttpRequest (and correspondingly GcsFileSystem): - Return status FAILED_PRECONDITION (i.e. "your buffer was too small") when CurlHttpRequest has a direct response buffer and the response is too large for the buffer, instead of UNAVAILABLE, since if the server resource is actually a fixed size, retrying automatically won't help at all. Also, include the message about the too-small buffer size in the returned Status as opposed to logging it, making it more obvious that it's treated as a message about a hard failure versus just a warning. - If the response was actually a 416 Range Not Satisfied response, fully pretend that the response had no body even if we got one (I'm looking at you GCS... it returns a 177-byte error message). This means: - Ignore a "buffer too small" error produced by the logic described above - Don't report the length of that body in GetResultBufferDirectBytesTransferred(), which looks to the client like data corruption, just report 0 (this fix makes it match the behavior of the non-direct-buffer response handling) I also tweaked the error messages, e.g. the message that includes an HTTP response code shouldn't report the CURLcode since it will always be CURLE_OK at that point. PiperOrigin-RevId: 197805003
* Always append the trailing slash when look up or insert a directory path in ↵Gravatar Ruoxin Sang2018-05-22
| | | | | | the stat cache. PiperOrigin-RevId: 197637482
* Validate the file signatures before issusing a content read in GCS. Remove theGravatar Ruoxin Sang2018-05-18
| | | | | | | | checkpoint hack. Enable the stat cache by default, as the block cache has already been enabled by default. Set the default stat cache max_age to 5s, which is the same in the Cloud TPU server. PiperOrigin-RevId: 197175258
* ClangTidy - Readability cleanup:/code-findings-fixes.Gravatar A. Unique TensorFlower2018-05-14
| | | | | | | | | | * unused using-declarations * redundant string conversions * C-style casts * redundant get() call on smart pointer * the 'empty' method should be used to check for emptiness instead of 'size' PiperOrigin-RevId: 196585984
* Export GCS object statting streamz metrics. Fix the wrong #define Guard name ↵Gravatar Ruoxin Sang2018-05-10
| | | | | | in gcs_file_system.h. PiperOrigin-RevId: 196184962
* Clear the stat cache of the target when renaming the file.Gravatar Ruoxin Sang2018-05-04
| | | | PiperOrigin-RevId: 195337886
* Change all std::bind usages in GCS to lambdas. Fix the wrong #define Guard ↵Gravatar Ruoxin Sang2018-05-03
| | | | | | name in retrying_file_system.h. PiperOrigin-RevId: 195297877
* Replaced calls to tensorflow::StringPiece::ToString with std::string ↵Gravatar Peter Hawkins2018-05-02
| | | | | | | | | | conversions. That is, instances of sp.ToString() are replaced with std::string(sp). This will allow tensorflow::StringPiece::ToString to be removed, which is necessary before it can be replaced with absl::string_view. PiperOrigin-RevId: 195162126
* Make RetryingFileSystem a template.Gravatar Brennan Saeta2018-04-27
| | | | PiperOrigin-RevId: 194614877
* Invalidate the StatCache as well as the FileBlockCache, as once the file isGravatar Ruoxin Sang2018-04-24
| | | | | | overwritten or removed, the stat will become outdated. PiperOrigin-RevId: 194148397
* Fix a bug in GcsFileSystem that inconsistent read error may not be revealed ifGravatar A. Unique TensorFlower2018-04-18
| | | | | | the requested read size is larger than the block size. PiperOrigin-RevId: 193422905
* Switch to WaitForNotification to fix the flaky test.Gravatar Jonathan Hseu2018-04-11
| | | | | | See: https://source.cloud.google.com/results/invocations/31632a30-3728-4635-a456-f89b9e8b9dfe/log PiperOrigin-RevId: 192544848
* GCS Filesystem should not cache checkpoint file as we need to read the ↵Gravatar Sourabh Bajaj2018-04-11
| | | | | | updated checkpoints from the contents. PiperOrigin-RevId: 192517819
* Replaced calls to deprecated tensorflow::StringPiece methods with theirGravatar A. Unique TensorFlower2018-04-04
| | | | | | | | tensorflow::str_util equivalents. This will allow the deprecated methods to be removed. PiperOrigin-RevId: 191627087
* Remove all_opensource_files. It's not needed any more.Gravatar Martin Wicke2018-03-28
| | | | PiperOrigin-RevId: 190878279
* Rollback of GCS change.Gravatar Jonathan Hseu2018-03-22
| | | | | | | | | END_PUBLIC BEGIN_PUBLIC Automated g4 rollback of changelist 189900977 PiperOrigin-RevId: 190169076
* Tensorflow/GCS: Check whether we are running under GCE before trying to ↵Gravatar A. Unique TensorFlower2018-03-21
| | | | | | obtain auth token from GCE. Before this change, if a process is not running under GCE, the token request to http://metadata server would time out after 3+ minutes of retry. After this change, the check is bypassed, and we return an empty token to the caller. At that point, the caller's request to read/write a file in GCS would either succeed or fail depending on whether the bucket is publicly accessible. PiperOrigin-RevId: 189900977
* This strengthens several checks of error codes returned by libcurl. In all ↵Gravatar A. Unique TensorFlower2018-03-08
| | | | | | | | | | | | | of the cases that are changed by this CL, a failure indicates a software bug, not a runtime condition that should be handled and continued beyond. Continuing to execute only promotes silently-ignored bugs. I also removed the useless call which attempts to set the HTTP protocol to HTTP/2, because this call always fails. I opened b/74351157 to track the possible feature of adding support for HTTP/2. Also simplified the code around constructing the error string when returning actual Status objects, by moving code into a lambda. PiperOrigin-RevId: 188363531
* Fix GCS uploads occasionally failing when retrying.Gravatar Jonathan Hseu2018-03-07
| | | | | | | | | GCS returns 400, invalid argument because it thinks the body is not empty. cURL, by default, sets "Transfer-Encoding: Chunked", which causes the server to ignore "Content-Length: 0": https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding The server considers the HTTP request incomplete and may non-deterministically fail. PiperOrigin-RevId: 188235030
* Add instrumentation interfaces to the GCS file system.Gravatar Brennan Saeta2018-03-07
| | | | PiperOrigin-RevId: 188187793
* Remove dead code. We're guaranteed to have CURLE_OK because we return early ↵Gravatar Jonathan Hseu2018-03-06
| | | | | | above. PiperOrigin-RevId: 188110480
* Don't log an error if we can't set HTTP/2.Gravatar Jonathan Hseu2018-03-05
| | | | PiperOrigin-RevId: 187895652
* GCS: Extract block cache interface from implementation.Gravatar Brennan Saeta2018-03-02
| | | | PiperOrigin-RevId: 187652953
* GCS: Update throttle state even if disabled.Gravatar Brennan Saeta2018-03-02
| | | | PiperOrigin-RevId: 187647263
* Improve CURL error reporting and handling in the GCS filesystem.Gravatar Jonathan Hseu2018-03-01
| | | | | | | | - The main semantics change is that we return immediately if curl_easy_perform doesn't return CURLE_OK. The CURL documentation indicates that it's not ok to fetch info if the curl call failed: https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html - LOG errors where we can't return a status. Otherwise return with a status immediately. PiperOrigin-RevId: 187562481
* Include the response upon any error.Gravatar Jonathan Hseu2018-03-01
| | | | PiperOrigin-RevId: 187556563
* GCS: HTTP error code 308 retries during upload. Previously, it would only ↵Gravatar Jonathan Hseu2018-02-28
| | | | | | | | | | | | permit 308 when getting the status of an upload. This matches the behavior of the official library: https://github.com/google/apitools/blob/master/apitools/base/py/transfer.py#L925 And the general description here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload PiperOrigin-RevId: 187400843
* Fix testsGravatar Brennan Saeta2018-02-01
| | | | PiperOrigin-RevId: 184220615
* GCS Throttle: 1 token == 1 KbGravatar Brennan Saeta2018-02-01
| | | | | | | Previously, 1 token was approximately 256 bytes. This is slightly less intuitive than 1 kb. PiperOrigin-RevId: 184212503
* RetryingFileSystem::FlushCaches() calls the base FileSystem's FlushCaches().Gravatar A. Unique TensorFlower2018-01-30
| | | | PiperOrigin-RevId: 183869325
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-27 20:15:03 +0000