Merge 3.2.x branch into master (#2648)

* Down-integrate internal changes to github.

* Update conformance test failure list.

* Explicitly import used class in nano test to avoid random test fail.

* Update _GNUC_VER to use the correct implementation of atomic operation
on Mac.

* maps_test.js: check whether Symbol is defined before using it (#2524)

Symbol is not yet available on older versions of Node.js and so this
test fails with them. This change just directly checks whether Symbol is
available before we try to use it.

* Added well_known_types_embed.cc to CLEANFILES so that it gets cleaned up

* Updated Makefile.am to fix out-of-tree builds

* Added Bazel genrule for generating well_known_types_embed.cc

In pull request #2517 I made this change for the CMake and autotools
builds but forgot to do it for the Bazel build.

* Update _GNUC_VER to use the correct implementation of atomic operation on Mac.

* Add new js file in extra dist.

* Bump version number to 3.2.0

* Fixed issue with autoloading - Invalid paths (#2538)

* PHP fix int64 decoding (#2516)

* fix int64 decoding

* fix int64 decoding + tests

* Fix int64 decoding on 32-bit machines.

* Fix warning in compiler/js/embed.cc

embed.cc: In function ‘std::string CEscape(const string&)’:
embed.cc:51:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (int i = 0; i < str.size(); ++i) {
                                ^

* Fix include in auto-generated well_known_types_embed.cc

Restore include style fix (e3da722) that has been trampled by
auto-generation of well_known_types_embed.cc

* Fixed cross compilations with the Autotools build

Pull request #2517 caused cross compilations to start failing, because
the js_embed binary was being built to run on the target platform
instead of on the build machine. This change updates the Autotools build
to use the AX_PROG_CXX_FOR_BUILD macro to find a suitable compiler for
the build machine and always use that when building js_embed.

* Minor fix for autocreated object repeated fields and maps.

- If setting/clearing a repeated field/map that was objects, check the class
  before checking the autocreator.
- Just to be paranoid, don’t mutate within copy/mutableCopy for the autocreated
  classes to ensure there is less chance of issues if someone does something
  really crazy threading wise.
- Some more tests for the internal AutocreatedArray/AutocreatedDictionary
  classes to ensure things are working as expected.
- Add Xcode 8.2 to the full_mac_build.sh supported list.

* Fix generation of extending nested messages in JavaScript (#2439)

* Fix generation of extending nested messages in JavaScript

* Added missing test8.proto to build

* Fix generated code when there is no namespace but there is enum definition.

* Decoding unknown field should succeed.

* Add embed.cc in src/Makefile.am to fix dist check.

* Fixed "make distcheck" for the Autotools build

To make the test pass I needed to fix out-of-tree builds and update
EXTRA_DIST and CLEANFILES.

* Remove redundent embed.cc from src/Makefile.am

* Update version number to 3.2.0-rc.1 (#2578)

* Change protoc-artifacts version to 3.2.0-rc.1

* Update version number to 3.2.0rc2

* Update change logs for 3.2.0 release.

* Update php README

* Update upb, fixes some bugs (including a hash table problem). (#2611)

* Update upb, fixes some bugs (including a hash table problem).

* Ruby: added a test for the previous hash table corruption.

Verified that this triggers the bug in the currently released
version.

* Ruby: bugfix for SEGV.

* Ruby: removed old code for dup'ing defs.

* Reverting deployment target to 7.0 (#2618)

The Protobuf library doesn’t require the 7.1 deployment target so
reverting it back to 7.0

* Fix typo that breaks builds on big-endian (#2632)

* Bump version number to 3.2.0

1 files changed, 82 insertions, 103 deletions

diff --git a/src/google/protobuf/compiler/cpp/cpp_file.cc b/src/google/protobuf/compiler/cpp/cpp_file.cc
index 9cfd6ec6..0e5e2f18 100644
--- a/src/google/protobuf/compiler/cpp/cpp_file.cc
+++ b/src/google/protobuf/compiler/cpp/cpp_file.cc
@@ -284,8 +284,13 @@ void FileGenerator::GenerateSource(io::Printer* printer) {
     if (IsMapEntryMessage(message_generators_[i]->descriptor_)) continue;
     printer->Print(
         "class $classname$DefaultTypeInternal : "
-        "public ::google::protobuf::internal::ExplicitlyConstructed<$classname$> {};\n"
-        "$classname$DefaultTypeInternal _$classname$_default_instance_;\n",
+        "public ::google::protobuf::internal::ExplicitlyConstructed<$classname$> {\n",
+        "classname", message_generators_[i]->classname_);
+    printer->Indent();
+    message_generators_[i]->GenerateExtraDefaultFields(printer);
+    printer->Outdent();
+    printer->Print(
+        "} _$classname$_default_instance_;\n",
         "classname", message_generators_[i]->classname_);
   }
 
@@ -301,6 +306,12 @@ void FileGenerator::GenerateSource(io::Printer* printer) {
     }
   }
 
+  printer->Print(
+      "\n"
+      "namespace $file_namespace$ {\n"
+      "\n",
+      "file_namespace", FileLevelNamespace(file_->name()));
+
   if (HasDescriptorMethods(file_, options_)) {
     printer->Print(
       "\n"
@@ -324,10 +335,6 @@ void FileGenerator::GenerateSource(io::Printer* printer) {
           "size", SimpleItoa(file_->service_count()));
     }
 
-    for (int i = 0; i < message_generators_.size(); i++) {
-      message_generators_[i]->GenerateDescriptorDeclarations(printer);
-    }
-
     printer->Print(
       "\n"
       "}  // namespace\n"
@@ -338,6 +345,12 @@ void FileGenerator::GenerateSource(io::Printer* printer) {
   // library, all this does is initialize default instances.)
   GenerateBuildDescriptors(printer);
 
+  printer->Print(
+      "\n"
+      "}  // namespace $file_namespace$\n"
+      "\n",
+      "file_namespace", FileLevelNamespace(file_->name()));
+
   // Generate enums.
   for (int i = 0; i < enum_generators_.size(); i++) {
     enum_generators_[i]->GenerateMethods(printer);
@@ -473,28 +486,16 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
 
   if (HasDescriptorMethods(file_, options_)) {
     if (!message_generators_.empty()) {
-      printer->Print(
-          "\n"
-          "const ::google::protobuf::uint32* $offsetfunname$() GOOGLE_ATTRIBUTE_COLD;\n"
-          "const ::google::protobuf::uint32* $offsetfunname$() {\n",
-          "offsetfunname", GlobalOffsetTableName(file_->name()));
-      printer->Indent();
-
-      printer->Print("static const ::google::protobuf::uint32 offsets[] = {\n");
+      printer->Print("const ::google::protobuf::uint32 TableStruct::offsets[] = {\n");
       printer->Indent();
       std::vector<std::pair<size_t, size_t> > pairs;
       for (int i = 0; i < message_generators_.size(); i++) {
         pairs.push_back(message_generators_[i]->GenerateOffsets(printer));
       }
       printer->Outdent();
-      printer->Outdent();
-      printer->Print(
-          "  };\n"
-          "  return offsets;\n"
-          "}\n"
-          "\n");
-
       printer->Print(
+          "};\n"
+          "\n"
           "static const ::google::protobuf::internal::MigrationSchema schemas[] = {\n");
       printer->Indent();
       {
@@ -508,25 +509,17 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
       printer->Outdent();
       printer->Print(
           "};\n"
-          "\n"
-          "static const ::google::protobuf::internal::DefaultInstanceData "
-          "file_default_instances[] = {\n");
+          "\nstatic "
+          "::google::protobuf::Message const * const file_default_instances[] = {\n");
       printer->Indent();
       for (int i = 0; i < message_generators_.size(); i++) {
         const Descriptor* descriptor = message_generators_[i]->descriptor_;
         if (IsMapEntryMessage(descriptor)) continue;
 
-        string oneof_default = "NULL";
-        if (message_generators_[i]->descriptor_->oneof_decl_count()) {
-          oneof_default =
-              "&" + ClassName(descriptor, false) + "_default_oneof_instance_";
-        }
         printer->Print(
-            "{reinterpret_cast<const "
-            "::google::protobuf::Message*>(&_$classname$_default_instance_), "
-            "$oneof_default$},\n",
-            "classname", ClassName(descriptor, false), "oneof_default",
-            oneof_default);
+            "reinterpret_cast<const "
+            "::google::protobuf::Message*>(&_$classname$_default_instance_),\n",
+            "classname", ClassName(descriptor, false));
       }
       printer->Outdent();
       printer->Print(
@@ -535,12 +528,11 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
     } else {
       // we still need these symbols to exist
       printer->Print(
-          "inline ::google::protobuf::uint32* $offsetfunname$() { return NULL; }\n"
+          // MSVC doesn't like empty arrays, so we add a dummy.
+          "const ::google::protobuf::uint32 TableStruct::offsets[] = { ~0u };\n"
           "static const ::google::protobuf::internal::MigrationSchema* schemas = NULL;\n"
-          "static const ::google::protobuf::internal::DefaultInstanceData* "
-          "file_default_instances = NULL;\n",
-          "offsetfunname",
-          GlobalOffsetTableName(file_->name()));
+          "static const ::google::protobuf::Message* const* "
+          "file_default_instances = NULL;\n");
     }
 
     // ---------------------------------------------------------------
@@ -557,11 +549,11 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
         // is requested *during* static init then AddDescriptors() may not have
         // been called yet, so we call it manually.  Note that it's fine if
         // AddDescriptors() is called multiple times.
-        "  $adddescriptorsname$();\n"
+        "  AddDescriptors();\n"
         "  ::google::protobuf::MessageFactory* factory = $factory$;\n"
         "  AssignDescriptors(\n"
         "      \"$filename$\", schemas, file_default_instances, "
-        "$offsetfunname$(), factory,\n"
+        "TableStruct::offsets, factory,\n"
         "      $metadata$, $enum_descriptors$, $service_descriptors$);\n"
         "}\n"
         "\n"
@@ -570,9 +562,7 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
         "  ::google::protobuf::GoogleOnceInit(&once, &protobuf_AssignDescriptors);\n"
         "}\n"
         "\n",
-        "adddescriptorsname", GlobalAddDescriptorsName(file_->name()),
-        "offsetfunname", GlobalOffsetTableName(file_->name()), "filename",
-        file_->name(), "metadata",
+        "filename", file_->name(), "metadata",
         !message_generators_.empty() ? "file_level_metadata" : "NULL",
         "enum_descriptors",
         !enum_generators_.empty() ? "file_level_enum_descriptors" : "NULL",
@@ -616,8 +606,7 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
   // ShutdownFile():  Deletes descriptors, default instances, etc. on shutdown.
   printer->Print(
     "\n"
-    "void $shutdownfilename$() {\n",
-    "shutdownfilename", GlobalShutdownFileName(file_->name()));
+    "void TableStruct::Shutdown() {\n");
   printer->Indent();
 
   for (int i = 0; i < message_generators_.size(); i++) {
@@ -630,13 +619,12 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
 
   // -----------------------------------------------------------------
 
-  // Now generate the InitDefaults() function.
+  // Now generate the InitDefaultsImpl() function.
   printer->Print(
-      "void $initdefaultsname$_impl() {\n"
+      "void TableStruct::InitDefaultsImpl() {\n"
       "  GOOGLE_PROTOBUF_VERIFY_VERSION;\n\n"
-      "",
-      // Vars.
-      "initdefaultsname", GlobalInitDefaultsName(file_->name()));
+      // Force initialization of primitive values we depend on.
+      "  ::google::protobuf::internal::InitProtobufDefaults();\n");
 
   printer->Indent();
 
@@ -645,17 +633,13 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
   for (int i = 0; i < file_->dependency_count(); i++) {
     const FileDescriptor* dependency = file_->dependency(i);
     // Print the namespace prefix for the dependency.
-    string add_desc_name = QualifiedFileLevelSymbol(
-        dependency->package(), GlobalInitDefaultsName(dependency->name()));
+    string file_namespace = QualifiedFileLevelSymbol(
+        dependency->package(), FileLevelNamespace(dependency->name()));
     // Call its AddDescriptors function.
-    printer->Print(
-      "$name$();\n",
-      "name", add_desc_name);
+    printer->Print("$file_namespace$::InitDefaults();\n", "file_namespace",
+                   file_namespace);
   }
 
-  // Force initialization of primitive values we depend on.
-  printer->Print("::google::protobuf::internal::InitProtobufDefaults();\n");
-
   // Allocate and initialize default instances.  This can't be done lazily
   // since default instances are returned by simple accessors and are used with
   // extensions.  Speaking of which, we also register extensions at this time.
@@ -672,21 +656,17 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
   printer->Print(
       "}\n"
       "\n"
-      "void $initdefaultsname$() {\n"
+      "void InitDefaults() {\n"
       "  static GOOGLE_PROTOBUF_DECLARE_ONCE(once);\n"
-      "  ::google::protobuf::GoogleOnceInit(&once, &$initdefaultsname$_impl);\n"
-      "}\n",
-      "initdefaultsname", GlobalInitDefaultsName(file_->name()));
+      "  ::google::protobuf::GoogleOnceInit(&once, &TableStruct::InitDefaultsImpl);\n"
+      "}\n");
 
   // -----------------------------------------------------------------
 
   // Now generate the AddDescriptors() function.
   printer->Print(
-      "void $adddescriptorsname$_impl() {\n"
-      "  $initdefaultsname$();\n",
-      // Vars.
-      "adddescriptorsname", GlobalAddDescriptorsName(file_->name()),
-      "initdefaultsname", GlobalInitDefaultsName(file_->name()));
+      "void AddDescriptorsImpl() {\n"
+      "  InitDefaults();\n");
 
   printer->Indent();
   if (HasDescriptorMethods(file_, options_)) {
@@ -702,12 +682,7 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
     printer->Print("static const char descriptor[] = {\n");
     printer->Indent();
 
-#ifdef _MSC_VER
-    bool breakdown_large_file = true;
-#else
-    bool breakdown_large_file = false;
-#endif
-    if (breakdown_large_file && file_data.size() > 66538) {
+    if (file_data.size() > 66535) {
       // Workaround for MSVC: "Error C1091: compiler limit: string exceeds 65535
       // bytes in length". Declare a static array of characters rather than use
       // a string literal. Only write 25 bytes per line.
@@ -748,42 +723,36 @@ void FileGenerator::GenerateBuildDescriptors(io::Printer* printer) {
   for (int i = 0; i < file_->dependency_count(); i++) {
     const FileDescriptor* dependency = file_->dependency(i);
     // Print the namespace prefix for the dependency.
-    string add_desc_name = QualifiedFileLevelSymbol(
-        dependency->package(), GlobalAddDescriptorsName(dependency->name()));
+    string file_namespace = QualifiedFileLevelSymbol(
+        dependency->package(), FileLevelNamespace(dependency->name()));
     // Call its AddDescriptors function.
-    printer->Print("$adddescriptorsname$();\n", "adddescriptorsname",
-                   add_desc_name);
+    printer->Print("$file_namespace$::AddDescriptors();\n", "file_namespace",
+                   file_namespace);
   }
 
   printer->Print(
-      "::google::protobuf::internal::OnShutdown(&$shutdownfilename$);\n",
-      "shutdownfilename", GlobalShutdownFileName(file_->name()));
+      "::google::protobuf::internal::OnShutdown(&TableStruct::Shutdown);\n");
 
   printer->Outdent();
   printer->Print(
       "}\n"
       "\n"
-      "GOOGLE_PROTOBUF_DECLARE_ONCE($adddescriptorsname$_once_);\n"
-      "void $adddescriptorsname$() {\n"
-      "  ::google::protobuf::GoogleOnceInit(&$adddescriptorsname$_once_,\n"
-      "                 &$adddescriptorsname$_impl);\n"
-      "}\n",
-      "adddescriptorsname", GlobalAddDescriptorsName(file_->name()));
+      "void AddDescriptors() {\n"
+      "  static GOOGLE_PROTOBUF_DECLARE_ONCE(once);\n"
+      "  ::google::protobuf::GoogleOnceInit(&once, &AddDescriptorsImpl);\n"
+      "}\n");
 
   if (!StaticInitializersForced(file_, options_)) {
-    printer->Print("#ifndef GOOGLE_PROTOBUF_NO_STATIC_INITIALIZER\n");
+    printer->Print("#ifdef GOOGLE_PROTOBUF_NO_STATIC_INITIALIZER\n");
   }
   printer->Print(
       // With static initializers.
       "// Force AddDescriptors() to be called at static initialization time.\n"
-      "struct StaticDescriptorInitializer_$filename$ {\n"
-      "  StaticDescriptorInitializer_$filename$() {\n"
-      "    $adddescriptorsname$();\n"
+      "struct StaticDescriptorInitializer {\n"
+      "  StaticDescriptorInitializer() {\n"
+      "    AddDescriptors();\n"
       "  }\n"
-      "} static_descriptor_initializer_$filename$_;\n",
-      // Vars.
-      "adddescriptorsname", GlobalAddDescriptorsName(file_->name()), "filename",
-      FilenameIdentifier(file_->name()));
+      "} static_descriptor_initializer;\n");
   if (!StaticInitializersForced(file_, options_)) {
     printer->Print("#endif  // GOOGLE_PROTOBUF_NO_STATIC_INITIALIZER\n");
   }
@@ -884,10 +853,11 @@ void FileGenerator::GenerateLibraryIncludes(io::Printer* printer) {
 
   // OK, it's now safe to #include other files.
   printer->Print(
-    "#include <google/protobuf/arena.h>\n"
-    "#include <google/protobuf/arenastring.h>\n"
-    "#include <google/protobuf/generated_message_util.h>\n"
-    "#include <google/protobuf/metadata.h>\n");
+      "#include <google/protobuf/io/coded_stream.h>\n"
+      "#include <google/protobuf/arena.h>\n"
+      "#include <google/protobuf/arenastring.h>\n"
+      "#include <google/protobuf/generated_message_util.h>\n"
+      "#include <google/protobuf/metadata.h>\n");
 
   if (!message_generators_.empty()) {
     if (HasDescriptorMethods(file_, options_)) {
@@ -982,12 +952,21 @@ void FileGenerator::GenerateGlobalStateFunctionDeclarations(
   // functions, so that we can declare them to be friends of each class.
   printer->Print(
       "\n"
+      "namespace $file_namespace$ {\n"
       "// Internal implementation detail -- do not call these.\n"
-      "void $dllexport_decl$$adddescriptorsname$();\n"
-      "void $dllexport_decl$$initdefaultsname$();\n",
-      "initdefaultsname", GlobalInitDefaultsName(file_->name()),
-      "adddescriptorsname", GlobalAddDescriptorsName(file_->name()),
-      "dllexport_decl",
+      "struct $dllexport_decl$TableStruct {\n"
+      "  static const ::google::protobuf::uint32 offsets[];\n"
+      // The following function(s) need to be able to access private members of
+      // the messages defined in the file. So we make them static members.
+      // This is the internal implementation of InitDefaults. It should only
+      // be called by InitDefaults which makes sure it will be called only once.
+      "  static void InitDefaultsImpl();\n"
+      "  static void Shutdown();\n"
+      "};\n"
+      "void $dllexport_decl$AddDescriptors();\n"
+      "void $dllexport_decl$InitDefaults();\n"
+      "}  // namespace $file_namespace$\n",
+      "file_namespace", FileLevelNamespace(file_->name()), "dllexport_decl",
       options_.dllexport_decl.empty() ? "" : options_.dllexport_decl + " ");
 }