Java: Scripts to package and upload to Maven Central.

The general idea is that the scripts here download release artifacts (Java
sources, JNI native libraries) built using bazel as part of the TensorFlow
release process and packages them into a maven repository.

See README.md for instructions and layout of the artifacts.

Final step for #6926
Change: 151096674

8 files changed, 510 insertions, 0 deletions

diff --git a/tensorflow/java/maven/.gitignore b/tensorflow/java/maven/.gitignore
new file mode 100644
index 0000000000..50eda94643
--- /dev/null
+++ b/tensorflow/java/maven/.gitignore
@@ -0,0 +1,9 @@
+# Files generated by the release script
+# (normally cleaned up by the script as well)
+target/
+libtensorflow/src
+libtensorflow/target
+libtensorflow_jni/src
+libtensorflow_jni/target
+tensorflow/src
+tensorflow/target
diff --git a/tensorflow/java/maven/README.md b/tensorflow/java/maven/README.md
new file mode 100644
index 0000000000..885ad4afc9
--- /dev/null
+++ b/tensorflow/java/maven/README.md
@@ -0,0 +1,118 @@
+# TensorFlow for Java using Maven
+
+The [TensorFlow Java
+API](https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/package-summary)
+is available through artifacts uploaded to [Maven
+Central](https://oss.sonatype.org/content/repositories/snapshots/org/tensorflow/).
+This document describes the process of updating the release artifacts. It does
+_not_ describe how to use the artifacts, for which the reader is referred to the
+[TensorFlow for Java installation instructions](https://www.tensorflow.org/code/tensorflow/java/README.md).
+
+## Background
+
+TensorFlow source (which is primarily in C++) is built using
+[bazel](https://bazel.build) and not [maven](https://maven.apache.org/).  The
+Java API wraps over this native code and thus depends on platform (OS,
+architecture) specific native code.
+
+Hence, the process for building and uploading release artifacts is not a single
+`mvn deploy` command.
+
+## Artifact Structure
+
+There are four artifacts and thus `pom.xml`s involved in this release:
+
+1.  `tensorflow`: The single dependency for projects requiring TensorFlow for
+    Java. This convenience package depends on the two below, and is the one that
+    should typically be used in other programs.
+
+2.  `libtensorflow`: Java-only code for the [TensorFlow Java API](https://www.tensorflow.org/api_docs/java/reference/org/tensorflow/package-summary).
+    The `.jar` itself has no native code, but requires the native code be either
+    already installed on the system or made available through
+    `libtensorflow_jni`.
+
+3.  `libtensorflow_jni`: The native libraries required by `libtensorflow`.
+    Native code for all supported platforms is packaged into a single `.jar`.
+
+4.  [`parentpom`](https://maven.apache.org/pom/index.html): Common settings
+    shared between the above three.
+
+## Updating the release
+
+The TensorFlow artifacts at Maven Central are created from files built as part
+of the TensorFlow release process (which uses `bazel`). The author's lack of
+familiarity with Maven best practices combined with the use of a different build
+system means that this process is possibly not ideal, but it's what we've got.
+Suggestions are welcome.
+
+In order to isolate the environment used for building, all release processes are
+conducted in a [Docker](https://www.docker.com) container.
+
+### Pre-requisites
+
+-   `docker`
+-   An account at [oss.sonatype.org](https://oss.sonatype.org/), that has
+    permissions to update artifacts in the `org.tensorflow` group. If your
+    account does not have permissions, then you'll need to ask someone who does
+    to [file a ticket](https://issues.sonatype.org/) to add to the permissions.
+-   A GPG signing key, required [to sign the release artifacts](http://central.sonatype.org/pages/apache-maven.html#gpg-signed-components).
+
+### Deploying to Maven Central
+
+1.  Create a file with your OSSRH credentials (or perhaps you use `mvn` and have
+    it in `~/.m2/settings.xml`):
+
+    ```sh
+    SONATYPE_USERNAME="your_sonatype.org_username_here"
+    SONATYPE_PASSWORD="your_sonatype.org_password_here"
+    GPG_PASSPHRASE="your_gpg_passphrase_here"
+    cat >/tmp/settings.xml <<EOF
+<settings>
+  <servers>
+    <server>
+      <id>ossrh</id>
+      <username>${SONATYPE_USERNAME}</username>
+      <password>${SONATYPE_PASSWORD}</password>
+    </server>
+  </servers>
+  <profiles>
+    <profile>
+      <id>ossrh</id>
+      <activation>
+        <activeByDefault>true</activeByDefault>
+      </activation>
+      <properties>
+        <gpg.executable>gpg2</gpg.executable>
+        <gpg.passphrase>${GPG_PASSPHRASE}</gpg.passphrase>
+      </properties>
+    </profile>
+  </profiles>
+</settings>
+EOF
+    ```
+
+2.  Run the `release.sh` script.
+
+3.  If the script above succeeds then the artifacts would have been uploaded to
+    the private staging repository. After verifying the release, visit
+    https://oss.sonatype.org/#stagingRepositories, find the `org.tensorflow`
+    release and click on either `Release` to finalize the release, or `Drop` to
+    abort.
+
+4.  Upon successful release, commit changes to all the `pom.xml` files
+    (which should have the updated version number).
+
+### Snapshots
+
+If the `TF_VERSION` provided to the `release.sh` script ends in `-SNAPSHOT`,
+then instead of using official release files, the nightly build artifacts from
+https://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow/ and
+https://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow-windows/ will
+be used to upload to the Maven Central snapshots repository.
+
+
+## References
+
+-   [Sonatype guide](http://central.sonatype.org/pages/ossrh-guide.html) for
+    hosting releases.
+-   [Ticket that created the `org/tensorflow` configuration](https://issues.sonatype.org/browse/OSSRH-28072) on OSSRH.
diff --git a/tensorflow/java/maven/libtensorflow/pom.xml b/tensorflow/java/maven/libtensorflow/pom.xml
new file mode 100644
index 0000000000..d8d6a50da7
--- /dev/null
+++ b/tensorflow/java/maven/libtensorflow/pom.xml
@@ -0,0 +1,54 @@
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+
+  <modelVersion>4.0.0</modelVersion>
+  <description>Pure-Java code for the TensorFlow machine intelligence library.</description>
+  <parent>
+    <groupId>org.tensorflow</groupId>
+    <artifactId>parentpom</artifactId>
+    <version>1.1.0-rc0</version>
+    <relativePath>../</relativePath>
+  </parent>
+  <artifactId>libtensorflow</artifactId>
+  <packaging>jar</packaging>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-compiler-plugin</artifactId>
+        <version>3.6.1</version>
+        <configuration>
+          <source>1.8</source>
+          <target>1.8</target>
+        </configuration>
+      </plugin>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-source-plugin</artifactId>
+        <version>2.2.1</version>
+        <executions>
+          <execution>
+            <id>attach-sources</id>
+            <goals>
+              <goal>jar-no-fork</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-javadoc-plugin</artifactId>
+        <version>2.9.1</version>
+        <executions>
+          <execution>
+            <id>attach-javadocs</id>
+            <goals>
+              <goal>jar</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+  </build>
+</project>
diff --git a/tensorflow/java/maven/libtensorflow_jni/pom.xml b/tensorflow/java/maven/libtensorflow_jni/pom.xml
new file mode 100644
index 0000000000..a675859638
--- /dev/null
+++ b/tensorflow/java/maven/libtensorflow_jni/pom.xml
@@ -0,0 +1,15 @@
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+
+  <modelVersion>4.0.0</modelVersion>
+  <description>Platform-dependent native code for the TensorFlow Java library.</description>
+  <parent>
+    <groupId>org.tensorflow</groupId>
+    <artifactId>parentpom</artifactId>
+    <version>1.1.0-rc0</version>
+    <relativePath>../</relativePath>
+  </parent>
+  <artifactId>libtensorflow_jni</artifactId>
+  <packaging>jar</packaging>
+</project>
+
diff --git a/tensorflow/java/maven/pom.xml b/tensorflow/java/maven/pom.xml
new file mode 100644
index 0000000000..35f9c2ecc3
--- /dev/null
+++ b/tensorflow/java/maven/pom.xml
@@ -0,0 +1,89 @@
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+
+  <name>Parent pom.xml</name>
+  <description>Parent POM for the artifacts for TensorFlow for Java</description>
+  <modelVersion>4.0.0</modelVersion>
+  <groupId>org.tensorflow</groupId>
+  <artifactId>parentpom</artifactId>
+  <version>1.1.0-rc0</version>
+  <packaging>pom</packaging>
+
+  <url>https://www.tensorflow.org</url>
+  <inceptionYear>2015</inceptionYear>
+
+  <licenses>
+    <license>
+      <name>The Apache Software License, Version 2.0</name>
+      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
+      <distribution>repo</distribution>
+    </license>
+  </licenses>
+
+  <scm>
+    <url>https://github.com/tensorflow/tensorflow.git</url>
+    <connection>git@github.com:tensorflow/tensorflow.git</connection>
+    <developerConnection>scm:git:https://github.com/tensorflow/tensorflow.git</developerConnection>
+  </scm>
+
+  <modules>
+    <module>libtensorflow</module>
+    <module>libtensorflow_jni</module>
+    <module>tensorflow</module>
+  </modules>
+
+
+  <!-- Requirements from http://central.sonatype.org/pages/apache-maven.html -->
+  <distributionManagement>
+    <snapshotRepository>
+      <id>ossrh</id>
+      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
+    </snapshotRepository>
+    <repository>
+      <id>ossrh</id>
+      <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
+    </repository>
+  </distributionManagement>
+  <!-- http://central.sonatype.org/pages/requirements.html#developer-information -->
+  <developers>
+    <developer>
+      <name>TensorFlowers</name>
+      <organization>TensorFlow</organization>
+      <organizationUrl>http://www.tensorflow.org</organizationUrl>
+    </developer>
+  </developers>
+  <build>
+    <plugins>
+      <!-- Staging repository configuration: http://central.sonatype.org/pages/apache-maven.html#nexus-staging-maven-plugin-for-deployment-and-release -->
+      <plugin>
+        <groupId>org.sonatype.plugins</groupId>
+        <artifactId>nexus-staging-maven-plugin</artifactId>
+        <version>1.6.7</version>
+        <extensions>true</extensions>
+        <configuration>
+           <serverId>ossrh</serverId>
+           <nexusUrl>https://oss.sonatype.org/</nexusUrl>
+
+           <autoReleaseAfterClose>false</autoReleaseAfterClose>
+        </configuration>
+      </plugin>
+      <!-- GPG signed components: http://central.sonatype.org/pages/apache-maven.html#gpg-signed-components -->
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-gpg-plugin</artifactId>
+        <version>1.5</version>
+        <executions>
+          <execution>
+            <id>sign-artifacts</id>
+            <phase>verify</phase>
+            <goals>
+              <goal>sign</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+  </build>
+
+</project>
+
diff --git a/tensorflow/java/maven/release.sh b/tensorflow/java/maven/release.sh
new file mode 100755
index 0000000000..0e14bc4964
--- /dev/null
+++ b/tensorflow/java/maven/release.sh
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+#
+# Script to upload release artifacts for the TensorFlow Java library to
+# Maven Central. See README.md for an explanation.
+
+TF_VERSION="$1"
+SETTINGS_XML="$2"
+shift
+shift
+CMD="$@"
+
+if [[ -z "${TF_VERSION}" ]]
+then
+  echo "Usage: $0 <version to release> [<path to settings.xml>] ["bash" for debugging]"
+  exit 1
+fi
+
+if [[ -z "${SETTINGS_XML}" ]]
+then
+  SETTINGS_XML="$HOME/.m2/settings.xml"
+fi
+
+if [[ -z "${CMD}" ]]
+then
+  CMD="bash run_inside_container.sh"
+fi
+
+if [[ ! -f "${SETTINGS_XML}" ]]
+then
+  echo "No settings.xml (containing credentials for upload) found"
+  exit 1
+fi
+
+set -ex
+docker run \
+  -e TF_VERSION="${TF_VERSION}" \
+  -v ${PWD}:/tensorflow \
+  -v "${SETTINGS_XML}":/root/.m2/settings.xml \
+  -v ${HOME}/.gnupg:/root/.gnupg \
+  -w /tensorflow \
+  -it \
+  maven:3.3.9-jdk-8  \
+  ${CMD}
diff --git a/tensorflow/java/maven/run_inside_container.sh b/tensorflow/java/maven/run_inside_container.sh
new file mode 100644
index 0000000000..169543b12d
--- /dev/null
+++ b/tensorflow/java/maven/run_inside_container.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+#
+# This script is intended to be run inside a docker container to provide a
+# hermetic process. See release.sh for the expected invocation.
+
+
+RELEASE_URL_PREFIX="https://storage.googleapis.com/tensorflow/libtensorflow"
+IS_SNAPSHOT="false"
+if [[ "${TF_VERSION}" == *"-SNAPSHOT" ]]; then
+  IS_SNAPSHOT="true"
+fi
+
+set -ex
+
+clean() {
+  # Clean up any existing artifacts
+  # (though if run inside a clean docker container, there won't be any dirty
+  # artifacts lying around)
+  mvn -q clean
+  rm -rf libtensorflow_jni/src libtensorflow_jni/target libtensorflow/src libtensorflow/target
+}
+
+update_version_in_pom() {
+  mvn versions:set -DnewVersion="${TF_VERSION}"
+}
+
+download_libtensorflow() {
+  if [[ "${IS_SNAPSHOT}" == "true" ]]; then
+    URL="http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow/TYPE=cpu-slave/lastSuccessfulBuild/artifact/lib_package/libtensorflow-src.jar"
+  else
+    URL="${RELEASE_URL_PREFIX}/libtensorflow-src-${TF_VERSION}.jar"
+  fi
+  curl -L "${URL}" -o /tmp/src.jar
+  cd "${DIR}/libtensorflow"
+  jar -xvf /tmp/src.jar
+  rm -rf META-INF
+  cd "${DIR}"
+}
+
+download_libtensorflow_jni() {
+  NATIVE_DIR="${DIR}/libtensorflow_jni/src/main/resources/org/tensorflow/native"
+  mkdir -p "${NATIVE_DIR}"
+  cd "${NATIVE_DIR}"
+
+  mkdir linux-x86_64
+  mkdir windows-x86_64
+  mkdir darwin-x86_64
+
+  if [[ "${IS_SNAPSHOT}" == "true" ]]; then
+    # Nightly builds from http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow/
+    # and http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow-windows/
+    curl -L "http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow/TYPE=cpu-slave/lastSuccessfulBuild/artifact/lib_package/libtensorflow_jni-cpu-linux-x86_64.tar.gz" | tar -xvz -C linux-x86_64
+    curl -L "http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow/TYPE=mac-slave/lastSuccessfulBuild/artifact/lib_package/libtensorflow_jni-cpu-darwin-x86_64.tar.gz" | tar -xvz -C darwin-x86_64
+    curl -L "http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow-windows/lastSuccessfulBuild/artifact/lib_package/libtensorflow_jni-cpu-windows-x86_64.zip" -o /tmp/windows.zip
+  else
+    curl -L "${RELEASE_URL_PREFIX}/libtensorflow_jni-cpu-linux-x86_64-${TF_VERSION}.tar.gz" | tar -xvz -C linux-x86_64
+    curl -L "${RELEASE_URL_PREFIX}/libtensorflow_jni-cpu-darwin-x86_64-${TF_VERSION}.tar.gz" | tar -xvz -C darwin-x86_64
+#    curl -L "${RELEASE_URL_PREFIX}/libtensorflow_jni-cpu-windows-x86_64-${TF_VERSION}.zip" -o /tmp/windows.zip
+  fi
+
+#  unzip /tmp/windows.zip -d windows-x86_64
+#  rm -f /tmp/windows.zip
+  # Updated timestamps seem to be required to get Maven to pick up the file.
+  touch linux-x86_64/*
+  touch darwin-x86_64/*
+  touch windows-x86_64/*
+  cd "${DIR}"
+}
+
+if [ -z "${TF_VERSION}" ]
+then
+  echo "Must set the TF_VERSION environment variable"
+  exit 1
+fi
+
+DIR="$(realpath $(dirname $0))"
+cd "${DIR}"
+
+# The meat of the script.
+# Comment lines out appropriately if debugging/tinkering with the release
+# process.
+# gnupg2 is required for signing
+apt-get -qq update && apt-get -qqq install -y gnupg2
+clean
+update_version_in_pom
+download_libtensorflow
+download_libtensorflow_jni
+# Build the release artifacts
+mvn verify
+# If successfully built, try to deploy.
+# If successfully deployed, clean.
+# If deployment fails, debug with
+#   ./release.sh ${TF_VERSION} ${SETTINGS_XML} bash
+# To get a shell to poke around the maven artifacts with.
+mvn deploy && clean
+
+set +ex
+if [[ "${IS_SNAPSHOT}" == "false" ]]; then
+  echo "Uploaded to the staging repository"
+  echo "After validating the release: "
+  echo "1. Login to https://oss.sonatype.org/#stagingRepositories"
+  echo "2. Find the 'org.tensorflow' staging release and click either 'Release' to release or 'Drop' to abort"
+else
+  echo "Uploaded to the snapshot repository"
+fi
diff --git a/tensorflow/java/maven/tensorflow/pom.xml b/tensorflow/java/maven/tensorflow/pom.xml
new file mode 100644
index 0000000000..d54face7c4
--- /dev/null
+++ b/tensorflow/java/maven/tensorflow/pom.xml
@@ -0,0 +1,49 @@
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+
+  <modelVersion>4.0.0</modelVersion>
+  <description>TensorFlow for Java: A software library for machine intelligence.</description>
+  <parent>
+    <groupId>org.tensorflow</groupId>
+    <artifactId>parentpom</artifactId>
+    <version>1.1.0-rc0</version>
+    <relativePath>../</relativePath>
+  </parent>
+  <artifactId>tensorflow</artifactId>
+  <packaging>jar</packaging>
+
+  <dependencies>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>libtensorflow</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>libtensorflow_jni</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <artifactId>maven-jar-plugin</artifactId>
+        <version>3.0.2</version>
+        <executions>
+          <execution>
+            <id>default-jar</id>
+            <configuration>
+              <archive>
+                <manifestEntries>
+                  <Class-Path>libtensorflow.jar libtensorflow_jni.jar</Class-Path>
+                </manifestEntries>
+              </archive>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+  </build>
+</project>
+