Support multiple mirror URLs for external repos

This change improves upon 4c67807964e37cfd55bbcda4c6374fcc480bcecc.

- A urls attribute has been added to the native workspace rules, with
  the exception of maven_jar and git_repository. The Skylark repository
  API also supports multiple URLs now.

- The earlier mirrors in the list are preferred. Failover will happen
  automatically in parallel.

- The first 32kB of data is checked before choosing a mirror in order
  to evade captive portals.

- If one's Internet goes down or a download times out, then the
  download will resume automatically where it left off, provided the
  server supports RFC7233 for that particular file. Please note that
  GitHub does not support this for archive snapshots. Files should
  always be mirrored to a CDN, e.g. GCS, because they support this.

- A semaphore is now used on downloads so only 8 can happen at once.

Fixes #1814
Fixes #2131
Fixes #2008
Fixes #1968
Fixes #1717
Fixes #943
Wont fix #1194
Fixes tensorflow/tensorflow#5933
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5432
See #1607
See #821
See tensorflow/tensorflow#5080
See tensorflow/tensorflow#5029
See tensorflow/tensorflow#4583
See tensorflow/tensorflow#4058

RELNOTES: A urls attribute has been added to repository rules to support multiple mirror URLs for reliably downloading files.

--
MOS_MIGRATED_REVID=140495736

1 files changed, 36 insertions, 0 deletions

diff --git a/src/test/java/com/google/devtools/build/lib/testutil/ManualSleeper.java b/src/test/java/com/google/devtools/build/lib/testutil/ManualSleeper.java
new file mode 100644
index 0000000000..6e16253499
--- /dev/null
+++ b/src/test/java/com/google/devtools/build/lib/testutil/ManualSleeper.java
@@ -0,0 +1,36 @@
+// Copyright 2014 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.devtools.build.lib.testutil;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.devtools.build.lib.util.Sleeper;
+
+/** Fake sleeper for testing. */
+public final class ManualSleeper implements Sleeper {
+
+  private final ManualClock clock;
+
+  public ManualSleeper(ManualClock clock) {
+    this.clock = checkNotNull(clock);
+  }
+
+  @Override
+  public void sleepMillis(long milliseconds) throws InterruptedException {
+    checkArgument(milliseconds >= 0, "sleeper can't time travel");
+    clock.advanceMillis(milliseconds);
+  }
+}