aboutsummaryrefslogtreecommitdiffhomepage
path: root/infra
diff options
context:
space:
mode:
authorGravatar Ravi Mistry <rmistry@google.com>2017-09-25 09:20:40 -0400
committerGravatar Skia Commit-Bot <skia-commit-bot@chromium.org>2017-09-25 13:53:53 +0000
commit773578544cb331f9d3e636ea31a7e5c0e9d1bec5 (patch)
treea8862bee385ba3f2ba16a3741ee671793785aefa /infra
parent5a932166e872dab74382f9cf23c9309a3b339638 (diff)
Make ct_skps and recreate_skps recipes shell to gsutil
# Trybots already ran @patchset3 NoTry: true Bug: skia:7065 Change-Id: Ib961c5d85f9b1491efe2539dae47f6813590b8fa Reviewed-on: https://skia-review.googlesource.com/50500 Commit-Queue: Ravi Mistry <rmistry@google.com> Reviewed-by: Eric Boren <borenet@google.com>
Diffstat (limited to 'infra')
-rw-r--r--infra/bots/README.recipes.md18
-rw-r--r--infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json120
-rw-r--r--infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json120
-rw-r--r--infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json120
-rw-r--r--infra/bots/recipes/ct_skps.py12
-rw-r--r--infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json7
-rw-r--r--infra/bots/recipes/recreate_skps.expected/failed_upload.json7
-rw-r--r--infra/bots/recipes/recreate_skps.py5
8 files changed, 78 insertions, 331 deletions
diff --git a/infra/bots/README.recipes.md b/infra/bots/README.recipes.md
index baaac70c3e..a2d235e739 100644
--- a/infra/bots/README.recipes.md
+++ b/infra/bots/README.recipes.md
@@ -691,9 +691,9 @@ Return a list of targets to build, depending on the builder type.
&mdash; **def [RunSteps](/infra/bots/recipe_modules/ct/examples/full.py#12)(api):**
### *recipes* / [ct\_skps](/infra/bots/recipes/ct_skps.py)
-[DEPS](/infra/bots/recipes/ct_skps.py#9): [depot\_tools/gsutil][depot_tools/recipe_modules/gsutil], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [ct](#recipe_modules-ct), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [skia\_swarming](#recipe_modules-skia_swarming), [vars](#recipe_modules-vars)
+[DEPS](/infra/bots/recipes/ct_skps.py#9): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [ct](#recipe_modules-ct), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [skia\_swarming](#recipe_modules-skia_swarming), [vars](#recipe_modules-vars)
-&mdash; **def [RunSteps](/infra/bots/recipes/ct_skps.py#44)(api):**
+&mdash; **def [RunSteps](/infra/bots/recipes/ct_skps.py#43)(api):**
### *recipes* / [env:examples/full](/infra/bots/recipe_modules/env/examples/full.py)
[DEPS](/infra/bots/recipe_modules/env/examples/full.py#6): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/step][recipe_engine/recipe_modules/step], [env](#recipe_modules-env)
@@ -755,20 +755,20 @@ Return a list of targets to build, depending on the builder type.
[DEPS](/infra/bots/recipes/perf.py#12): [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
-&mdash; **def [RunSteps](/infra/bots/recipes/perf.py#351)(api):**
+&mdash; **def [RunSteps](/infra/bots/recipes/perf.py#356)(api):**
&mdash; **def [nanobench\_flags](/infra/bots/recipes/perf.py#29)(api, bot):**
-&mdash; **def [perf\_steps](/infra/bots/recipes/perf.py#237)(api):**
+&mdash; **def [perf\_steps](/infra/bots/recipes/perf.py#242)(api):**
Run Skia benchmarks.
### *recipes* / [recreate\_skps](/infra/bots/recipes/recreate_skps.py)
-[DEPS](/infra/bots/recipes/recreate_skps.py#9): [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [depot\_tools/gsutil][depot_tools/recipe_modules/gsutil], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
+[DEPS](/infra/bots/recipes/recreate_skps.py#9): [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
Recipe for the Skia RecreateSKPs Bot.
-&mdash; **def [RunSteps](/infra/bots/recipes/recreate_skps.py#59)(api):**
+&mdash; **def [RunSteps](/infra/bots/recipes/recreate_skps.py#58)(api):**
### *recipes* / [run:examples/full](/infra/bots/recipe_modules/run/examples/full.py)
[DEPS](/infra/bots/recipe_modules/run/examples/full.py#6): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [run](#recipe_modules-run), [vars](#recipe_modules-vars)
@@ -804,17 +804,17 @@ benchmark Skia using skpbench.
[DEPS](/infra/bots/recipes/test.py#9): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [vars](#recipe_modules-vars)
-&mdash; **def [RunSteps](/infra/bots/recipes/test.py#861)(api):**
+&mdash; **def [RunSteps](/infra/bots/recipes/test.py#865)(api):**
&mdash; **def [dm\_flags](/infra/bots/recipes/test.py#27)(api, bot):**
-&mdash; **def [key\_params](/infra/bots/recipes/test.py#687)(api):**
+&mdash; **def [key\_params](/infra/bots/recipes/test.py#691)(api):**
Build a unique key from the builder name (as a list).
E.g. arch x86 gpu GeForce320M mode MacMini4.1 os Mac10.6
-&mdash; **def [test\_steps](/infra/bots/recipes/test.py#704)(api):**
+&mdash; **def [test\_steps](/infra/bots/recipes/test.py#708)(api):**
Run the DM test.
### *recipes* / [update\_meta\_config](/infra/bots/recipes/update_meta_config.py)
diff --git a/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json
index b0f7c0b470..78e89a365e 100644
--- a/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json
+++ b/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json
@@ -1257,12 +1257,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1",
@@ -1273,19 +1268,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2",
@@ -1296,10 +1283,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (2)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (2)"
},
{
"cmd": [
@@ -1399,12 +1383,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1",
@@ -1415,19 +1394,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (3)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (3)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2",
@@ -1438,10 +1409,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (4)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (4)"
},
{
"cmd": [
@@ -1541,12 +1509,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1",
@@ -1557,19 +1520,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (5)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (5)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2",
@@ -1580,10 +1535,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (6)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (6)"
},
{
"cmd": [
@@ -1683,12 +1635,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1",
@@ -1699,19 +1646,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (7)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (7)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2",
@@ -1722,10 +1661,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (8)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (8)"
},
{
"cmd": [
@@ -1825,12 +1761,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1",
@@ -1841,19 +1772,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (9)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (9)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2",
@@ -1864,10 +1787,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (10)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (10)"
},
{
"name": "$result",
diff --git a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json
index 4a637ec97d..24e26a9393 100644
--- a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json
+++ b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json
@@ -1272,12 +1272,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1",
@@ -1288,19 +1283,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2",
@@ -1311,10 +1298,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (2)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (2)"
},
{
"cmd": [
@@ -1414,12 +1398,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1",
@@ -1430,19 +1409,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (3)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (3)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2",
@@ -1453,10 +1424,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (4)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (4)"
},
{
"cmd": [
@@ -1556,12 +1524,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1",
@@ -1572,19 +1535,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (5)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (5)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2",
@@ -1595,10 +1550,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (6)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (6)"
},
{
"cmd": [
@@ -1698,12 +1650,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1",
@@ -1714,19 +1661,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (7)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (7)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2",
@@ -1737,10 +1676,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (8)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (8)"
},
{
"cmd": [
@@ -1840,12 +1776,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1",
@@ -1856,19 +1787,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (9)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (9)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2",
@@ -1879,10 +1802,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (10)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (10)"
},
{
"name": "$result",
diff --git a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json
index 39b4820cd9..1d84d0c5db 100644
--- a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json
+++ b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json
@@ -1272,12 +1272,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1",
@@ -1288,19 +1283,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2",
@@ -1311,10 +1298,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (2)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (2)"
},
{
"cmd": [
@@ -1414,12 +1398,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1",
@@ -1430,19 +1409,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (3)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (3)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2",
@@ -1453,10 +1424,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (4)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (4)"
},
{
"cmd": [
@@ -1556,12 +1524,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1",
@@ -1572,19 +1535,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (5)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (5)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2",
@@ -1595,10 +1550,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (6)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (6)"
},
{
"cmd": [
@@ -1698,12 +1650,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1",
@@ -1714,19 +1661,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (7)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (7)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2",
@@ -1737,10 +1676,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (8)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (8)"
},
{
"cmd": [
@@ -1840,12 +1776,7 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1",
@@ -1856,19 +1787,11 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (9)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (9)"
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py",
- "--",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "----",
+ "gsutil",
"cp",
"-R",
"[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2",
@@ -1879,10 +1802,7 @@
"BOTO_CONFIG": null
},
"infra_step": true,
- "name": "gsutil upload json output (10)",
- "~followup_annotations": [
- "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@"
- ]
+ "name": "upload json output (10)"
},
{
"name": "$result",
diff --git a/infra/bots/recipes/ct_skps.py b/infra/bots/recipes/ct_skps.py
index 6f9885b40a..1dcbee1ddf 100644
--- a/infra/bots/recipes/ct_skps.py
+++ b/infra/bots/recipes/ct_skps.py
@@ -9,7 +9,6 @@ import math
DEPS = [
'core',
'ct',
- 'depot_tools/gsutil',
'flavor',
'recipe_engine/context',
'recipe_engine/file',
@@ -219,18 +218,13 @@ def RunSteps(api):
output_dir = api.skia_swarming.tasks_output_dir.join(
task.title).join('0')
utc = api.time.utcnow()
- gs_dest_dir = 'ct/%s/%d/%02d/%02d/%02d/' % (
+ gs_dest_dir = 'gs://skia-perf/ct/%s/%d/%02d/%02d/%02d/' % (
ct_page_type, utc.year, utc.month, utc.day, utc.hour)
for json_output in api.file.listdir(
'listdir output dir', output_dir, test_data=['file 1', 'file 2']):
with api.context(env=env):
- api.gsutil.upload(
- name='upload json output',
- source=json_output,
- bucket='skia-perf',
- dest=gs_dest_dir,
- args=['-R']
- )
+ cmd = ['gsutil', 'cp', '-R', json_output, gs_dest_dir]
+ api.step('upload json output', cmd=cmd, infra_step=True)
except api.step.StepFailure as e:
# Add SKP links for convenience.
diff --git a/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json b/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json
index 9e3fc7a9e7..6a9988df07 100644
--- a/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json
+++ b/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json
@@ -232,16 +232,13 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "--",
+ "gsutil",
"cp",
"gs://skia-buildbots/artifacts/server/.gitcookies_update-skps",
"[START_DIR]/update_skps.git_cookies"
],
"infra_step": true,
- "name": "gsutil cp"
+ "name": "download gitcookies"
},
{
"cmd": [
diff --git a/infra/bots/recipes/recreate_skps.expected/failed_upload.json b/infra/bots/recipes/recreate_skps.expected/failed_upload.json
index 7aa7275fbf..2e7c4ad013 100644
--- a/infra/bots/recipes/recreate_skps.expected/failed_upload.json
+++ b/infra/bots/recipes/recreate_skps.expected/failed_upload.json
@@ -245,16 +245,13 @@
},
{
"cmd": [
- "python",
- "-u",
- "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py",
- "--",
+ "gsutil",
"cp",
"gs://skia-buildbots/artifacts/server/.gitcookies_update-skps",
"[START_DIR]/update_skps.git_cookies"
],
"infra_step": true,
- "name": "gsutil cp"
+ "name": "download gitcookies"
},
{
"cmd": [
diff --git a/infra/bots/recipes/recreate_skps.py b/infra/bots/recipes/recreate_skps.py
index 60b44b69a2..155bf396d3 100644
--- a/infra/bots/recipes/recreate_skps.py
+++ b/infra/bots/recipes/recreate_skps.py
@@ -9,7 +9,6 @@
DEPS = [
'core',
'depot_tools/gclient',
- 'depot_tools/gsutil',
'infra',
'recipe_engine/context',
'recipe_engine/file',
@@ -47,8 +46,8 @@ class DownloadGitCookies(object):
self._api = api
def __enter__(self):
- gsutil_args = ['cp', self._gs_path, self._local_path]
- self._api.gsutil(gsutil_args, use_retry_wrapper=False)
+ cmd = ['gsutil', 'cp', self._gs_path, self._local_path]
+ self._api.step('download gitcookies', cmd=cmd, infra_step=True)
def __exit__(self, exc_type, _value, _traceback):
if self._api.path.exists(self._local_path):