diff options
author | Ravi Mistry <rmistry@google.com> | 2017-09-25 09:20:40 -0400 |
---|---|---|
committer | Skia Commit-Bot <skia-commit-bot@chromium.org> | 2017-09-25 13:53:53 +0000 |
commit | 773578544cb331f9d3e636ea31a7e5c0e9d1bec5 (patch) | |
tree | a8862bee385ba3f2ba16a3741ee671793785aefa /infra | |
parent | 5a932166e872dab74382f9cf23c9309a3b339638 (diff) |
Make ct_skps and recreate_skps recipes shell to gsutil
# Trybots already ran @patchset3
NoTry: true
Bug: skia:7065
Change-Id: Ib961c5d85f9b1491efe2539dae47f6813590b8fa
Reviewed-on: https://skia-review.googlesource.com/50500
Commit-Queue: Ravi Mistry <rmistry@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
Diffstat (limited to 'infra')
8 files changed, 78 insertions, 331 deletions
diff --git a/infra/bots/README.recipes.md b/infra/bots/README.recipes.md index baaac70c3e..a2d235e739 100644 --- a/infra/bots/README.recipes.md +++ b/infra/bots/README.recipes.md @@ -691,9 +691,9 @@ Return a list of targets to build, depending on the builder type. — **def [RunSteps](/infra/bots/recipe_modules/ct/examples/full.py#12)(api):** ### *recipes* / [ct\_skps](/infra/bots/recipes/ct_skps.py) -[DEPS](/infra/bots/recipes/ct_skps.py#9): [depot\_tools/gsutil][depot_tools/recipe_modules/gsutil], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [ct](#recipe_modules-ct), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [skia\_swarming](#recipe_modules-skia_swarming), [vars](#recipe_modules-vars) +[DEPS](/infra/bots/recipes/ct_skps.py#9): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [ct](#recipe_modules-ct), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [skia\_swarming](#recipe_modules-skia_swarming), [vars](#recipe_modules-vars) -— **def [RunSteps](/infra/bots/recipes/ct_skps.py#44)(api):** +— **def [RunSteps](/infra/bots/recipes/ct_skps.py#43)(api):** ### *recipes* / [env:examples/full](/infra/bots/recipe_modules/env/examples/full.py) [DEPS](/infra/bots/recipe_modules/env/examples/full.py#6): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/step][recipe_engine/recipe_modules/step], [env](#recipe_modules-env) @@ -755,20 +755,20 @@ Return a list of targets to build, depending on the builder type. [DEPS](/infra/bots/recipes/perf.py#12): [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [recipe\_engine/time][recipe_engine/recipe_modules/time], [core](#recipe_modules-core), [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [vars](#recipe_modules-vars) -— **def [RunSteps](/infra/bots/recipes/perf.py#351)(api):** +— **def [RunSteps](/infra/bots/recipes/perf.py#356)(api):** — **def [nanobench\_flags](/infra/bots/recipes/perf.py#29)(api, bot):** -— **def [perf\_steps](/infra/bots/recipes/perf.py#237)(api):** +— **def [perf\_steps](/infra/bots/recipes/perf.py#242)(api):** Run Skia benchmarks. ### *recipes* / [recreate\_skps](/infra/bots/recipes/recreate_skps.py) -[DEPS](/infra/bots/recipes/recreate_skps.py#9): [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [depot\_tools/gsutil][depot_tools/recipe_modules/gsutil], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars) +[DEPS](/infra/bots/recipes/recreate_skps.py#9): [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [infra](#recipe_modules-infra), [run](#recipe_modules-run), [vars](#recipe_modules-vars) Recipe for the Skia RecreateSKPs Bot. -— **def [RunSteps](/infra/bots/recipes/recreate_skps.py#59)(api):** +— **def [RunSteps](/infra/bots/recipes/recreate_skps.py#58)(api):** ### *recipes* / [run:examples/full](/infra/bots/recipe_modules/run/examples/full.py) [DEPS](/infra/bots/recipe_modules/run/examples/full.py#6): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step], [run](#recipe_modules-run), [vars](#recipe_modules-vars) @@ -804,17 +804,17 @@ benchmark Skia using skpbench. [DEPS](/infra/bots/recipes/test.py#9): [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step], [core](#recipe_modules-core), [env](#recipe_modules-env), [flavor](#recipe_modules-flavor), [run](#recipe_modules-run), [vars](#recipe_modules-vars) -— **def [RunSteps](/infra/bots/recipes/test.py#861)(api):** +— **def [RunSteps](/infra/bots/recipes/test.py#865)(api):** — **def [dm\_flags](/infra/bots/recipes/test.py#27)(api, bot):** -— **def [key\_params](/infra/bots/recipes/test.py#687)(api):** +— **def [key\_params](/infra/bots/recipes/test.py#691)(api):** Build a unique key from the builder name (as a list). E.g. arch x86 gpu GeForce320M mode MacMini4.1 os Mac10.6 -— **def [test\_steps](/infra/bots/recipes/test.py#704)(api):** +— **def [test\_steps](/infra/bots/recipes/test.py#708)(api):** Run the DM test. ### *recipes* / [update\_meta\_config](/infra/bots/recipes/update_meta_config.py) diff --git a/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json index b0f7c0b470..78e89a365e 100644 --- a/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json +++ b/infra/bots/recipes/ct_skps.expected/CT_CPU_BENCH_10k_SKPs.json @@ -1257,12 +1257,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1", @@ -1273,19 +1268,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2", @@ -1296,10 +1283,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (2)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (2)" }, { "cmd": [ @@ -1399,12 +1383,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1", @@ -1415,19 +1394,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (3)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (3)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2", @@ -1438,10 +1409,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (4)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (4)" }, { "cmd": [ @@ -1541,12 +1509,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1", @@ -1557,19 +1520,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (5)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (5)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2", @@ -1580,10 +1535,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (6)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (6)" }, { "cmd": [ @@ -1683,12 +1635,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1", @@ -1699,19 +1646,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (7)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (7)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2", @@ -1722,10 +1661,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (8)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (8)" }, { "cmd": [ @@ -1825,12 +1761,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1", @@ -1841,19 +1772,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (9)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (9)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2", @@ -1864,10 +1787,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (10)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (10)" }, { "name": "$result", diff --git a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json index 4a637ec97d..24e26a9393 100644 --- a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json +++ b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_10k_SKPs.json @@ -1272,12 +1272,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1", @@ -1288,19 +1283,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2", @@ -1311,10 +1298,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (2)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (2)" }, { "cmd": [ @@ -1414,12 +1398,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1", @@ -1430,19 +1409,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (3)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (3)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2", @@ -1453,10 +1424,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (4)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (4)" }, { "cmd": [ @@ -1556,12 +1524,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1", @@ -1572,19 +1535,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (5)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (5)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2", @@ -1595,10 +1550,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (6)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (6)" }, { "cmd": [ @@ -1698,12 +1650,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1", @@ -1714,19 +1661,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (7)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (7)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2", @@ -1737,10 +1676,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (8)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (8)" }, { "cmd": [ @@ -1840,12 +1776,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1", @@ -1856,19 +1787,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (9)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (9)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2", @@ -1879,10 +1802,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (10)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (10)" }, { "name": "$result", diff --git a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json index 39b4820cd9..1d84d0c5db 100644 --- a/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json +++ b/infra/bots/recipes/ct_skps.expected/CT_GPU_BENCH_1k_SKPs.json @@ -1272,12 +1272,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 1", @@ -1288,19 +1283,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-1/0/file 2", @@ -1311,10 +1298,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (2)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (2)" }, { "cmd": [ @@ -1414,12 +1398,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 1", @@ -1430,19 +1409,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (3)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (3)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-2/0/file 2", @@ -1453,10 +1424,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (4)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (4)" }, { "cmd": [ @@ -1556,12 +1524,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 1", @@ -1572,19 +1535,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (5)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (5)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-3/0/file 2", @@ -1595,10 +1550,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (6)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (6)" }, { "cmd": [ @@ -1698,12 +1650,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 1", @@ -1714,19 +1661,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (7)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (7)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-4/0/file 2", @@ -1737,10 +1676,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (8)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (8)" }, { "cmd": [ @@ -1840,12 +1776,7 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 1", @@ -1856,19 +1787,11 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (9)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (9)" }, { "cmd": [ - "python", - "-u", - "RECIPE_MODULE[depot_tools::gsutil]/resources/gsutil_smart_retry.py", - "--", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "----", + "gsutil", "cp", "-R", "[START_DIR]/swarming_temp_dir/outputs/ct-nanobench-5/0/file 2", @@ -1879,10 +1802,7 @@ "BOTO_CONFIG": null }, "infra_step": true, - "name": "gsutil upload json output (10)", - "~followup_annotations": [ - "@@@STEP_LINK@gsutil.upload@https://storage.cloud.google.com/skia-perf/ct/10k/2012/05/14/12/@@@" - ] + "name": "upload json output (10)" }, { "name": "$result", diff --git a/infra/bots/recipes/ct_skps.py b/infra/bots/recipes/ct_skps.py index 6f9885b40a..1dcbee1ddf 100644 --- a/infra/bots/recipes/ct_skps.py +++ b/infra/bots/recipes/ct_skps.py @@ -9,7 +9,6 @@ import math DEPS = [ 'core', 'ct', - 'depot_tools/gsutil', 'flavor', 'recipe_engine/context', 'recipe_engine/file', @@ -219,18 +218,13 @@ def RunSteps(api): output_dir = api.skia_swarming.tasks_output_dir.join( task.title).join('0') utc = api.time.utcnow() - gs_dest_dir = 'ct/%s/%d/%02d/%02d/%02d/' % ( + gs_dest_dir = 'gs://skia-perf/ct/%s/%d/%02d/%02d/%02d/' % ( ct_page_type, utc.year, utc.month, utc.day, utc.hour) for json_output in api.file.listdir( 'listdir output dir', output_dir, test_data=['file 1', 'file 2']): with api.context(env=env): - api.gsutil.upload( - name='upload json output', - source=json_output, - bucket='skia-perf', - dest=gs_dest_dir, - args=['-R'] - ) + cmd = ['gsutil', 'cp', '-R', json_output, gs_dest_dir] + api.step('upload json output', cmd=cmd, infra_step=True) except api.step.StepFailure as e: # Add SKP links for convenience. diff --git a/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json b/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json index 9e3fc7a9e7..6a9988df07 100644 --- a/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json +++ b/infra/bots/recipes/recreate_skps.expected/Housekeeper-Weekly-RecreateSKPs.json @@ -232,16 +232,13 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "--", + "gsutil", "cp", "gs://skia-buildbots/artifacts/server/.gitcookies_update-skps", "[START_DIR]/update_skps.git_cookies" ], "infra_step": true, - "name": "gsutil cp" + "name": "download gitcookies" }, { "cmd": [ diff --git a/infra/bots/recipes/recreate_skps.expected/failed_upload.json b/infra/bots/recipes/recreate_skps.expected/failed_upload.json index 7aa7275fbf..2e7c4ad013 100644 --- a/infra/bots/recipes/recreate_skps.expected/failed_upload.json +++ b/infra/bots/recipes/recreate_skps.expected/failed_upload.json @@ -245,16 +245,13 @@ }, { "cmd": [ - "python", - "-u", - "RECIPE_PACKAGE_REPO[depot_tools]/gsutil.py", - "--", + "gsutil", "cp", "gs://skia-buildbots/artifacts/server/.gitcookies_update-skps", "[START_DIR]/update_skps.git_cookies" ], "infra_step": true, - "name": "gsutil cp" + "name": "download gitcookies" }, { "cmd": [ diff --git a/infra/bots/recipes/recreate_skps.py b/infra/bots/recipes/recreate_skps.py index 60b44b69a2..155bf396d3 100644 --- a/infra/bots/recipes/recreate_skps.py +++ b/infra/bots/recipes/recreate_skps.py @@ -9,7 +9,6 @@ DEPS = [ 'core', 'depot_tools/gclient', - 'depot_tools/gsutil', 'infra', 'recipe_engine/context', 'recipe_engine/file', @@ -47,8 +46,8 @@ class DownloadGitCookies(object): self._api = api def __enter__(self): - gsutil_args = ['cp', self._gs_path, self._local_path] - self._api.gsutil(gsutil_args, use_retry_wrapper=False) + cmd = ['gsutil', 'cp', self._gs_path, self._local_path] + self._api.step('download gitcookies', cmd=cmd, infra_step=True) def __exit__(self, exc_type, _value, _traceback): if self._api.path.exists(self._local_path): |