Cloud Run Function for instance self-deletion (#11610)
A Cloud Functions proxy enabling GCE VMs in a Managed Instance Group to
delete themselves. GCE Managed instance groups don't have any good way
to handle autoscaling for long-running workloads. With the autoscaler
configured to scale in, instances get only 90 seconds warning to shut
down. So we set the autoscaler to only scale out and have the VMs tear
themselves down when they're down with their work. This is the approach
suggested by the managed instance group team:
https://drive.google.com/file/d/1XlwxF_0T7pUnbzhL5ePDoW-Q3GAaLO11. But
anything that brings down the VM other than a delete call to the
instance group manager API makes the VM get considered "unhealthy",
which means it gets recreated in exactly the same configuration,
regardless of any update or autoscaling settings. Making the correct
API call requires broad permissions on the instance group manager,
which we don't want to give the VMs. To scope permissions to individual
instances, this proxy service makes use of instance identity tokens to
allow an instance to make a call only to delete itself (see
https://cloud.google.com/compute/docs/instances/verifying-instance-identity).
This makes use of the GCP Cloud Functions serverless offering. It's
another level of abstraction on top of Cloud Run, where you don't even
need to create your own docker container.
I wrote unit tests for this in addition to testing the server locally,
but I don't think it's worth adding them to the CI right now because
they come with an entire server framework of dependencies and this
isn't a part of the project I expect to see active development.
Probably more worthwhile would be finding another home for this as a
standalone project separate from IREE.
skip-ci: not tested by CI
diff --git a/.gitignore b/.gitignore
index 0409f07..d2056f0 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
*.pyc
**/.ipynb_checkpoints/
.pytype/
+**/__pycache__/**
# Visual Studio files
.vs/
diff --git a/build_tools/github_actions/runner/config/health_server/health_server.py b/build_tools/github_actions/runner/config/health_server/health_server.py
index 6ad5518..9c114f1 100755
--- a/build_tools/github_actions/runner/config/health_server/health_server.py
+++ b/build_tools/github_actions/runner/config/health_server/health_server.py
@@ -21,7 +21,7 @@
import http.server
-class HealthCheckServer(http.server.BaseHTTPRequestHandler):
+class HealthCheckHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
@@ -30,7 +30,7 @@
def main(args):
- webServer = http.server.HTTPServer(("", args.port), HealthCheckServer)
+ webServer = http.server.HTTPServer(("", args.port), HealthCheckHandler)
print(f"Server started on port {args.port}. Ctrl+C to stop.")
try:
diff --git a/build_tools/github_actions/runner/instance_deleter/main.py b/build_tools/github_actions/runner/instance_deleter/main.py
new file mode 100644
index 0000000..e6234c6
--- /dev/null
+++ b/build_tools/github_actions/runner/instance_deleter/main.py
@@ -0,0 +1,252 @@
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""A Cloud Functions proxy enabling GCE VMs in a Managed Instance Group to delete themselves.
+
+GCE Managed instance groups don't have any good way to handle autoscaling for
+long-running workloads. With the autoscaler configured to scale in, instances
+get only 90 seconds warning to shut down. So we set the autoscaler to only scale
+out and have the VMs tear themselves down when they're down with their work.
+This is the approach suggested by the managed instance group team:
+
+https://drive.google.com/file/d/1XlwxF_0T7pUnbzhL5ePDoW-Q3GAaLO11
+
+But anything that brings down the VM other than a delete call to the instance
+group manager API makes the VM get considered "unhealthy", which means it gets
+recreated in exactly the same configuration, regardless of any update or
+autoscaling settings. Making the correct API call requires broad permissions on
+the instance group manager, which we don't want to give the VMs. To scope
+permissions to individual instances, this proxy service makes use of instance
+identity tokens to allow an instance to make a call only to delete itself.
+
+See
+https://cloud.google.com/compute/docs/instances/verifying-instance-identity
+
+This makes use of the GCP Cloud Functions serverless offering. It's another
+level of abstraction on top of Cloud Run, where you don't even need to create your
+own docker container. For local development:
+
+ functions-framework --target=delete_self
+ curl -X DELETE -v --header "Authorization: Bearer $(cat /tmp/token.txt)" localhost:8080
+
+You'll need to get a token that corresponds to an actual instance though or
+you'll get an error:
+
+ gcloud compute ssh github-runner-testing-presubmit-cpu-us-west1-h58j \
+ --user-output-enabled=false \
+ --command "curl -sSfL \
+ -H 'Metadata-Flavor: Google' \
+ 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=localhost&format=full'" \
+ > /tmp/token.txt
+
+To deploy:
+ gcloud functions deploy instance-self-deleter \
+ --gen2 \
+ --runtime=python310 \
+ --region=us-central1 \
+ --source=. \
+ --entry-point=delete_self \
+ --trigger-http \
+ --run-service-account=managed-instance-deleter@iree-oss.iam.gserviceaccount.com \
+ --service-account=managed-instance-deleter@iree-oss.iam.gserviceaccount.com \
+ --ingress-settings=internal-only \
+ --timeout=30s \
+ --set-env-vars ALLOWED_MIG_PATTERN='github-runner-.*'
+
+
+See https://cloud.google.com/functions/docs for more details.
+"""
+
+import os
+import re
+from http.client import (BAD_REQUEST, FORBIDDEN, INTERNAL_SERVER_ERROR,
+ NOT_FOUND, UNAUTHORIZED)
+
+import flask
+import functions_framework
+import google.api_core.exceptions
+import google.auth.exceptions
+import requests
+from google.auth import transport
+from google.cloud import compute
+from google.oauth2 import id_token
+
+AUTH_HEADER_PREFIX = "Bearer "
+MIG_METADATA_KEY = "created-by"
+ALLOWED_MIG_PATTERN_ENV_VARIABLE = "ALLOWED_MIG_PATTERN"
+
+instances_client = compute.InstancesClient()
+migs_client = compute.RegionInstanceGroupManagersClient()
+session = requests.Session()
+
+print("Server started")
+
+
+def _verify_token(token: str) -> dict:
+ """Verify token signature and return the token payload"""
+ request = transport.requests.Request(session)
+ payload = id_token.verify_oauth2_token(token, request=request)
+ return payload
+
+
+def _get_region(zone: str) -> str:
+ """Extract region name from zone name"""
+ # Drop the trailing zone identifier to get the region. Yeah it kinda does seem
+ # like there should be a better way to do this...
+ region, _ = zone.rsplit("-", maxsplit=1)
+ return region
+
+
+def _get_name_from_resource(resource: str) -> str:
+ """Extract just the final name component from a fully scoped resource name."""
+ _, name = resource.rsplit("/", maxsplit=1)
+ return name
+
+
+def _get_from_items(items: compute.Items, key: str):
+ # Why would the GCP Python API return something as silly as a dictionary?
+ return next((item.value for item in items if item.key == key), None)
+
+
+@functions_framework.http
+def delete_self(request):
+ """HTTP Cloud Function.
+ Args:
+ request (flask.Request): The request object.
+ <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
+ Returns:
+ The response text, or any set of values that can be turned into a
+ Response object using `make_response`
+ <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
+ Note:
+ For more information on how Flask integrates with Cloud
+ Functions, see the `Writing HTTP functions` page.
+ <https://cloud.google.com/functions/docs/writing/http#http_frameworks>
+ """
+ if request.method != "DELETE":
+ return flask.abort(
+ BAD_REQUEST,
+ f"Invalid method {request.method}. Only DELETE is supported.")
+
+ # No path is needed, since the token contains all the information we need.
+ if request.path != "/":
+ return flask.abort(
+ BAD_REQUEST,
+ f"Invalid request path {request.path}. Only root path is valid).")
+
+ auth_header = request.headers.get("Authorization")
+ if auth_header is None:
+ return flask.abort(UNAUTHORIZED, "Authorization header is missing")
+ if not auth_header.startswith(AUTH_HEADER_PREFIX):
+ return flask.abort(
+ UNAUTHORIZED,
+ f"Authorization header does not start with expected string"
+ f" {AUTH_HEADER_PREFIX}.")
+
+ token = auth_header[len(AUTH_HEADER_PREFIX):]
+
+ try:
+ # We don't verify audience here because Cloud IAM will have already done so
+ # and jwt's matching of audiences is exact, which means trailing slashes or
+ # http vs https matters and that's pretty brittle.
+ token_payload = _verify_token(token)
+ except (ValueError, google.auth.exceptions.GoogleAuthError) as e:
+ print(e)
+ return flask.abort(UNAUTHORIZED, "Decoding bearer token failed.")
+
+ print(f"Token payload: {token_payload}")
+
+ try:
+ compute_info = token_payload["google"]["compute_engine"]
+ except KeyError:
+ return flask.abort(
+ UNAUTHORIZED,
+ "Bearer token payload does not have expected field google.compute")
+
+ project = compute_info["project_id"]
+ zone = compute_info["zone"]
+ instance_name = compute_info["instance_name"]
+ try:
+ instance = instances_client.get(instance=instance_name,
+ project=project,
+ zone=zone)
+ except (google.api_core.exceptions.NotFound,
+ google.api_core.exceptions.Forbidden) as e:
+ print(e)
+ return flask.abort(
+ e.code,
+ f"Cannot view {instance_name} in zone={zone}, project={project}")
+
+ instance_id = int(compute_info["instance_id"])
+ # Verify it's *actually* the same instance. Names get reused, but IDs
+ # don't. For some reason you can't reference instances by their ID in any
+ # of the APIs.
+ if instance.id != instance_id:
+ return flask.abort(
+ BAD_REQUEST,
+ f"Existing instance of the same name {instance.name} has a different"
+ f" ID {instance.id} than token specifies {instance_id}.")
+
+ mig = _get_from_items(instance.metadata.items, MIG_METADATA_KEY)
+
+ if mig is None:
+ return flask.abort(BAD_REQUEST,
+ (f"Instance is not part of a managed instance group."
+ f" Did not find {MIG_METADATA_KEY} in metadata."))
+ mig = _get_name_from_resource(mig)
+
+ # General good practice would be to compile the regex once, but the only way
+ # to do that is to make it a global, which makes this difficult to test and
+ # compiling this regex should not be expensive.
+ allowed_mig_pattern = os.environ.get(ALLOWED_MIG_PATTERN_ENV_VARIABLE)
+ if allowed_mig_pattern is None:
+ flask.abort(
+ INTERNAL_SERVER_ERROR,
+ f"Missing required environment variable {ALLOWED_MIG_PATTERN_ENV_VARIABLE}"
+ )
+
+ if not re.fullmatch(allowed_mig_pattern, mig):
+ return flask.abort(FORBIDDEN, f"No access to MIG {mig}")
+
+ try:
+ operation = migs_client.delete_instances(
+ instance_group_manager=mig,
+ project=project,
+ region=_get_region(zone),
+ # For some reason we can't just use a list of instance names and need to
+ # build this RhymingRythmicJavaClasses proto. Also, unlike all the other
+ # parameters, the instance has to be a fully-specified URL for the
+ # instance, not just its name.
+ region_instance_group_managers_delete_instances_request_resource=(
+ compute.RegionInstanceGroupManagersDeleteInstancesRequest(
+ instances=[instance.self_link])))
+ except (google.api_core.exceptions.Forbidden,
+ google.api_core.exceptions.Unauthorized) as e:
+ print(e)
+ return flask.abort(e.code,
+ f"Error requesting that {mig} delete {instance_name}.")
+ except Exception as e:
+ # We'll call any other error here a server error.
+ print(e)
+ return flask.abort(INTERNAL_SERVER_ERROR,
+ f"Error requesting that {mig} delete {instance_name}.")
+
+ try:
+ # This is actually an extended operation that you have to poll to get its
+ # status, but we just check the status once because it appears that errors
+ # always show up here.
+ operation.result()
+ except google.api_core.exceptions.ClientError as e:
+ print(e)
+ # Unpack the actual usable error message
+ msg = f"Error requesting that {mig} delete {instance_name}:" "\n" + "\n".join(
+ [f"{err.code}: {err.message}" for err in e.response.error.errors])
+ print(msg)
+ # We're not actually totally sure whether this is a client or server error
+ # for the overall request, but let's call it a client error (the only client
+ # here is our VM instances, so I think we can be a bit loose).
+ return flask.abort(BAD_REQUEST, msg)
+
+ return f"{instance_name} has been marked for deletion by {mig}."
diff --git a/build_tools/github_actions/runner/instance_deleter/main_test.py b/build_tools/github_actions/runner/instance_deleter/main_test.py
new file mode 100644
index 0000000..3b3fbc5
--- /dev/null
+++ b/build_tools/github_actions/runner/instance_deleter/main_test.py
@@ -0,0 +1,450 @@
+# Copyright 2022 The IREE Authors
+#
+# Licensed under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Note that these tests are not run on CI. Doing so would require adding a ton
+# of extra dependencies and the frequency of code changes here is not worth the
+# extra maintenance burden. Please do run the tests when making changes to the
+# service though.
+
+import json
+import unittest
+from unittest import mock
+
+import google.api_core.exceptions
+import werkzeug.exceptions
+from google.cloud import compute
+from werkzeug.wrappers import Request
+
+import main
+
+# Don't rely on any of the specific values in these
+INVALID_TOKEN = "INVALID_TOKEN"
+ID1 = 1234
+ID2 = 4567
+REGION = "us-central1"
+ZONE = "us-west1-a"
+PROJECT = "iree-oss"
+INSTANCE_LINK_PREFIX = "https://www.googleapis.com/compute/v1/projects/iree-oss/zones/us-east1-b/instances/"
+INSTANCE_NAME = "some_instance_name"
+MIG_PATH_PREFIX = "projects/794014424711/regions/us-north1/instanceGroupManagers/"
+MIG_NAME = "some_mig_name"
+
+
+def get_message(ctx):
+ return ctx.exception.get_response().get_data(as_text=True)
+
+
+# A fake for oauth2 token verification that pretends the encoding scheme is just
+# JSON.
+def fake_verify_oauth2_token(token, request):
+ del request
+ return json.loads(token)
+
+
+def make_token(payload: dict):
+ return json.dumps(payload)
+
+
+@mock.patch("google.oauth2.id_token.verify_oauth2_token",
+ fake_verify_oauth2_token)
+class InstanceDeleterTest(unittest.TestCase):
+
+ def setUp(self):
+ self.addCleanup(mock.patch.stopall)
+ instances_client_patcher = mock.patch("main.instances_client",
+ autospec=True)
+ self.instances_client = instances_client_patcher.start()
+ migs_client_patcher = mock.patch("main.migs_client", autospec=True)
+ self.migs_client = migs_client_patcher.start()
+ os_environ_patcher = mock.patch.dict(
+ "os.environ", {main.ALLOWED_MIG_PATTERN_ENV_VARIABLE: ".*"})
+ self.environ = os_environ_patcher.start()
+
+ def test_happy_path(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": f"{REGION}-a",
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ self_link = f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}"
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=self_link,
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{MIG_NAME}")
+ ]))
+ self.instances_client.get.return_value = instance
+
+ ext_operation = mock.MagicMock(
+ google.api_core.extended_operation.ExtendedOperation)
+ ext_operation.result.return_value = None
+
+ response = main.delete_self(req)
+
+ self.assertIn(MIG_NAME, response)
+ self.assertIn(INSTANCE_NAME, response)
+
+ self.migs_client.delete_instances.assert_called_once_with(
+ instance_group_manager=MIG_NAME,
+ project=PROJECT,
+ region=REGION,
+ region_instance_group_managers_delete_instances_request_resource=compute
+ .RegionInstanceGroupManagersDeleteInstancesRequest(
+ instances=[instance.self_link]))
+
+ def test_narrow_allowed_migs(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": f"{REGION}-a",
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ mig_name = "github-runner-foo-bar"
+ self.environ[main.ALLOWED_MIG_PATTERN_ENV_VARIABLE] = "github-runner-.*"
+ self_link = f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}"
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=self_link,
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{mig_name}")
+ ]))
+ self.instances_client.get.return_value = instance
+
+ ext_operation = mock.MagicMock(
+ google.api_core.extended_operation.ExtendedOperation)
+ ext_operation.result.return_value = None
+
+ response = main.delete_self(req)
+
+ self.assertIn(mig_name, response)
+ self.assertIn(INSTANCE_NAME, response)
+
+ self.migs_client.delete_instances.assert_called_once_with(
+ instance_group_manager=mig_name,
+ project=PROJECT,
+ region=REGION,
+ region_instance_group_managers_delete_instances_request_resource=compute
+ .RegionInstanceGroupManagersDeleteInstancesRequest(
+ instances=[instance.self_link]))
+
+ def test_bad_method(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "GET"
+
+ with self.assertRaises(werkzeug.exceptions.BadRequest) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("Invalid method", get_message(ctx))
+
+ def test_bad_path(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+ req.path = "/foo/bar"
+
+ with self.assertRaises(werkzeug.exceptions.BadRequest) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("Invalid request path", get_message(ctx))
+
+ def test_missing_header(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ with self.assertRaises(werkzeug.exceptions.Unauthorized) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("Authorization header is missing", get_message(ctx))
+
+ def test_malformed_header(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+ req.headers = {"Authorization": "UnknownScheme token"}
+
+ with self.assertRaises(werkzeug.exceptions.Unauthorized) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("Authorization header does not start", get_message(ctx))
+
+ def test_invalid_token(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+ req.headers = {"Authorization": f"Bearer {INVALID_TOKEN}"}
+
+ with self.assertRaises(werkzeug.exceptions.Unauthorized) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("token", get_message(ctx))
+
+ def test_bad_token_payload(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({"aud": "localhost"})
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ with self.assertRaises(werkzeug.exceptions.Unauthorized) as ctx:
+ main.delete_self(req)
+
+ self.assertIn("token", get_message(ctx))
+
+ def test_nonexistent_instance(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": ZONE,
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ self.instances_client.get.side_effect = google.api_core.exceptions.NotFound(
+ "Instance not found")
+
+ with self.assertRaises(werkzeug.exceptions.NotFound) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(INSTANCE_NAME, get_message(ctx))
+
+ def test_id_mismatch(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": ZONE,
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ instance = compute.Instance(id=ID2, name=INSTANCE_NAME)
+
+ self.instances_client.get.return_value = instance
+
+ with self.assertRaises(werkzeug.exceptions.BadRequest) as ctx:
+ main.delete_self(req)
+
+ msg = get_message(ctx)
+ self.assertIn(str(ID1), msg)
+ self.assertIn(str(ID2), msg)
+
+ def test_missing_mig_metadata(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": ZONE,
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ instance = compute.Instance(id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=f"http://foo/bar/{INSTANCE_NAME}")
+
+ self.instances_client.get.return_value = instance
+
+ with self.assertRaises(werkzeug.exceptions.BadRequest) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(main.MIG_METADATA_KEY, get_message(ctx))
+
+ def test_mig_pattern_unset(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": f"{REGION}-a",
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ self_link = f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}"
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=self_link,
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{MIG_NAME}")
+ ]))
+ self.instances_client.get.return_value = instance
+
+ del self.environ[main.ALLOWED_MIG_PATTERN_ENV_VARIABLE]
+
+ with self.assertRaises(werkzeug.exceptions.InternalServerError) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(main.ALLOWED_MIG_PATTERN_ENV_VARIABLE, get_message(ctx))
+
+ def test_no_migs_allowed(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": f"{REGION}-a",
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}",
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{MIG_NAME}")
+ ]))
+ self.instances_client.get.return_value = instance
+
+ self.environ[main.ALLOWED_MIG_PATTERN_ENV_VARIABLE] = ""
+
+ with self.assertRaises(werkzeug.exceptions.Forbidden) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(MIG_NAME, get_message((ctx)))
+
+ def test_mig_not_allowed(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": f"{REGION}-a",
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ mig_name = "not-github-runner"
+ self.environ[main.ALLOWED_MIG_PATTERN_ENV_VARIABLE] = "github-runner-.*"
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}",
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{mig_name}")
+ ]))
+ self.instances_client.get.return_value = instance
+
+ with self.assertRaises(werkzeug.exceptions.Forbidden) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(mig_name, get_message((ctx)))
+
+ def test_bad_deletion_request_server(self):
+ req = Request({}, populate_request=False, shallow=True)
+ req.method = "DELETE"
+
+ token = make_token({
+ "google": {
+ "compute_engine": {
+ "project_id": PROJECT,
+ "zone": ZONE,
+ "instance_name": INSTANCE_NAME,
+ "instance_id": str(ID1),
+ }
+ }
+ })
+
+ req.headers = {"Authorization": f"Bearer {token}"}
+
+ instance = compute.Instance(
+ id=ID1,
+ name=INSTANCE_NAME,
+ zone=ZONE,
+ self_link=f"{INSTANCE_LINK_PREFIX}{INSTANCE_NAME}",
+ metadata=compute.Metadata(items=[
+ compute.Items(key=main.MIG_METADATA_KEY,
+ value=f"{MIG_PATH_PREFIX}{MIG_NAME}")
+ ]))
+
+ self.instances_client.get.return_value = instance
+ self.migs_client.delete_instances.side_effect = ValueError("Bad request")
+
+ with self.assertRaises(werkzeug.exceptions.InternalServerError) as ctx:
+ main.delete_self(req)
+
+ self.assertIn(MIG_NAME, get_message(ctx))
+
+ # Testing of server errors is unimplemented. ExtendedOperation is not
+ # documented well enough for me to produce a reasonable fake and a bad fake is
+ # worse than nothing.
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/build_tools/github_actions/runner/instance_deleter/requirements.txt b/build_tools/github_actions/runner/instance_deleter/requirements.txt
new file mode 100644
index 0000000..4865ba2
--- /dev/null
+++ b/build_tools/github_actions/runner/instance_deleter/requirements.txt
@@ -0,0 +1,6 @@
+functions-framework>=3.2
+flask>=2.1
+google-cloud-error-reporting>=1.6
+google-cloud-compute>=1.8
+google-auth>=2.15
+requests>=2.27
diff --git a/build_tools/pytype/check_diff.sh b/build_tools/pytype/check_diff.sh
index a7756b7..97604d4 100755
--- a/build_tools/pytype/check_diff.sh
+++ b/build_tools/pytype/check_diff.sh
@@ -46,14 +46,15 @@
return "${1?}"
fi
- # We disable import-error because pytype doesn't have access to bazel.
- # We disable pyi-error because of the way the bindings imports work.
- # xargs is set to high arg limits to avoid multiple Bazel invocations and will
- # hard fail if the limits are exceeded.
+ # We disable import-error and module-attr because pytype doesn't have access
+ # to all dependencies and pyi-error because of the way the bindings imports
+ # work.
+ # xargs is set to high arg limits to avoid multiple pytype invocations and
+ # will hard fail if the limits are exceeded.
# See https://github.com/bazelbuild/bazel/issues/12479
echo "${@:2}" | \
xargs --max-args 1000000 --max-chars 1000000 --exit \
- python3 -m pytype --disable=import-error,pyi-error -j $(nproc)
+ python3 -m pytype --disable=import-error,pyi-error,module-attr -j $(nproc)
EXIT_CODE="$?"
echo
if [[ "${EXIT_CODE?}" -gt "${1?}" ]]; then
