Skip to main content

Remove an unused project gateway record

Enterprise Tier

A project gateway record should be deleted only after its URL, host authorization, and traffic have been checked. Draining the data plane first prevents live requests from reaching it while the record is removed and its routes are refreshed.

Persona: Platform operator with management API administrator access, data plane cluster access, and access to the load balancers and request logs.

Estimated time: 15 to 30 minutes, including configuration propagation and inference checks.

Outcomes​

By the end of this runbook:

  • A confirmed-unused project gateway record is removed by its UUID.
  • The remaining record's URL is saved again to refresh routes.
  • Inference is verified on the regional and shared hostnames before traffic is restored.

Prerequisites​

  • A data plane serving the project through a shared hostname, with at least one other data plane that can continue serving the project during the change.
  • A management key with platform administrator access.
  • Access to every load balancer that can send project traffic to the data plane, its generated host authorization configuration, and request logs or equivalent traffic metrics.
  • A project API key and a model enabled for the project.

Set the management API host, management key, data plane ID, organization slug, and each hostname and load balancer address used below:

export MP_API="https://<management-plane-api-host>"
export MANAGEMENT_KEY="<management-key>"
export ORGANIZATION_SLUG="<organization-slug>"
export DATA_PLANE_ID="<data-plane-id>"
export DATA_PLANE_NLB="<data-plane-network-load-balancer>"
export REGIONAL_HOST="<data-plane-hostname>"
export SHARED_HOST="<shared-hostname>"
export PROJECT_API_KEY="<project-api-key>"
export MODEL="<model-enabled-for-the-project>"

Step 1: list and record the project gateway records​

Call ListProjectGateways for the data plane. Save the full response, including the record UUIDs, project gateway IDs, display names, and URL values, for verification and rollback:

curl -sS "$MP_API/admin/v1/project-gateways?workspaceId=$DATA_PLANE_ID&pageSize=100" \
  -H "Authorization: Bearer $MANAGEMENT_KEY" \
  | tee project-gateway-records.json \
  | jq '(.projectGateways // .project_gateways)[] | {
      id,
      projectGatewayId: (.projectGatewayId // .project_gateway_id),
      displayName: name,
      url,
      projects: (.attachedProjectIds // .attached_project_ids)
    }'

If nextPageToken or next_page_token is non-empty, repeat the request with pageToken until every record has been listed. Save each page to a different file so later responses do not overwrite earlier ones. Identify the record to remove by its UUID, not its project gateway ID or display name. Record the candidate and remaining records' UUIDs, project gateway IDs, display names, exact URLs, and project associations for verification and rollback.

Step 2: prove the record is unused​

Delete the record only when all of these checks pass:

  • The candidate has no url value.
  • The data plane's generated host authorization configuration has no entry for the candidate's hostname.
  • Request logs and load balancer metrics show no traffic for that hostname during the agreed observation window, and no active load balancer or DNS record sends callers to it.
  • The remaining project gateway record has a URL and at least one attached project.

Do not treat attachedProjectIds by itself as evidence that a record is serving or unused. Project attachments can be shared across records with the same name. If the candidate has a URL, appears in host authorization, receives traffic, or its state cannot be established, stop and do not delete it.

Step 3: drain the data plane​

  1. Remove the data plane from each load balancer target set that can send it traffic for the project, or set its weight to zero.
  2. Wait for in-flight requests to finish and confirm that the remaining data plane is serving the shared hostname.
  3. Keep the data plane drained until the verification in Step 6 passes. Its readiness probe can remain green during this procedure and does not prove that inference works.

Step 4: delete the candidate and refresh routes​

Call DeleteProjectGateway with the candidate record's UUID:

export UNUSED_RECORD_ID="<candidate-record-uuid>"

curl -sS -X DELETE "$MP_API/admin/v1/project-gateways/$UNUSED_RECORD_ID" \
  -H "Authorization: Bearer $MANAGEMENT_KEY"

Immediately call SetProjectGatewayURL for the remaining record, using the exact URL recorded in Step 1. Saving the same value causes routes to be rendered for the updated record state without changing the hostname:

export REMAINING_RECORD_ID="<remaining-record-uuid>"
export REMAINING_RECORD_URL="<exact-url-from-step-1>"

curl -sS -X POST "$MP_API/admin/v1/project-gateways/$REMAINING_RECORD_ID/url" \
  -H "Authorization: Bearer $MANAGEMENT_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"url\":\"$REMAINING_RECORD_URL\"}"

If the URL update is refused because the remaining hostname is not declared by the data plane, keep the data plane drained and correct the hostname declaration before continuing.

Step 5: wait for configuration propagation​

Allow up to 5 minutes for the data plane to receive the updated authorization and routes. If cluster access is available, confirm its generated host authorization contains the remaining serving record and no longer contains the deleted record. A healthy readiness probe alone is not sufficient to continue.

Step 6: verify inference through the data plane​

Send a chat completion through the data plane's network load balancer, preserving each hostname for both TLS SNI and the HTTP Host header. Repeat until both requests return HTTP 200 with a completion, for up to 5 minutes after the URL is saved:

for host in "$REGIONAL_HOST" "$SHARED_HOST"; do
  curl -sS --connect-to "$host:443:$DATA_PLANE_NLB:443" \
    -H "Authorization: Bearer $PROJECT_API_KEY" \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK\"}]}" \
    -w '\nHTTP %{http_code}\n' \
    "https://$host/v1/chat/completions"
done

Do not restore the data plane to its load balancers until inference succeeds on both hostnames. If either request still fails after 5 minutes, save the response and request ID, keep the data plane drained, and roll back or investigate the route configuration.

Step 7: restore load balancer traffic​

Restore the data plane's previous target registration or weight on each load balancer. Confirm that requests continue to succeed through the shared hostname and that traffic is reaching the restored data plane.

Roll back​

Keep the data plane drained while restoring the record:

  1. Call CreateProjectGateway with the original projectGatewayId, the data plane ID, and the organization slug. Restore the display name if one was set. If it was empty, omit name. Omit url:

    curl -sS -X POST "$MP_API/admin/v1/project-gateways" \
      -H "Authorization: Bearer $MANAGEMENT_KEY" \
      -H 'Content-Type: application/json' \
      -d "{\"customerId\":\"$ORGANIZATION_SLUG\",\"workspaceId\":\"$DATA_PLANE_ID\",\"projectGatewayId\":\"<original-project-gateway-id>\",\"name\":\"<original-display-name>\"}"

  2. Restore any project associations recorded in Step 1 only after confirming the same-named record on another data plane does not already represent those associations.

    export RESTORED_RECORD_ID="<id returned by CreateProjectGateway>"
    export PROJECT_ID="<project-id from the saved record state>"

    curl -sS -X POST "$MP_API/admin/v1/project-gateways/$RESTORED_RECORD_ID/projects" \
      -H "Authorization: Bearer $MANAGEMENT_KEY" \
      -H 'Content-Type: application/json' \
      -d "{\"projectId\":\"$PROJECT_ID\"}"

  3. Save the remaining serving record's original URL again with SetProjectGatewayURL, as in Step 4.

  4. Confirm inference succeeds on the regional hostname through the drained data plane's network load balancer, then restore its load balancer targets as in Step 7.

Where to go next

Manage a data plane's addresses and projects

Review the addresses and project labels on a data plane.

Fail over between data planes

Verify shared-hostname inference before configuring failover.