4.3 Agent Starter Pack, CI/CD Evaluation Gates & Safe Rollouts

1. Key Concept Definitions: The AgentOps & Release Stack

Moving an agent from prototype to production relies on five core release engineering building blocks:

1. Agent Starter Pack

Google Cloud’s official production accelerator that scaffolds a complete repository with Terraform IaC, CI/CD pipelines, RAG data ingestion, evaluation harnesses, and OpenTelemetry + BigQuery sinks.

2. CI/CD Evaluation Gate

Runs AgentEvaluator.evaluate() automatically during Pull Requests and Staging builds—halting deployment if trajectory or rubric scores drop below test_config.json thresholds.

3. Session-Pinned Canary

Routes a small slice (e.g., 5% to 25%) of brand-new sessions to Agent v2 while pinning existing in-flight session_id conversations to Agent v1 so state schemas never break mid-chat.

4. Blue/Green Deployment

Brings the new revision (Green) up alongside the stable revision (Blue) and switches traffic cleanly, keeping Blue warm for an instantaneous 1-click rollback if telemetry alerts fire.

5. Shadow (Dark Launch) Mode

Mirrors live production requests asynchronously to Agent v2 in the background (with all mutating/write tools mocked) to evaluate real-world performance with zero user impact.

2. Interactive Simulator: Agent Starter Pack, CI/CD Gates & Rollouts

Switch between the 3 Production Release Modes below and click the 3 stages to explore how the Agent Starter Pack, Automated CI/CD Evaluation Gates, and Stateful Traffic Rollouts work together:

AGENT STARTER PACK: SCAFFOLDING & TERRAFORM IAC

1. Multi-Environment Terraform & Agent Templates

What Happens at This Stage

Generates modular Terraform templates that provision least-privilege IAM, Artifact Registry, Cloud Storage, and runtime targets identically across Dev, Staging, and Prod.

Architecture Decision Rule

Adopt the Agent Starter Pack whenever you need to standardize Terraform IaC, CI/CD, and observability for a new agent project without weeks of manual setup!

3. Rollout Strategy Decision Matrix: Canary vs. Blue/Green vs. Shadow vs. A/B

Rollout Strategy	How Traffic Is Routed	Critical Agentic Constraint	Primary Architectural Use Case
Canary Deployment	Shifts a small slice (e.g., 5% -> 25% -> 100%) of live requests to `v2` while monitoring metrics.	Must use Session-Pinned Routing so existing `session_id` threads don’t flip between `v1` and `v2` mid-conversation!	Gradually validating a new prompt, model version, or toolset in production with minimal blast radius.
Blue/Green Deployment	Runs full `v2` (Green) alongside `v1` (Blue) and switches router traffic cleanly once verified.	Keep `v1` warm so active long-running sessions drain gracefully and rollback takes 1 click.	Major architectural upgrades requiring instant cutover and zero-downtime rollback capability.
Shadow / Dark Launch	Mirrors live user requests asynchronously to `v2` in the background; only `v1`’s reply is shown to the user.	Must mock/disable mutating tools in the shadow `v2` environment so `v2` never executes duplicate database writes or payments!	Testing a new foundation model (e.g., upgrading to a new Gemini release) on real production traffic with zero user-facing risk.
A/B Experimentation	Splits cohorts (e.g., 50% Group A / 50% Group B) with sticky user assignment to measure business KPIs.	Requires statistical sample sizing and user-level cohort sticky routing (`user_id` hash).	Measuring which agent persona or conversational flow drives higher customer conversion or CSAT scores.

4. Real-World Deep Dive: Stateful Rollout & Shadow Tool-Mocking Mechanics

Architect Question	How It Works Under the Hood in Production
“How does an API Gateway or Load Balancer actually pin an active conversation to `v1` during a Canary rollout while routing 10% of new conversations to `v2`?”	Consistent Hashing / Header-Based Routing on `session_id`: When a brand-new conversation starts, the gateway assigns it to `v1` (90%) or `v2` (10%) and tags the session metadata (or sets a signed routing cookie/header). On Turns 2 through 10, the gateway inspects the `session_id` routing tag and deterministically forwards the request to the exact revision where Turn 1 started!
“In Shadow (Dark Launch) mode, how do we prevent the Shadow `v2` agent from executing duplicate wire transfers without rewriting the agent’s Python tool code?”	Environment-Scoped MCP Server / Callback Interception: Point the Shadow `v2` deployment’s MCP Toolbox configuration (`tools.yaml`) at a read-only/sandboxed mock endpoint, OR attach an environment-aware `before_tool_callback` (`if ENV == 'SHADOW' and tool.is_mutating: return {'status': 'mocked_success'}`) so write calls are short-circuited safely!
“Why do prompt edits require the exact same CI/CD evaluation gate as Python code changes?”	Prompts Are Executable Control Flow! Changing three words in an `LlmAgent` instruction or tool `description` can alter which sub-agent gets selected or skip a compliance check—even though `flake8` and standard unit tests pass 100%. Running `AgentEvaluator.evaluate()` in CI/CD catches behavioral drift before deployment.