4.5 Section 4 Master Takeaway: Evaluation, Runtimes & Observability Summary

1. Interactive Production Readiness Switcher: Top 3 Section 4 Design Pitfalls

Click each button below to compare common evaluation, runtime, and rollout pitfalls against the winning Google Cloud architecture:

Why the Naive Design Fails

Using MatchType.EXACT on a ParallelAgent causes random test failures when concurrent tools finish in a different order; using response_match_score (ROUGE-1) fails valid open-ended replies when phrasing varies.

The Winning Architectural Fix

Switch ToolTrajectoryCriterion to MatchType.ANY_ORDER for parallel tool calls, and use RubricBasedCriterion (rubric_based_final_response_quality_v1 or rubric_based_tool_use_quality_v1) for LLM-judged domain rules!

2. The 4-Pillar Architecture Summary Grid

4.1 Trajectory & Rubrics

EXACT, IN_ORDER, ANY_ORDER

EXACT = strict 1-to-1; IN_ORDER = extra helper tools allowed; ANY_ORDER = parallel fan-out. Grade qualitative rules with RubricBasedCriterion & PointwiseMetric.

4.2 Production Runtimes

Agent Runtime vs. Run vs. GKE

Agent Runtime = zero-ops AdkApp + native Sessions/Memory. Cloud Run = custom Docker + scale-to-zero. GKE Agent Sandbox = gVisor kernel isolation for untrusted code.

4.3 Starter Pack & CI/CD

Automated Release Gates

Agent Starter Pack scaffolds Terraform IaC, CI/CD, RAG ingestion, and BQ sinks. Block regressions with AgentEvaluator.evaluate(). Pin Canary by session_id.

4.4 Trace & Optimization

OpenTelemetry & Flywheel

Debug multi-agent latency hops in Cloud Trace (W3C traceparent). Export logs to BigQuery to harvest golden evalsets. Reserve capacity with Provisioned Throughput.

3. Deep-Dive Rubric & Evaluation Syntax Cheat Sheet

Review the exact Rubric class names, criterion keys, and template parameters across ADK and the Platform Evaluation SDK:

Rubric / Evaluation Identifier	Exact Class & Parameter Syntax	What It Grades & How It Works	Primary Selection Trigger
`rubric_based_final_response_quality_v1` (ADK Criterion Key)	`RubricBasedCriterion(threshold=0.8, judge_model_options=JudgeModelOptions(...), rubrics=[Rubric(rubric_id="...", rubric_content=RubricContent(text_property="..."))])`	Uses an LLM Autorater to grade the agent’s final response against custom pass/fail domain rules (`text_property`), averaging scores across rubrics.	Open-ended responses that must obey strict compliance disclosures, tone, or structure without a fixed reference string.
`rubric_based_tool_use_quality_v1` (ADK Criterion Key)	`RubricBasedCriterion(threshold=1.0, judge_model_options=..., rubrics=[Rubric(rubric_id="...", rubric_content=RubricContent(text_property="..."))])`	Uses an LLM Autorater to grade intermediate tool selection and parameter quality against natural-language rules.	Dynamic tool calls (e.g., generated SQL or search queries) where exact string matching in `tool_trajectory_avg_score` is too brittle.
`tool_trajectory_avg_score` (ADK Criterion Key)	`ToolTrajectoryCriterion(threshold=1.0, match_type=MatchType.EXACT \| MatchType.IN_ORDER \| MatchType.ANY_ORDER)`	Deterministically compares actual `intermediate_data.tool_uses` (tool name + args) against the golden `EvalCase`.	Verifying deterministic tool sequences (`EXACT`), prerequisite order (`IN_ORDER`), or concurrent parallel calls (`ANY_ORDER`).
`final_response_match_v2` vs. `response_match_score`	`LlmAsAJudgeCriterion(threshold=0.8)` vs. Float threshold (`0.8`)	`final_response_match_v2` uses an LLM judge for semantic equivalence against `final_response`; `response_match_score` computes lexical ROUGE-1 overlap.	Upgrade from `response_match_score` to `final_response_match_v2` when valid answers fail due to synonyms or paraphrasing!
`hallucinations_v1` & `safety_v1` (ADK Criterion Keys)	`LlmAsAJudgeCriterion(threshold=1.0, evaluate_intermediate_nl_responses=True)`	`hallucinations_v1` verifies that every factual claim in the response is strictly supported by retrieved context/tool outputs.	Preventing ungrounded fabrications in financial, medical, or legal RAG agents.
`PointwiseMetric` & `PointwiseMetricPromptTemplate` (Platform Eval SDK)	`PointwiseMetric(metric="custom_name", metric_prompt_template=PointwiseMetricPromptTemplate(criteria={"rule1": "..."}, rating_rubric={"1": "Pass", "0": "Fail"}, input_variables=[...]))`	Evaluates a single model/agent output per dataset row against custom `criteria` and `rating_rubric` definitions.	Custom single-agent rubric scoring inside `EvalTask(dataset=df, metrics=[...]).evaluate()`.
`RubricBasedMetric` (Platform Eval SDK)	`RubricBasedMetric(generation_config=RubricGenerationConfig(prompt_template=...), critique_metric=PointwiseMetric(...))`	Dynamically generates prompt-specific pass/fail rubrics per test case (or uses predefined rubrics) and critiques the response against each rubric item.	Evaluating diverse datasets where each prompt requires its own dynamically tailored checklist of rubrics!
`PairwiseMetric` & `PairwiseMetricPromptTemplate` (Platform Eval SDK)	`PairwiseMetric(metric="v1_vs_v2", baseline_model=baseline_fn, metric_prompt_template=PairwiseMetricPromptTemplate(criteria={...}, rating_rubric={...}))`	Runs Side-by-Side (A/B) LLM Autorater judging between `baseline_model` and the candidate model, outputting win/loss/tie rates.	Choosing whether Candidate Agent v2 outperforms Baseline Agent v1 before promoting v2 to production!

4. Master “If You See X, Pick Y” Decision Matrix (Section 4)

Architectural Requirement / Scenario	Recommended Google Cloud Solution	Anti-Pattern to Avoid
Fix intermittent trajectory test failures on a `ParallelAgent`	Set `match_type="ANY_ORDER"` inside `ToolTrajectoryCriterion` (`tool_trajectory_avg_score`).	Keeping `match_type="EXACT"` or removing trajectory evaluation.
Allow optional diagnostic helper tools between mandatory compliance tools	Set `match_type="IN_ORDER"` inside `ToolTrajectoryCriterion`.	Using `ANY_ORDER` (which ignores prerequisite order) or `EXACT` (which fails on helper tools).
Grade whether an agent’s reply includes mandatory legal disclaimers without a reference text	Configure `rubric_based_final_response_quality_v1` (`RubricBasedCriterion` with `RubricContent`) or `PointwiseMetric`.	`response_match_score` (ROUGE-1 requires a fixed reference string and only counts word overlap).
Execute untrusted user-submitted code or binaries with strict kernel isolation	GKE with Agent Sandbox (gVisor / microVM kernel isolation + warm sandbox pools).	Standard shared-kernel containers or unisolated local execution.
Fix lost conversation history when Cloud Run scales past 1 container instance	Configure an external persistent `SessionService` (`VertexAiSessionService` or `DatabaseSessionService`).	Leaving `InMemorySessionService` active or relying only on load-balancer sticky cookies.
Test a new agent version on live production traffic with zero user impact	Shadow / Dark Launch Mode with all mutating/write tools mocked or disabled.	Running Shadow mode against live production write APIs (causes duplicate transactions!).