4.1 Agent Evaluation: Trajectory Modes, Rubric Syntax & Autoraters

The Rosku Bank Pre-Launch Failure: Right Answer, Wrong Path!

Before launching Rosku Bank’s Mortgage Underwriting Agent to production, the QA team tests it using simple final-text string matching (response_match_score). The tests pass—but in staging, three critical flaws appear:

1. The agent approves a loan by guessing the customer’s credit score instead of calling verify_identity() before check_credit_score()!
2. When a ParallelAgent runs check_fraud() and check_income() concurrently, the evaluation suite randomly fails whenever check_income() finishes first!
3. When evaluating open-ended financial explanations, lexical ROUGE matching (response_match_score) marks valid answers as failures just because the wording changed slightly!

1. Key Concept Definitions: The Agent Evaluation Stack

Evaluating an autonomous agent requires grading both the intermediate path (Trajectory) and the final output (Response & Rubrics) across five core building blocks:

1. EvalSet & EvalCase (*.evalset.json)

ADK’s golden test dataset format. Each EvalCase stores multi-turn Invocation objects containing user_content, expected intermediate_data.tool_uses (tool names + arguments), and reference final_response.

2. ToolTrajectoryCriterion

Configures tool_trajectory_avg_score inside test_config.json to verify tool sequences using three match modes: EXACT (strict 1-to-1), IN_ORDER (allows extra helper tools), or ANY_ORDER (parallel calls).

3. ADK RubricBasedCriterion

Powers rubric_based_final_response_quality_v1 and rubric_based_tool_use_quality_v1 using an LLM Autorater to grade custom pass/fail domain rules defined in RubricContent(text_property=…).

4. PointwiseMetric & RubricBasedMetric

Platform Evaluation SDK classes passed into EvalTask.evaluate() to score a single agent’s output against static criteria (PointwiseMetricPromptTemplate) or dynamically generated per-prompt rubrics.

5. PairwiseMetric (A/B Autorater)

Compares a Candidate Agent (v2) directly against a Baseline Agent (v1) side-by-side using PairwiseMetricPromptTemplate to compute Win/Loss/Tie percentages before promoting v2.

---

2. Interactive Simulator: Trajectory Match Modes & Rubric Autoraters

Switch between the 3 Evaluation Modes below and click the 3 stages in each view to inspect EXACT vs. IN_ORDER vs. ANY_ORDER trajectory matching, ADK Rubric Criteria, and Platform SDK Metrics:

1. 3 Trajectory Match Modes2. ADK Rubric Criteria Syntax3. Platform EvalTask & Metrics

← PrevStage 1 / 3Next →

MATCH_TYPE = ‘EXACT’ (DEFAULT STRICT MODE)

1. MatchType.EXACT: Zero Extra Tools, Exact Chronological Order

Syntax & Mechanism

Expected: [verify_id, execute_wire]. Actual must be [verify_id, execute_wire]. If the agent calls an extra get_time helper tool in between, EXACT fails!

Architecture Decision Rule

Use EXACT for strict compliance workflows where calling any unexpected tool or altering the execution sequence is a policy violation.

---

3. Detailed Rubric & Evaluation Syntax Matrix (ADK + Platform SDK)

Review these exact function names, criterion keys, and class parameters across ADK and the Platform Evaluation SDK:

Function / Criterion / Class Name	Key Parameters & Syntax	What It Evaluates	Primary Architectural Use Case
AgentEvaluator.evaluate() (ADK Python Runner)	AgentEvaluator.evaluate(agent_module, eval_dataset_file_path_or_dir, num_runs=2)	Runs automated pytest / CI checks against *.evalset.json or *.test.json files.	Primary programmatic entry point for running ADK evaluation suites inside CI/CD pipelines.
tool_trajectory_avg_score (ADK Criterion Key)	ToolTrajectoryCriterion(threshold=1.0, match_type="EXACT" | "IN_ORDER" | "ANY_ORDER")	Compares actual tool_uses (name + args) against expected trajectory in EvalCase.	Use EXACT for strict sequences, IN_ORDER when extra helper tools are acceptable, and ANY_ORDER for parallel tool calls.
response_match_score (ADK Criterion Key)	Float threshold (e.g., 0.8) in test_config.json	Computes lexical ROUGE-1 token overlap between actual output and reference final_response.	Fast and zero-LLM-cost, but fails when valid answers use different phrasing or synonyms.
final_response_match_v2 (ADK Criterion Key)	LlmAsAJudgeCriterion(threshold=0.8, judge_model_options=...)	Uses an LLM Autorater to verify semantic equivalence against the reference final_response.	Eliminates false negatives from response_match_score when a reference answer exists.
rubric_based_final_response_quality_v1 (ADK Rubric Criterion)	RubricBasedCriterion(threshold=0.9, rubrics=[Rubric(rubric_id="...", rubric_content=RubricContent(text_property="..."))])	Grades the agent’s final response against custom domain rules/rubrics without needing a verbatim reference string.	Grading open-ended responses for tone, legal disclosures, or formatting rules.
rubric_based_tool_use_quality_v1 (ADK Rubric Criterion)	RubricBasedCriterion(threshold=1.0, rubrics=[Rubric(...)])	Uses an LLM Autorater to evaluate how tools were selected and parameterized against qualitative rules.	Evaluating dynamic tool arguments (like free-text search queries) where static JSON equality is too brittle.
hallucinations_v1 & safety_v1 (ADK Built-In Autoraters)	LlmAsAJudgeCriterion(threshold=1.0, evaluate_intermediate_nl_responses=True)	Checks whether every claim in the final answer is strictly grounded in tool outputs (hallucinations_v1) and safe (safety_v1).	Essential pre-deployment gate to catch ungrounded fabrications.
PointwiseMetric & PointwiseMetricPromptTemplate (Platform Eval SDK)	PointwiseMetric(metric="...", metric_prompt_template=PointwiseMetricPromptTemplate(criteria={...}, rating_rubric={...}))	Grades a single agent’s output on a defined rubric score scale (e.g., 1 to 5).	Passed into EvalTask(dataset=df, metrics=[...]).evaluate() for single-model scoring.
PairwiseMetric & PairwiseMetricPromptTemplate (Platform Eval SDK)	PairwiseMetric(metric="...", baseline_model=..., metric_prompt_template=PairwiseMetricPromptTemplate(criteria={...}, rating_rubric={...}))	Compares Baseline Agent vs. Candidate Agent side-by-side to compute Win/Loss/Tie rates.	Selecting whether a new model or prompt version (v2) outperforms the current production baseline (v1).

---

4. Real-World Deep Dive: Trajectory & Rubric Brain-Twisters

Architect Question	How It Works Under the Hood in Production
“Because LLM agents are non-deterministic, a single test run might pass by luck. How does ADK handle run-to-run variance during evaluation?”	Multi-Run Averaging (num_runs): In AgentEvaluator.evaluate(..., num_runs=3), ADK executes each EvalCase multiple times and averages the resulting trajectory and rubric scores before comparing against the threshold in test_config.json!
“When should I use final_response_match_v2 vs. rubric_based_final_response_quality_v1?”	Reference Text vs. Rule Checklist: Use final_response_match_v2 when your EvalCase has a specific target answer ("The wire fee is $25") and you want to allow synonyms ("It costs twenty-five dollars"). Use rubric_based_final_response_quality_v1 when the output is open-ended (e.g., a personalized retirement plan) and you need to verify compliance rules (“Includes APR disclaimer” and “Uses empathetic tone”).
“What happens if an agent calls [check_credit, verify_id] when the golden trajectory is [verify_id, check_credit]?”	EXACT Fails, IN_ORDER Fails, ANY_ORDER Passes! Both EXACT and IN_ORDER require verify_id to happen chronologically before check_credit. Only ANY_ORDER ignores sequence order!

Golden Rule for Module 4.1

1. Parallel tool calls failing trajectory tests randomly? Switch tool_trajectory_avg_score to ANY_ORDER.
2. Valid answers failing response_match_score due to phrasing? Upgrade to final_response_match_v2 (if you have a reference answer) or rubric_based_final_response_quality_v1 / PointwiseMetric (for custom rubric criteria)!
3. Comparing Agent v1 vs. Agent v2 head-to-head? Use PairwiseMetric inside EvalTask.evaluate()!

Official Google Cloud Documentation & Further Reading

Consult the official ADK and Vertex AI GenAI Evaluation Service documentation for complete schema definitions and metric templates:

ADK Agent Evaluation & Criteria Guide →GenAI Evaluation Service (EvalTask & Rubric Metrics) →Pointwise vs. Pairwise Autorater Metrics →