Pattern Match

Generic Metric

Introduction

The Pattern Match metric validates that your AI Agent's response matches a specific regex (regular expression) pattern. Unlike other metrics, this one is deterministic — it does not use an LLM for evaluation. The response either matches the pattern or it doesn't.

---

When to Use This Metric

• You need to verify the response contains specific keywords or phrases.
• You're testing that the Agent's output follows a required format (e.g., dates, phone numbers, IDs).
• You want a fast, deterministic check that doesn't depend on LLM judgment.
• You need to validate structured outputs like JSON keys, URLs, or email addresses.
• You're testing that the Agent includes required disclaimers or legal text.

---

Configuration

Parameter	Type	Default	Required	Description
threshold	float	1.0	No	Score threshold for passing (0.0–1.0).
strict_mode	boolean	false	No	Rounds score to 1.0 or 0.0 based on threshold.
pattern	string	—	Yes	The regex pattern to match against the AI's output.
ignore_case	boolean	false	No	Whether to ignore case when matching the pattern.

warning

The pattern parameter is required. You must provide a valid regex pattern. Without it, the metric cannot perform the evaluation.

---

How It Works

1. The AI Agent receives the input message and generates a response.
2. The metric applies the configured regex pattern to the response text.
3. If ignore_case is enabled, the pattern matching is case-insensitive.
4. The score is 1.0 if the pattern is found anywhere in the response, or 0.0 if not.

---

Scoring

• Range: 0.0 or 1.0 (binary).
• 1.0: The response matches the regex pattern.
• 0.0: The response does not match the regex pattern.
• Pass condition: The score must be greater than or equal to the configured threshold (default 1.0, meaning the pattern must match).

---

Example

Pattern: \b[A-Z]{2,3}-\d{4,6}\b

Input: "What's the status of my support ticket?"

AI Response: "Your support ticket TKT-12345 is currently being reviewed by our team. We'll have an update for you within 24 hours."

Score: 1.0

Result: Passed (threshold: 1.0)

The response contains "TKT-12345", which matches the pattern for a ticket ID format.

---

Common Regex Patterns

Here are some useful patterns for common validation scenarios:

Use Case	Pattern	Matches
Email address	[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}	user@example.com
Phone number (US)	\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}	(555) 123-4567
URL	https?://[^\s]+	https://example.com
Date (YYYY-MM-DD)	\d{4}-\d{2}-\d{2}	2025-01-15
Ticket/Order ID	[A-Z]{2,5}-\d{4,8}	TKT-12345
Contains keyword	refund policy	Any response mentioning "refund policy"
JSON-like key	"status"\s*:\s*"[^"]+"	"status": "active"

---

Tips for Improving Scores

• Test your regex pattern on sample responses before using it in an eval case.
• Use ignore_case: true when you don't care about letter casing.
• Keep patterns as simple as possible — complex regex can be hard to debug when tests fail.
• If you need to match one of several possible patterns, use alternation (e.g., pattern1|pattern2).
• Remember that the pattern uses a "search" behavior — it matches if the pattern is found anywhere in the response, not just at the beginning.