Every engineering team deploying large language models goes through the same initial honeymoon phase: an engineer tweaks a prompt, tests five tricky queries in an interactive playground, watches the model produce articulate answers, and concludes: "Looks great, ship it."
In AI engineering, this is colloquially known as vibe checking. And in any system handling real production volume, vibe checking is the single most common cause of catastrophic failure.
┌───────────────────────────────────────────────────────────────────────────┐
│ THE VIBE CHECKING TRAP │
├───────────────────────────────────────────────────────────────────────────┤
│ Prompt v1 ────────► 5 Manual Tests (Pass) ──► Ship to Prod │
│ │ │
│ Tweak Prompt for Case A │ │
│ ▼ │
│ Prompt v2 ────────► Fixes Case A │
│ Silent Regress Cases B, C, D (-28%) ──► Prod Outage │
└───────────────────────────────────────────────────────────────────────────┘
The fundamental problem with large language models is not that they are stochastic—it is that prompt changes are non-locally coupled. Fixing a formatting flaw for customer invoices frequently degrades extraction precision for credit notes, yet manual spot-testing will almost never catch the regression before your customers do.
To build software people can actually depend on, we must treat LLM outputs with the same rigor we apply to database migrations and distributed consensus protocols. Here is how we design and deploy deterministic evaluation gates at RightQA.
1. The Three Layers of Production AI Evaluation
A reliable evaluation pipeline does not rely on a single monolithic test. It separates verification into three decoupled layers:
Layer A: Deterministic Invariant Checks (Zero Cost, Zero Latency)
Before spending tokens on model-based judging, test everything that can be evaluated deterministically:
- Schema & Type Enforcement: Did the response parse into the expected Pydantic schema?
- Regex Boundary Audits: Are required identifiers (order IDs, SKU codes, ISO dates) present in canonical formats?
- Forbidden Substrings: Did the model leak system prompt instructions or proprietary prompt templates?
- Latency & Token Ceilings: Did execution exceed the 1,800ms budget or consume more than 2,048 tokens?
Layer B: Retrieval Groundedness & Context Relevance
For RAG architectures, 70% of downstream hallucinations originate in the retrieval stage rather than the generator:
- Context Relevance: What fraction of the chunks retrieved by the vector index actually pertain to the user's intent?
- Faithfulness / Groundedness: Can every single factual claim in the generated answer be directly mapped to a specific citation in the retrieved context?
Layer C: Semantic Behavioral Judges (Calibrated LLM-as-a-Judge)
For open-ended generation, stylistic compliance, and reasoning chains, we use an evaluator model running on a strict rubric with calibrated few-shot anchors.
2. Implementing a Production Pytest Eval Gate
Here is a simplified version of the automated evaluation gate we build into our clients' repository CI/CD pipelines. It runs alongside standard unit tests on every pull request.
import pytest
from pydantic import BaseModel, Field
from typing import List
from eval_framework import evaluate_groundedness, calculate_confidence_interval
class SynthesizedAnalysis(BaseModel):
key_findings: List[str] = Field(min_length=1)
risk_score: float = Field(ge=0.0, le=1.0)
cited_chunk_ids: List[str] = Field(min_length=1)
# Golden benchmark suite: representative failure scenarios
BENCHMARK_FIXTURES = [
{
"query": "What is the liability cap under Section 14.2 for service outages?",
"context_chunks": ["chunk_legal_sec14_v2", "chunk_legal_sla_2026"],
"expected_risk_floor": 0.45,
},
{
"query": "Does the enterprise SLA cover force majeure network events?",
"context_chunks": ["chunk_legal_sla_force_majeure"],
"expected_risk_floor": 0.80,
}
]
@pytest.mark.eval
@pytest.mark.parametrize("scenario", BENCHMARK_FIXTURES)
def test_production_agent_reliability_gate(scenario, ai_client):
"""
CI gate enforcing deterministic validation, structured schema compliance,
and a statistical groundedness threshold of >= 0.92.
"""
response = ai_client.run_agent_pipeline(
query=scenario["query"],
temperature=0.0 # Eliminate non-essential temperature jitter in CI
)
# 1. Deterministic Structural Gate
parsed = SynthesizedAnalysis.model_validate_json(response.raw_text)
assert parsed.risk_score >= scenario["expected_risk_floor"], (
f"Underestimated legal risk score: {parsed.risk_score} < {scenario['expected_risk_floor']}"
)
# 2. Strict Citation Boundary Gate
invalid_citations = set(parsed.cited_chunk_ids) - set(scenario["context_chunks"])
assert not invalid_citations, f"Hallucinated citations detected: {invalid_citations}"
# 3. Groundedness Evaluation
groundedness_score = evaluate_groundedness(
answer=parsed.key_findings,
reference_chunks=scenario["context_chunks"]
)
assert groundedness_score >= 0.92, (
f"Groundedness score dropped below threshold: {groundedness_score:.3f} < 0.920"
)
3. Controlling LLM-as-a-Judge Drift
When using an evaluator model (such as Gemini 1.5 Pro or Claude 3.5 Sonnet) to grade responses, the judge itself can suffer from cognitive bias:
- Verbosity Bias: The tendency to rate longer, padded answers higher than concise, accurate ones.
- Self-Enhancement Bias: A model preferring outputs generated by its own model family.
- Positional Bias: Favoring the first response when comparing two candidates.
To neutralize these biases, your evaluation harness must enforce:
┌────────────────────────────────────────────────────────┐
│ CALIBRATING THE EVAL JUDGE │
├────────────────────────────────────────────────────────┤
│ 1. Zero Temperature ──► Deterministic decoding │
│ 2. Few-Shot Rubric ──► Concrete 1-to-5 anchors │
│ 3. Binary Questions Only ──► Yes/No factual claims │
│ 4. Pairwise Position Swap ──► Eliminate order bias │
└────────────────────────────────────────────────────────┘
Rather than asking the judge "Rate the clarity of this answer from 1 to 5", break the output into discrete atomic propositions:
- "Does claim 1 assert that the SLA guarantees 99.99% uptime? (YES/NO)"
- "Is this assertion supported verbatim by chunk_04? (YES/NO)"
Aggregating binary assertions produces a reproducible percentage metric with known confidence intervals, transforming subjective vibes into reproducible engineering telemetry.
4. The Pull Request Gate: How to Block Regressions
In a healthy AI engineering workflow, a prompt engineer or developer cannot merge code to main without running the evaluation suite against a curated Golden Dataset.
# Example GitHub Actions step in .github/workflows/evals.yml
- name: Run RightQA Regression Benchmarks
run: |
poetry run pytest tests/evals/ -m eval \
--benchmark-dataset=data/golden_eval_suite_v3.json \
--min-accuracy=0.94 \
--max-p95-latency=2200 \
--fail-on-regression
If a prompt tweak boosts performance on new edge cases but drops overall benchmark accuracy from 96.2% to 91.8%, the CI runner fails with a clear diff. The developer sees exactly which scenarios broke, what the previous output was, and how the new response deviated.
Summary Checklist for Engineering Teams
Before deploying your next agent or LLM workflow to users, ask:
- Do you have a versioned dataset of at least 50 representative, adversarial user queries?
- Can you run your entire eval suite locally with a single terminal command (
pytest tests/evals)? - Does your CI system automatically block pull requests that degrade accuracy or increase hallucination rates?
- Are you measuring groundedness, context precision, and latency independently?
If the answer to any of these is no, your system is running on vibes.
At RightQA, we help engineering teams design resilient architectures and build automated evaluation suites that catch failures before production. Discuss your system with our principal engineers.