Know whether last week's prompt change made the assistant better or worse.
A model version changes, someone edits a system prompt, the retrieval index gets rebuilt, and support tickets start mentioning wrong answers. We set up the evaluation, tracing and release process that tells you what changed and whether to ship it.
Amazon BedrockMicrosoft FoundryGemini Enterprise Agent Platform (Vertex AI)OpenAI API
in short·15-second read
LLMOps is the operational practice of testing, releasing, monitoring and paying for applications built on large language models.
Cohash builds evaluation suites from your real cases, wires tracing with OpenTelemetry GenAI conventions or tools like MLflow and Langfuse, gates releases on eval results, tracks token cost per feature, and tests model upgrades before they reach users.
you probably need this if
Prompts are edited directly in production and nobody keeps the old version.
A provider deprecation notice arrived and nobody knows how the replacement model will behave.
The token bill doubled and nobody can say which feature caused it.
From the first call to a setup your own team can run.
Here's what usually prompts a call about your LLMOps & Model Evaluation, what we do about it and in which order, and where you end up.
01 · where it hurts02 · what we change03 · how it runs04 · where you end up
incident board · llmops & model evaluation3 open
openPrompts are edited directly in production and nobody keeps the old version.
recurringA provider deprecation notice arrived and nobody knows how the replacement model will behave.
blockedThe token bill doubled and nobody can say which feature caused it.
exposure
Evaluation suites from real…Tracing and observabilityRelease gates in CIModel upgrade and migration…Cost and capacity controlsProduction feedback loops
startrehearsehandover
Instrument what's running
Build the first eval set
Gate releases
Operate and hand over
we are here
ready to hand over
They switched on a date of their choosing and have a written comparison of where answers…
Finance can see cost per feature.
Tracing instrumentation for model, retrieval and tool calls
Labeled evaluation set with metrics and pass thresholds
chapter 01 · where it hurts
Why teams reach out
Prompts are edited directly in production and nobody keeps the old version.
A provider deprecation notice arrived and nobody knows how the replacement model will behave.
The token bill doubled and nobody can say which feature caused it.
chapter 02 · what we change
How we solve it
Evaluation suites from real traffic
Tracing and observability
Release gates in CI
Model upgrade and migration testing
Cost and capacity controls
Production feedback loops
chapter 03 · how it runs
Our approach
Instrument what's running
Build the first eval set
Gate releases
Operate and hand over
chapter 04 · where you end up
What changes
They switched on a date of their choosing and have a written comparison of where answers changed.
Finance can see cost per feature.
Tracing instrumentation for model, retrieval and tool calls
Labeled evaluation set with metrics and pass thresholds
what we do
Where we spend the hours
6 areas of work, each one done by an engineer who has run it in production.
Test sets drawn from production questions and expert-labeled answers, covering normal cases, known failures and adversarial inputs. Graded with a mix of exact checks, rubric-based model grading and human review on a sample.
Every model call, retrieval step and tool call gets a span recording prompt version, model, token counts and latency. We follow the OpenTelemetry GenAI semantic conventions where your stack supports them. That keeps traces from being locked to one vendor.
We keep prompts, retrieval settings and model choices in version control. Every change runs the eval suite in the pipeline, and a drop on any tracked metric blocks the merge until someone signs off.
If a provider retires a model or ships a new one, we put both through your eval set. We compare quality, latency and cost and write up where behavior changed, before you switch.
We break token spend down by feature, team and customer and watch rate limits. Caching goes in where it's safe. Simpler requests move to smaller models, but only once eval results show quality holds.
User ratings, escalations and failed tool calls flow back into a review queue. Confirmed failures become new eval cases, so the suite grows with the application.
capability 01 / 06in production
Evaluation suites from real traffic
Test sets drawn from production questions and expert-labeled answers, covering normal cases, known failures and adversarial inputs. Graded with a mix of exact checks, rubric-based model grading and human review on a sample.
Every model call, retrieval step and tool call gets a span recording prompt version, model, token counts and latency. We follow the OpenTelemetry GenAI semantic conventions where your stack supports them. That keeps traces from being locked to one vendor.
We keep prompts, retrieval settings and model choices in version control. Every change runs the eval suite in the pipeline, and a drop on any tracked metric blocks the merge until someone signs off.
If a provider retires a model or ships a new one, we put both through your eval set. We compare quality, latency and cost and write up where behavior changed, before you switch.
We break token spend down by feature, team and customer and watch rate limits. Caching goes in where it's safe. Simpler requests move to smaller models, but only once eval results show quality holds.
User ratings, escalations and failed tool calls flow back into a review queue. Confirmed failures become new eval cases, so the suite grows with the application.
We pieced these together from LLMOps & Model Evaluation situations that come up a lot. They're patterns. You won't find a specific client in them.
case file 01
The policy assistant's model is being retired next quarter
composite · not a client story
Beforethe situation
An internal assistant answers benefit policy questions on a model version the provider retires next quarter. Nobody has built an eval set. So far the testing amounts to a few people trying the replacement in a playground.
What we dothe work
Pull several weeks of real questions from the logs and have HR policy owners label a few hundred. Run both models on them with identical prompts. Someone reviews every answer where the two disagree. Then tune the prompt for the new model and run it again.
Afterwhere you end up
They switched on a date of their choosing and have a written comparison of where answers changed. The eval set stays in CI for the next model change.
case file 02
Token costs rising on support ticket summaries
composite · not a client story
Beforethe situation
Ticket summaries run on a large frontier model on every ticket update, even one-line status changes. Monthly spend has climbed along with ticket volume, and finance wants an explanation.
What we dothe work
First add cost tracing per feature. Then test two changes against a labeled sample of summaries: a smaller model, and a rule that skips trivial updates. Ship whatever mix keeps the eval score inside the agreed threshold.
Afterwhere you end up
Finance can see cost per feature. Status changes no longer get summarized, and the next model swap has to pass the eval first.
How evaluation and tracing wrap an LLM application
Nothing changes in prompts, retrieval or model choice without passing CI, where the eval suite runs. In production, traces of each model and tool call feed dashboards and a review queue, and confirmed failures turn into new eval cases.
01 · Change
Prompt and config repoprompts · retrieval · model → CI pipeline · pull request
Review queueratings · escalations → Eval suite · new cases
Read the diagram as text
Prompt and config repo → CI pipeline: pull request
CI pipeline → Eval suite: run
Graders → Eval suite: scores
Eval suite → LLM application: release gate
LLM application → Model platform: inference
LLM application → Traces: spans
Traces → Dashboards: metrics
Traces → Review queue: flagged runs
Review queue → Eval suite: new cases
how it runs
A typical engagement
We don't start a step until the last one has produced something you can check.
step 01
Instrument what's running
Add tracing to the current application and collect a few weeks of real requests before changing anything.
step 02
Build the first eval set
Pull a sample of real requests. Have subject experts label the answers they'd expect, then agree the metrics and pass thresholds with the product owner.
step 03
Gate releases
We moved prompts and configuration into version control and ran evals in CI on every change, with a dashboard showing results by release.
step 04
Operate and hand over
We hand over cost and quality dashboards, alert thresholds and an upgrade runbook for model changes, and train your team on adding eval cases.
you keep
What you're left with
We write these for your LLMOps & Model Evaluation setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
design01
Tracing instrumentation for model, retrieval and tool calls
yours to keep
document02
Labeled evaluation set with metrics and pass thresholds
yours to keep
tests03
CI pipeline step that runs evals and blocks regressions
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Put these to anyone bidding on your LLMOps & Model Evaluation work. The ones who've done it answer differently from the ones who've read about it.
Q1
Where do our eval cases come from, and who labels them?
Synthetic test sets tend to miss the odd things real users ask. If your subject experts label real traffic, you can believe the score.
Q2
How will you check that a model-based grader agrees with our experts?
A grader nobody has checked can keep passing bad answers. Ask for its agreement rate against human labels before you rely on it.
Q3
Does your tracing tie us to one observability vendor?
The OpenTelemetry GenAI conventions are still marked as in development. Building on them anyway means your traces can come with you when you change tools.
Q4
What blocks a release, and who is allowed to override it?
If the eval suite only produces a report, people stop reading it. Settle the thresholds and the override process before the first release goes out.
Q5
How will token cost be attributed to features and teams?
If you can't see cost per feature, optimization is guesswork and budget conversations get harder than they need to be.
01What's the difference between MLOps and LLMOps?
MLOps grew up around models you train yourself, so it cares about feature pipelines, training runs and drift on numeric inputs. With LLMOps you usually call a hosted model you didn't train. The hard parts become versioning prompts and retrieval, grading free-text answers, tracing tool calls and keeping token cost in check.
02Can a model grade another model's answers?
For many criteria, yes, if the grading prompt is specific and you've checked the grader against human labels on a sample first. Where a wrong answer has regulatory or financial consequences, keep a person reviewing a slice of traffic anyway.
03Which tools do you use?
It depends on your stack. MLflow fits teams already on Databricks. Langfuse is open source and can be self-hosted. Promptfoo works well for CI evals and red-teaming. We'd rather fit what you have than add a new platform.
04How big does an eval set need to be?
Big enough to cover your main request types and known failure modes, with enough cases per type that one flipped answer doesn't swing the score. A few hundred well-chosen cases often beats thousands of random ones. Grow it from production failures.
05How do we prepare for a model being retired?
Keep an eval set and a script that runs it against any model ready before the notice arrives. Then switching becomes a measured test run against the replacement instead of a rushed rewrite.
06Do you also monitor AI agents that call tools?
Yes. Agent traces need spans for each step and each tool call, plus checks on whether the agent chose the right tool with the right arguments. The permission side is on our AI security page.
related
Often paired with
Projects on LLMOps & Model Evaluation usually end up touching one of these, so they're worth a look.
