AI pilot to production with forward deployed engineers
Cohash takes AI pilots into production with forward deployed engineers who sit with the business team that will actually use the thing. They narrow the pilot to one workflow you can measure and build an eval set from real cases. The workflow gets connected to live systems through scoped identities, with human review wherever a mistake would be expensive, plus tracing and cost limits. Then it goes to a named owner, along with runbooks and tests that keep running after we leave.
- 01One workflow, one measure
- 02Eval set from real cases
- 03Real systems, scoped identities
- 04Human review where it counts
- 05Tracing, cost and model changes
- 06Handover
The story
Told without the client's name or anything that could identify them.
An insurer's claims operations team had an AI pilot that impressed leadership, then sat in a sandbox running on a personal API key and a copy of production data. Nobody could say how often it was right, and it was built on an agent service that was being retired. Our forward deployed engineers sat with the claims team, agreed on one workflow and one success measure, and built an eval set from real cases before touching the prompt. We replaced copied data with live API and MCP connections under the workflow's own identity, added human review for costly decisions, and set up tracing and spend limits. The business owner and an internal engineering team now run it, with the eval suite running in CI.
why it was urgentAgent platforms have matured and older ones are closing. AWS made Bedrock AgentCore generally available on 13 October 2025 and closed Bedrock Agents Classic to new customers on 30 July 2026. Microsoft retires Foundry Workflows on 1 December 2026, and OpenAI shuts down Agent Builder on 30 November 2026. A pilot built on a retiring service has to move anyway.
- openThe demo impressed leadership months ago. It still runs on someone's laptop, or in a sandbox.
- recurringNobody can say how often the pilot is right, because there's no test set.
- blockedThe pilot uses a personal API key and a copy of production data.
- openIt was built on a service that's now closed to new customers or scheduled to retire.
The estate, before and after
Before: a pilot in a sandbox
The pilot ran on a copy of production data with a personal API key, on an agent service scheduled to retire. There was no eval set, no tracing and no owner outside the people who built the demo.
Read the diagram as text
- Copied production data → Pilot notebook or app: manual export
- Personal API key → Pilot notebook or app: auth
- Pilot notebook or app → Retiring agent service: agent calls
- Pilot notebook or app → Leadership demo: demo output
After: one workflow in production with evals and an owner
The workflow reads live systems through APIs and MCP servers under its own scoped identity, routes costly decisions to a reviewer and traces every request. The eval suite runs in CI so prompt and model changes are scored before release.
Read the diagram as text
- System of record → Production workflow: API reads
- MCP servers → Production workflow: tool calls
- Production workflow → Model platform: prompt · response
- Production workflow → Human review queue: flagged outputs
- Human review queue → Eval suite in CI: reviewer feedback
- Eval suite in CI → Production workflow: release gate
- Production workflow → Tracing and cost limits: traces · spend
- Human review queue → System of record: approved update
How we run it, phase by phase.
6 phases take you from AI pilot or proof of concept to Production AI workflow with evals and an owner. Each one ends with something you can check before the next starts. As you scroll, the blocks show the estate moving across.
- phase 01
One workflow, one measure
Work with the business owner to pick the workflow the pilot should change and how success is measured: cases closed, time per review, error rate. Features that don't serve it wait.
- phase 02
Eval set from real cases
Collect real inputs with expected outputs, including hard and edge cases, and score the pilot against them before changing anything. Every later change runs against the same set.
- phase 03
Real systems, scoped identities
Replace copied data and shared keys with live connections through APIs or MCP servers. The workflow gets its own identity, user-delegated tokens where actions belong to a person, and the narrowest scopes that work.
- phase 04
Human review where it counts
We decide which outputs go straight through and which need a person, and the deciding factor is cost of error. Whatever reviewers flag goes back into the eval set.
- phase 05
Tracing, cost and model changes
Every request gets traced, from retrieval to model calls to tool use. Put spend limits in. If you plan for model retirements early, a forced model change becomes one more eval run.
- phase 06
Handover
Then the business owner and an engineering team take it over. They get runbooks and dashboards, the eval suite running in CI, and a written path for making changes. At that point our forward deployed engineers step back.
What the client got
What changed for the client once the work was done. Described, not scored: we don't publish client figures.
- result 01
The workflow is scored on real cases
We built the eval set from real inputs and expected outputs, hard and edge cases included, and it runs in CI. Every prompt change or model upgrade gets scored against that same set.
- result 02
No more personal keys or copied data
The workflow has its own identity, uses user-delegated tokens where an action belongs to a person, and reaches live systems through APIs or MCP servers with narrow scopes.
- result 03
People review the decisions that cost the most
Outputs are split into straight-through and human-reviewed by cost of error. Reviewer corrections feed back into the eval set.
- result 04
A model retirement is just another eval run
Requests are traced through retrieval, model calls and tool use, spend limits are set, and a model retirement plan means a forced change gets scored before it ships.
- result 05
A named owner runs it
The business owner and an engineering team signed off on handover with runbooks, dashboards and a documented change path.
Risks we plan for
-
Demo data versus real data
Pilots tend to run on clean samples. Real inputs are messier, so quality usually drops at first. Expect that, and measure it.
-
No baseline
Without an eval set you're guessing. Nobody can say whether the last prompt change or model upgrade made anything better or worse.
-
Forced model changes
Models get retired on a schedule. Gemini 2.5 models go on 20 October 2026, Google says. Without evals, each of those dates turns into a fire drill.
-
No owner after launch
If the engineers who built it leave and nobody in the business owns it, the workflow decays. Ownership is agreed before production, not after.
Where AI helps, and where it needs guardrails
Forward deployed engineers were Palantir's model first. Now OpenAI and Anthropic hire them as well, and send them to put models into production with customers. The cloud agent platforms have added evaluation and policy controls on their side, and AgentCore Evaluations has been generally available since 31 March 2026. We give each production agent its own identity and scoped access to MCP servers. A person approves anything irreversible.
forward deployed engineers →What you're left with
We write these for your setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
- document01
Workflow definition with a success measure agreed by the business owner
yours to keep - document02
Eval set from real cases, wired into CI
yours to keep - document03
Production integration with scoped identities and approval steps
yours to keep - document04
Tracing, cost limits and alerting
yours to keep - runbook05
Runbooks, model retirement plan and handover sign-off
yours to keep
Platforms and services on this move
Certified engineers, wherever you need them.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Onsite
Engineers at your sites in the US
Nearshore
Overlapping hours with US teams
Offshore
Overnight progress on your backlog
All time zones
Coverage handed over across time zones
01What does a forward deployed engineer do that a regular consultant doesn't?
They sit with the business team and work on its real data and systems. They ship working software instead of a deck of recommendations, and until handover, whether the workflow actually works is on them.
02How long does it take to get a pilot into production?
It depends on how many systems it touches, how clean the data is and how much human review the risk needs. A first eval run shows the gap between pilot and production quality, which turns into a real estimate.
03Should we rebuild on a different model or platform?
Only when there's a reason to, like a retiring service, missing controls or cost. With an eval set you can compare the options on your own cases instead of on benchmarks.
04Who owns it after you leave?
You'll need a named business owner and an engineering team. We settle that before production, and at handover they get the runbooks, dashboards and eval suite.
Often on the same roadmap
Apache Airflow 2 to Airflow 3
Upgrading self-managed Apache Airflow 2, now past end of life, to Airflow 3, and fixing removed features, auth and scheduling changes before DAGs move.
read the case studyMicrosoft 365 Copilot or Gemini Enterprise rollout with real adoption
Rolling out Microsoft 365 Copilot or Gemini Enterprise with oversharing fixed first, use cases tied to real tasks and adoption measured past login counts.
read the case studyEnterprise search and RAG that respects document permissions
Enterprise search and RAG that only return what each user may open. We capture source ACLs at ingestion, filter at query time and leak-test.
read the case study420b2221 - 01Palantir careers: Forward Deployed Software Engineerjobs.lever.co ↗
- 02OpenAI careers: Forward Deployed Engineerjobs.ashbyhq.com ↗
- 03Anthropic careers: Forward Deployed Engineerjob-boards.greenhouse.io ↗
- 04AWS What's New: AgentCore generally available (13 Oct 2025)aws.amazon.com ↗
- 05AWS What's New: AgentCore Evaluations GA (31 Mar 2026)aws.amazon.com ↗
- 06Amazon Bedrock User Guide: document history (Agents Classic)docs.aws.amazon.com ↗
- 07Microsoft Learn: Foundry general availability and feature statuslearn.microsoft.com ↗
- 08OpenAI API deprecationsdevelopers.openai.com ↗
- 09Google Cloud: Model versions and lifecycledocs.cloud.google.com ↗
Planning this move?
A 30-minute working session with a senior engineer. No slides.
- The platform and the version you're on
- Where you need to get to, and by when
- What's custom: rules, plugins, integrations
- Any audit finding or deadline driving it