case studies · ai and agents

AI and agents, move by move.

Pilots that worked in a demo and stalled on security review. These are the AI rollouts we carried into production, with scoped identities for agents, evaluation before launch and a human on anything that can't be undone.

8 case studies7 AI rollouts1 upgrade18 platforms
featured case study

AI pilot to production with forward deployed engineers

Told without the client's name or anything that could identify them.

what happened

An insurer's claims operations team had an AI pilot that impressed leadership, then sat in a sandbox running on a personal API key and a copy of production data. Nobody could say how often it was right, and it was built on an agent service that was being retired. Our forward deployed engineers sat with the claims team, agreed on one workflow and one success measure, and built an eval set from real cases before touching the prompt. We replaced copied data with live API and MCP connections under the workflow's own identity, added human review for costly decisions, and set up tracing and spend limits. The business owner and an internal engineering team now run it, with the eval suite running in CI.

what the client got
  • The workflow is scored on real casesWe built the eval set from real inputs and expected outputs, hard and edge cases included, and it runs in CI. Every prompt change or model upgrade gets scored against that same set.
  • No more personal keys or copied dataThe workflow has its own identity, uses user-delegated tokens where an action belongs to a person, and reaches live systems through APIs or MCP servers with narrow scopes.
  • People review the decisions that cost the mostOutputs are split into straight-through and human-reviewed by cost of error. Reviewer corrections feed back into the eval set.
Read the full case study
after / ai-pilot-to-production delivered
beforeAI pilot or proof of concept afterProduction AI workflow with evals and an owner
01 · BUSINESS SYSTEMS 02 · AI WORKFLOW 03 · CONTROL System of record: claims · cases · tickets System of record claims · cases ·tickets MCP servers: scoped tools MCP servers scoped tools Production workflow: own identity · narrow scopes Production workflow own identity · narrowscopes Model platform: Bedrock · Foundry · Vertex AI Model platform Bedrock · Foundry ·Vertex AI Human review queue: costly or irreversible actions Human review queue costly or irreversibleactions Eval suite in CI: real cases · edge cases Eval suite in CI real cases · edgecases Tracing and cost limits: retrieval · model · tool calls Tracing and costlimits retrieval · model ·tool calls
every case study

All 8 in ai and agents

Read any of them and you get the story, the before-and-after architecture, the plan phase by phase and the results.

AI rollout AI and agents

AI pilot to production with forward deployed engineers

Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.

read the case study
AI rollout AI and agents

Classic RPA bots to agentic automation

Moving classic RPA bots to agentic automation in UiPath or Power Automate. APIs first, agents only where judgment is needed, and controls on both.

read the case study
AI rollout AI and agents

Enterprise search and RAG that respects document permissions

Enterprise search and RAG that only return what each user may open. We capture source ACLs at ingestion, filter at query time and leak-test.

read the case study
AI rollout AI and agents

Identity, scopes and audit for AI agents and MCP servers

Giving AI agents their own identities, putting remote MCP servers behind OAuth with per-tool scopes, and logging which agent did what for whom.

read the case study
AI rollout AI and agents

MCP servers for SAP, Salesforce, ServiceNow and internal APIs

We connect AI agents to SAP, Salesforce, ServiceNow and internal APIs through MCP servers, with OAuth, scoped tools, approval for writes and audit.

read the case study
AI rollout AI and agents

Microsoft 365 Copilot or Gemini Enterprise rollout with real adoption

Rolling out Microsoft 365 Copilot or Gemini Enterprise with oversharing fixed first, use cases tied to real tasks and adoption measured past login counts.

read the case study
upgrade AI and agents

OpenAI Assistants API to the Responses API and the Agents SDK

Getting apps off the shut-down OpenAI Assistants API onto the Responses and Conversations APIs or the Agents SDK, with history kept and outputs re-tested.

read the case study
AI rollout AI and agents

Turning on Agentforce, Now Assist, Joule and Copilot safely

Turning on vendor AI in Salesforce, ServiceNow, SAP and Dynamics 365, with permissions checked, an owner for each use case, agents tested and costs watched.

read the case study
what goes wrong

Risks that keep coming up

One risk from each of these moves, and the case study that shows how we handled it when it hit.

  1. Demo data versus real data ↗

    Pilots tend to run on clean samples. Real inputs are messier, so quality usually drops at first. Expect that, and measure it.

  2. Agents where rules would do ↗

    An agent adds variability. Steps that must give the same answer every time, such as posting a payment, stay deterministic.

  3. Stale permissions ↗

    When ACLs refresh only nightly, someone whose access was revoked can keep retrieving content for hours. How long that window is allowed to be is a decision the sync design has to make.

  4. Token passthrough ↗

    If an MCP server forwards the client's token to a downstream API, audience checks stop meaning anything and one server can act as another. The spec forbids this. We test for it anyway.

  5. Tools that do too much ↗

    Give an agent a tool that runs any SOQL query or calls any OData service, and you've given it the same reach as an admin. OWASP calls this excessive agency.

  6. Oversharing becomes discoverable ↗

    A bad permission on an old site used to be hard to stumble on. Once an assistant can search everything a user can open, it isn't.

01UiPath or Power Automate?

That depends on what you run now, your Microsoft licensing and how much attended or desktop automation you have. Teams with a big UiPath estate usually extend it. Microsoft-heavy shops with simpler flows often consolidate on Power Automate and Copilot Studio.

02Should we build our own RAG or use Copilot or a platform search?

If your content lives mostly in Microsoft 365 or one data platform, its built-in search usually enforces permissions with less work. Custom RAG makes sense when sources span systems those tools don't cover, or when answers must follow your own rules.

03Entra Agent ID or Okta for agents?

It mostly follows your existing identity setup. Microsoft 365 and Copilot Studio agents belong in Entra Agent ID. Mixed SaaS estates on Okta tend to suit Agent SSO and Cross App Access. Running both is fine if the boundary is clear.

04Can agents write to SAP or Salesforce?

Yes, with scoped tools. We start read-only and add writes one at a time. Anything with financial or customer impact needs approval until we know the error rate.

05Copilot or Gemini Enterprise?

Usually the one that matches your productivity suite. Copilot fits a Microsoft 365 tenant and Gemini Enterprise fits Google Workspace, and some companies use both. The rollout work doesn't change. It's oversharing, task design and measurement either way.

other areas

More case studies by area

The same approach, applied to other parts of the estate.

sources 8 sources linked last checked 2026-10-20 content fingerprint 9e437cba
  1. 01Palantir careers: Forward Deployed Software Engineerjobs.lever.co ↗
  2. 02OpenAI careers: Forward Deployed Engineerjobs.ashbyhq.com ↗
  3. 03Anthropic careers: Forward Deployed Engineerjob-boards.greenhouse.io ↗
  4. 04AWS What's New: AgentCore generally available (13 Oct 2025)aws.amazon.com ↗
  5. 05AWS What's New: AgentCore Evaluations GA (31 Mar 2026)aws.amazon.com ↗
  6. 06Amazon Bedrock User Guide: document history (Agents Classic)docs.aws.amazon.com ↗
  7. 07Microsoft Learn: Foundry general availability and feature statuslearn.microsoft.com ↗
  8. 08OpenAI API deprecationsdevelopers.openai.com ↗
Reviewed by the Cohash team Product names are trademarks of their owners. how we handle facts and data →
next step

Facing one of these moves?

Tell us what you're on now and where you need to be. We'll come back with how we'd run the move.

useful to bring
  • The platform and the version you're on
  • Where you need to get to, and by when
  • What's custom: rules, plugins, integrations
  • Any audit finding or deadline driving it