One Databricks workspace that data, ML and security teams share without fighting over it.
Your tables now have agents and models sitting beside them, and somebody has to govern those. We build Databricks lakehouses, move workloads onto Unity Catalog, and set up Lakeflow pipelines and model serving.
External lineage and geospatial types (16 Jun 2026)
In Public Preview
Domains, external access to managed Delta tables
release watch
Unity Catalog at Data + AI Summit 202616 Jun 2026
Agent Bricks at Data + AI Summit 202616 Jun 2026
connects with
in short·15-second read
We implement and run Databricks lakehouses.
That means designing workspaces and Unity Catalog, migrating the Hive metastore and legacy Spark jobs, building Lakeflow pipelines and Delta or Iceberg tables, and setting up model serving and Agent Bricks. Since Data + AI Summit 2026 we also help teams govern models, MCP servers and AI spend in Unity Catalog, using features like external lineage, which is now generally available.
you probably need this if
Tables, jobs and mount points still depend on the legacy Hive metastore.
Notebooks run on a schedule in production and there are no tests and no deployment process behind any of them.
All-purpose clusters stay up overnight and nobody owns the compute policies.
16 Jun 2026Unity Catalog at Data + AI Summit 2026 External lineage, and geospatial types in Delta and Iceberg v3, are generally available. Domains and external access to managed Delta tables are in Public Preview. Contextual Service Policies are in Beta. source ↗
16 Jun 2026Agent Bricks at Data + AI Summit 2026 Document Intelligence generally available, with the ai_parse_document, ai_extract and ai_classify SQL functions. Agent memory on Lakebase announced. source ↗
16 Jun 2026Unity AI Gateway Governance for models, agents and MCP servers, with hard spend caps and costs attributed by user and by team. Guardrail policies are still in Beta. source ↗
today · Oct 22, 2026
the story · databricks estate
From the first call to a setup your own team can run.
Here's what usually prompts a call about your Databricks estate, what we do about it and in which order, and where you end up.
01 · where it hurts02 · what we change03 · how it runs04 · where you end up
incident board · databricks estate3 open
openTables, jobs and mount points still depend on the legacy Hive metastore.
recurringNotebooks run on a schedule in production and there are no tests and no deployment process behind any of them.
blockedAll-purpose clusters stay up overnight and nobody owns the compute policies.
exposure
Unity Catalog design and migrationLakeflow pipelinesPlatform setup and cost controlAI governance in Unity CatalogAgents and model servingInteroperability with Snowflake…
startrehearsehandover
Workspace and metastore…
Unity Catalog design
Pipelines as code
AI governance
Handover
we are here
ready to hand over
Tables all have owners and their grants sit in Unity Catalog.
Finance can see what support's agents cost last month.
Hive metastore inventory of tables, mounts, jobs and owners, sorted into waves
Unity Catalog design with the catalog, schema and grant model per domain
chapter 01 · where it hurts
Why teams reach out
Tables, jobs and mount points still depend on the legacy Hive metastore.
Notebooks run on a schedule in production and there are no tests and no deployment process behind any of them.
All-purpose clusters stay up overnight and nobody owns the compute policies.
chapter 02 · what we change
How we solve it
Unity Catalog design and migration
Lakeflow pipelines
Platform setup and cost control
AI governance in Unity Catalog
Agents and model serving
Interoperability with Snowflake and others
chapter 03 · how it runs
Our approach
Workspace and metastore inventory
Unity Catalog design
Pipelines as code
AI governance
Handover
chapter 04 · where you end up
What changes
Tables all have owners and their grants sit in Unity Catalog.
Finance can see what support's agents cost last month.
Hive metastore inventory of tables, mounts, jobs and owners, sorted into waves
Unity Catalog design with the catalog, schema and grant model per domain
what we do
Our Databricks work
6 kinds of Databricks work, each run by engineers who have done it on live tenants.
We lay out the metastore, catalogs and schemas, then set grants and attribute-based policies. Tables and jobs still on the legacy Hive metastore get migrated across.
Ingestion with Lakeflow Connect and declarative pipelines for transformation, with expectations, retries and alerts defined per table.
Workspaces, compute policies, serverless settings, budgets and system tables for cost reporting, deployed with Terraform or Databricks Asset Bundles.
Foundation model access, MCP servers and agent tools registered and permissioned in Unity Catalog, with AI Gateway limits and tracing turned on.
Agent Bricks and model serving endpoints built on governed data, evaluated against real questions and monitored after launch.
Delta Sharing, Iceberg and external access to managed tables set up so other engines read the same data without extra copies.
capability 01 / 06in production
Unity Catalog design and migration
We lay out the metastore, catalogs and schemas, then set grants and attribute-based policies. Tables and jobs still on the legacy Hive metastore get migrated across.
We pieced these together from Databricks situations that come up a lot. They're patterns. You won't find a specific client in them.
case file 01
A Hive metastore with 3,000 tables and DBFS mounts nobody owns
composite · not a client story
Beforethe situation
Two workspaces still use the legacy Hive metastore, with about 3,000 tables and dozens of DBFS mounts whose storage keys are pasted into notebooks. Jobs run on all-purpose clusters, and lineage stops at the workspace edge.
What we dothe work
Inventory first: tables, mounts, jobs and owners, from system tables and the workspace APIs. Mounts give way to external locations and storage credentials. We move tables to Unity Catalog domain by domain, in waves, and switch each wave's jobs to job compute with three-level names.
Afterwhere you end up
Tables all have owners and their grants sit in Unity Catalog. No notebook has a storage key in it any more, and lineage follows the jobs feeding downstream reports past the old workspace edge.
case file 02
Six teams calling foundation models with no record of who spends what
composite · not a client story
Beforethe situation
Data science, support and finance each built their own agents on model serving endpoints and external models. Some months the bill jumps and nobody can explain it, and two teams call MCP servers nobody registered.
What we dothe work
We register models, serving endpoints and MCP servers in Unity Catalog. Traffic goes through Unity AI Gateway with rate limits and hard spend caps per team, and we turn tracing on. Guardrail policies stay in development while they're in Beta.
Afterwhere you end up
Finance can see what support's agents cost last month. A runaway job hits its team's cap before it reaches the invoice. And the platform team has a list of which MCP servers agents can reach.
We don't start a step until the last one has produced something you can check.
step 01
Workspace and metastore inventory
First a list of workspaces, clusters, jobs, tables, mounts and grants. Then system tables, to see where compute spend is going.
step 02
Unity Catalog design
We lay out catalogs, schemas, grants and attribute-based policies by domain and environment. Then Hive metastore tables and jobs migrate in waves.
step 03
Pipelines as code
We move Lakeflow pipelines and jobs into Databricks Asset Bundles or Terraform and add expectations and CI. Dev, test and prod each get their own target.
step 04
AI governance
We register every model, serving endpoint and MCP server in Unity Catalog, put AI Gateway rate limits and spend caps on each team, and turn tracing on.
step 05
Handover
Your platform team keeps the compute policies, budgets, deployment templates and a runbook for onboarding new teams.
ai on databricks
Where AI fits on Databricks, and where it needs guardrails.
Databricks keeps AI next to the data. Agent Bricks builds agents, there's model serving, and there are AI functions such as ai_parse_document. Unity Catalog governs the models, agents and MCP servers. Unity AI Gateway adds hard spend caps and cost attribution. A lot of the AI governance is new and some is still Beta, so we check status feature by feature.
AI agentasks for data or an action
Scoped identityonly the access it needs
MCP serverthe one door into the platform
Logged, approveda person signs off on writes
Databricksdoes the work
Databricks publishes managed MCP servers such as Genie One and Databricks SQL, and teams can register external or custom ones. Access follows Unity Catalog permissions. Unity AI Gateway controls credentials and monitors use. We run agents as service principals with narrow grants, trace their calls and cap spend per team.
Agent Bricks Document Intelligence
Document Intelligence in Agent Bricks is GA. ai_parse_document, ai_extract and ai_classify are SQL functions, so the extraction runs in a query. We use them on invoices, contracts and claims, and nothing feeds a downstream table until it scores well against our labelled samples.
Unity AI Gateway
AI Gateway governs traffic to models, agents and MCP servers, with hard spend caps and cost attribution by user and team. Guardrail policies are in Beta. We set rate limits and caps per team, turn on tracing and keep Beta guardrails out of regulated workloads.
Managed MCP servers
Databricks provides managed MCP servers, Genie One for business questions and Databricks SQL for queries among them. It also supports external servers and custom ones built on Databricks Apps. Unity Catalog permissions apply to all of them. We keep a register of every server an agent can reach, and anything not on it doesn't run.
Agent memory on Lakebase
Databricks announced agent memory on Lakebase at Data + AI Summit 2026. Announced isn't shipped. We'd prototype it in development and wait for general availability before any agent depends on it.
How data moves through Lakeflow into a Unity Catalog lakehouse
Data from databases, SaaS apps and streams arrives through Lakeflow Connect and declarative pipelines and lands in Delta or Iceberg tables. Those tables are governed by Unity Catalog, whether a SQL user, an agent or another engine such as Snowflake is reading them.
These are working documents, written against your configuration, so your team can run them without us.
inventory01
Hive metastore inventory of tables, mounts, jobs and owners, sorted into waves
yours to keep
inventory02
Unity Catalog design with the catalog, schema and grant model per domain
yours to keep
tests03
Asset Bundle templates with dev, test and prod targets and CI
yours to keep
report04
Compute policies and budgets per team, with system-table cost reports
yours to keep
document05
AI Gateway configuration with rate limits and spend caps per team
yours to keep
talent and delivery · databricks
Certified engineers, wherever you need them.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Put these to anyone bidding on your Databricks work. The ones who've done it answer differently from the ones who've read about it.
Q1
How will you find every DBFS mount and hard-coded storage key before we migrate?
When tables move to Unity Catalog, it's the mounts and the keys pasted into notebooks that break. Security usually flags those same keys, so the inventory does double duty.
Q2
Which of the Data + AI Summit 2026 features you're proposing are actually GA, and which are Preview or Beta?
Only some of it is GA. External lineage is, Domains is Public Preview, and Contextual Service Policies is Beta. Don't put regulated workloads on anything whose behavior can still change.
Q3
How will you attribute compute and model spend to teams?
System tables record usage, but attribution only works when tags, compute policies and AI Gateway settings are consistent. Without that, cost reviews end in arguments about whose cluster it was.
Q4
What goes into a Databricks Asset Bundle, and what stays in Terraform?
Workspaces, networking and metastores usually live in Terraform, while jobs and pipelines live in bundles. A fuzzy boundary means two tools fight over the same resources.
Q5
How will Snowflake or another engine read our tables without a second copy?
Delta Sharing, Iceberg reads and external access to managed tables aren't equally mature or equally governed. Whichever you pick decides who controls access and where lineage gets recorded.
01Do we have to migrate to Unity Catalog?
New governance, lineage and AI features from Databricks are built on Unity Catalog. Stay on the Hive metastore and you miss them. The effort comes down to how many tables, jobs and mount points you have, so we inventory those and then migrate in waves.
02What drives Databricks cost?
Compute type and runtime, mainly. All-purpose clusters left running, oversized job clusters and unbounded model serving are common causes. System tables show where spend goes, and compute policies and budgets keep it in check.
03Which Data + AI Summit 2026 features are ready for production?
Look up each feature's status. External lineage and geospatial types are GA according to Databricks, and many governance features aren't. Beta is fine in development. Regulated work waits for GA.
04Should we pick Databricks or Snowflake?
Depends on the workload mix and the skills you have. Databricks is a good fit for heavy Spark, streaming and ML engineering. Snowflake fits analytics teams that work SQL-first. A lot of organizations run both and share data through open table formats.
05Can you set up Databricks with Terraform from day one?
Yes. We usually put accounts, workspaces, networking and Unity Catalog metastores in Terraform, and jobs and pipelines in Databricks Asset Bundles. That split keeps the platform team and data teams from editing the same resources.
services
Related services
Projects on Databricks usually end up touching one of these, so they're worth a look.
