Governed Self-Serve Onboarding on Databricks: A Lakeflow + Unity Catalog Case Study

How a reusable Lakeflow custom connector, a bronze/silver/gold Declarative Pipeline, and Unity Catalog grants turn team onboarding from a ticket queue into a repeatable, governed template — deployed and verified on a live Databricks workspace.

Quick answer: This case study shows a governed, clonable Databricks onboarding template — a reusable Lakeflow custom connector, a bronze/silver/gold Declarative Pipeline, and Unity Catalog access-tier enforcement — deployed and verified live against a real workspace, cutting new-team onboarding from a manual, multi-day ticket process down to five commands with no new code required. Full source: rehman04/lakeflow-onboarding-template.

The Challenge

Manually onboarding each new analytics team onto a shared Databricks workspace creates governance drift: every team gets a slightly different catalog setup, a slightly different grant scope, and a slightly different cluster policy, because a human re-does the work from scratch each time under time pressure.

Three specific problems compound this:

• Inconsistent access scope. Under deadline pressure, engineers grant access at the schema level instead of the table level, because narrowing it properly takes longer.
• No enforcement mechanism. Guardrails documented in a wiki page are guardrails nobody is forced to follow — access rules exist as intentions, not as code.
• No repeatable ingestion path. Each team's ingestion pipeline gets hand-built, so there is no single connector to maintain, patch, or audit — just N slightly different ones.

The workspace in question added two more constraints on top of this: it had no default managed storage root in its Unity Catalog metastore, so brand-new catalogs could not be created per team, and it required serverless compute for all Declarative Pipelines, so a standard cluster policy could not attach to the pipeline directly. Any fix had to work inside both constraints, not around them.

The Approach

The solution is a single Databricks Asset Bundle that packages a reusable connector, a standardized pipeline, and enforced Unity Catalog grants into one deployable unit per team — so onboarding a new team means changing configuration variables, not writing new code.

Component 1: A reusable ingestion connector

src/ingest_api.py is a Lakeflow custom connector generic REST API reader, built on the Python Data Source API implementation for Lakeflow Connect. It is written once and reused across every source by changing a single url parameter. Credentials are never hardcoded — the connector retrieves them from a Databricks secret scope at runtime.

Component 2: A standardized medallion pipeline

src/pipeline_transforms.py defines a Databricks Declarative Pipeline bronze silver gold template with a fixed naming convention baked in: bronze_<entity> → silver_<entity> → gold_<entity>. Every team's pipeline has the same shape, which is what makes it supportable at scale rather than bespoke per team. (The connector class is duplicated inline in this file due to a Databricks pipeline-library serialization quirk — not a design choice.)

Component 3: Enforced Unity Catalog guardrails

Because the metastore has no default managed storage root, the template provisions one schema per team inside the existing shared main catalog (main.<team>) instead of a new catalog — a working example of managing Databricks schemas without a default managed storage root. scripts/grant_gold_access.py then performs automated table-level SELECT grants for gold layer tables only, once the pipeline has produced that table. Bronze and silver stay closed by default. Because serverless pipelines can't carry a cluster policy directly, policies/onboarding_guardrail.json instead enforces a spend-cap cluster policy on any classic compute a team spins up outside the pipeline — Unity Catalog access-tier enforcement for analytics teams covers the pipeline side of the boundary instead.

Component 4: Multi-team orchestration

databricks.yml ties all of the above into a Databricks Asset Bundle multi-team target configuration example, with team_a and team_b as two live targets. scripts/onboard_team.py handles idempotent service principal creation for each team's Unity Catalog identity — a service principal is used instead of a workspace-local group because this metastore doesn't recognize workspace-local groups as valid grant principals.

Full Architecture

Manual onboarding vs. this template

The Results

Deploying a new team onto this template comes down to five commands, with no new connector or pipeline code written:

The guardrail was verified against the live workspace, not just described in config. After deploying team_a end-to-end, the actual permission dump read:

The team can see their schema and query the finished gold table; bronze and silver — where raw and half-cleaned data lives — stay invisible by default. The pipeline was then run end-to-end against a real source (the Open-Meteo weather API), returning a live SQL query result from gold_weather — proof the connector, transforms, and grant all connect correctly in practice.

Reuse was proven, not assumed. The template was deployed twice, as team_a and team_b, against two different sources and entities, using the identical connector code, pipeline shape, and guardrail logic — differing only in bundle variables. That's the actual test of a template, as opposed to a pipeline that only happens to work once.

Why This Matters for Data Platform Teams

If manual onboarding is currently costing your organization engineering time or governance drift — and for most organizations running Databricks with more than a handful of analytics teams, it usually is both — this pattern generalizes cleanly beyond this one repository:

• A reusable connector plus configuration keeps engineering effort roughly constant as teams multiply, instead of scaling linearly with headcount.
• Guardrails defined as code mean an access audit is a git diff, not a console review.
• Designing around real metastore constraints — shared catalogs, serverless-only compute, identity quirks — is what separates a template that survives a production workspace from one that only works in a demo.

Frequently Asked Questions

What is a Lakeflow custom connector, and when do you need one?

A Lakeflow custom connector is a Python Data Source API implementation that lets Lakeflow Connect ingest from a source without a pre-built connector — most often a REST API. You need one whenever your source isn't already covered by a native Lakeflow connector, and a generic REST reader like the one in this template can usually be reused across sources by changing configuration rather than code.

Why use a schema-per-team model instead of a catalog-per-team model in Unity Catalog?

Schema-per-team is the right pattern when the metastore has no default managed storage root, because new catalogs can't be created freely in that setup. Provisioning main.<team> schemas inside one shared catalog achieves team isolation without needing catalog-creation rights, and keeps the blast radius of any single team's resources contained to its own schema.

How do you enforce a cluster spend cap on a serverless Declarative Pipeline?

You don't attach the policy to the pipeline directly, since serverless pipelines can't carry a classic cluster policy. Instead, the spend-cap policy is applied to any classic compute a team creates outside the pipeline, while the pipeline's own access boundary is enforced separately through Unity Catalog table grants.

Why grant SELECT only on the gold table, and not bronze or silver?

Bronze and silver layers contain raw and partially cleaned data, which usually includes quality issues, duplicate fields, or inconsistent formats that shouldn't reach downstream analysts. Granting SELECT only on gold_<entity>, after the pipeline has produced it, means analysts only ever see finished, business-ready data — least privilege by default.

Why use a service principal instead of a workspace group for team identity?

In some Unity Catalog metastores, workspace-local groups aren't recognized as valid grant principals, which makes a service principal the only reliable way to represent a team's identity for access grants. Creating it idempotently means re-running onboarding for an existing team doesn't fail or create duplicate resources.

How do you prove an onboarding template actually reuses cleanly across teams?

The only real proof is deploying it a second time against a different source and entity, using the same connector code, pipeline shape, and guardrail logic, and changing only configuration variables. In this template, that was done with team_a and team_b as two live Databricks Asset Bundle targets.

Where should API credentials be stored in a Lakeflow ingestion pipeline?

API credentials should live in a Databricks secret scope and be referenced by name inside the connector's read() method — never written directly into databricks.yml, a team config file, or committed to the repository.

Where Cyfradane Fits In

This template is a working reference for the kind of platform engineering we do at Cyfradane: governed Databricks environments where self-serve access for analytics and data science teams doesn't cost you control over who can see what, and where onboarding a new team is a deployment, not a project.

If your organization is managing Unity Catalog access by hand, provisioning clusters and catalogs per team on request, or finding that "temporary" access grants never quite get cleaned up, this is exactly the kind of pattern we help enterprise teams implement — tailored to your metastore's real constraints, not a generic best-practice diagram.

Get in touch with Cyfradane to talk through what a governed, self-serve onboarding pattern would look like on your workspace.

Full source code, architecture diagrams, and deployment instructions are available on GitHub: rehman04/lakeflow-onboarding-template.