Sr. Databricks Platform Engineer

The Role

Were engaging a hands-on / Senior Databricks platform engineer/builder for our client - one of global leading strategy-led tech consultancy company. The objective of role is to lead the design and operationalisation of a governed multi-tenant data platform on Azure Databricks - ingesting and onboarding data from multi external organisations across Databricks Snowflake API and file-based sources.

Design and implement governance from the ground up and make Unity Catalog and Delta Sharing work at scale in production contributing to the low level design of the architecture already selected.

What Youll be doing

• Unity Catalog governance design and implement the catalog/schema structure privilege model data ownership and lineage for a platform serving multiple business units and external partners

• Delta Sharing architect the provider/recipient model for external data onboarding; manage refresh strategy access scoping and cross-platform sharing (Databricks-to-Databricks Databricks-to-Snowflake)

• Identity & access integration Microsoft Entra ID integration with Unity Catalog; service principal design; RBAC across environment tiers

• Pipeline modernisation refactor existing Python pipelines into a production-grade Databricks medallion architecture (bronze/silver/gold)

• Platform engineering workspace structure environment promotion CI/CD cost optimisation and capacity planning

• Architecture contribution this is a live engagement with an existing recommendation in place; we want someone who will challenge and improve it not just implement it

What you should have:

Must-Have

• Hands-on Unity Catalog implementation in a production environment not workshop-level exposure. You should be able to talk through catalog structure privilege model and lineage from real deployments.
• Real Delta Sharing experience provider/recipient architecture external data onboarding operational challenges youve actually hit and solved
• Strong Python and PySpark with production pipeline migration experience (not just greenfield builds)
• Microsoft Entra ID integration experience service principals RBAC scoped access
• Track record of making platform architecture decisions yourself not just executing someone elses design
• 7 years data platform engineering 3 years hands-on Azure Databricks

Nice to Have

Databricks Genie evaluation/implementation experience
Azure Data Factory orchestration
CI/CD and DataOps (Azure DevOps Databricks Asset Bundles)
FinOps / platform cost optimisation
Regulated industry or enterprise governance experience

If youre an experienced builder in applying Unity Catalog & Delta Sharing to external Data integration and looking to work alongside a top-tier global consultancy on a high visibility platform build / product-grade delivery wed like to hear from you.