Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / ArchitectHands-on Platform Engineering Serverless FinOps POSIT/RStudio Migration 6 Month contract Inside IR35 - 500 a day London/Hybrid Role Purpose We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level. The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio. Key Responsibilities1. Databricks Serverless Enablement and Optimisation Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost. Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines. Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads. Implement compute policies, autoscaling, quotas, budget controls and operational guardrails. Measure cost and performance outcomes, identify idle or oversized compute, and recommend ..... full job details .....