Data Engineer (SC Cleared, Databricks)

Senior Data Engineer (Azure Databricks) - SC ClearedLocation: London, UK (3 days per week onsite)Salary: Up to £75,000 per annumJob Type: Permanent, Full-TimeSecurity Clearance: Active SC Clearance RequiredThe OpportunityWe are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities.This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices. You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights.Key ResponsibilitiesData Pipeline Development andamp; OptimisationDesign, develop, and maintain scalable and reliable data pipelines.Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers.Optimise pipeline performance, scalability, and cost efficiency.Implement data quality controls, validation processes, and monitoring solutions.Data Transformation andamp; ProcessingDevelop complex data transformation logic using Spark (PySpark or Scala).Build data cleansing, enrichment, aggregation, and processing solutions.Ensure data accuracy, consistency, and reliability throughout the data life cycle.Azure Databricks DevelopmentDevelop and support ..... full job details .....