Senior Data Engineer (contract)

Overview

Senior Data Engineer

— On-site — Full timeMethods Business and Digital Technology Limited is a UK-based IT Services Consultancy with a strong focus on transforming public sector operations and data solutions for central government departments and agencies.Methods was acquired by the Alten Group in early 2022.Responsibilities

We are seeking a seasoned Senior Data Engineer to design, build, and maintain sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The role involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, contributing to organisational growth and innovation.Develop and Manage Data Pipelines: design, construct, and maintain efficient and reliable data pipelines using Python/Go/Azure Data Factory, supporting streaming and batch processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.Hybrid Cloud and Data Storage Solutions: implement and manage data storage solutions leveraging on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.Containerisation and Orchestration: utilise Docker for containerisation and Kubernetes for orchestration to enable scalable deployment across cloud and on-premises environments.Workflow Automation: use tools such as Azure Data Factory to automate data flows and manage complex workflows within hybrid environments.Event Streaming Experience: utilise event-driven technologies such as Kafka and NATS to handle real-time data streams.Security and Compliance: manage security setups and access controls, including tools like Keycloak to protect data integrity and comply with legal standards.Database Development: design and develop PostgreSQL databases with high performance and availability across deployment scenarios.Essential Skills and Experience

Strong Python Skills: expertise in Python for scripting and automating data processes across varied environments.Experience with ETL/ELT: demonstrable experience in developing and optimizing ETL or ELT workflows in hybrid (on-premises and Azure) environments.Expertise in Hybrid Cloud Data Architecture: knowledge of integrating on-premises infrastructure with Azure cloud services.Containerisation and Orchestration Expertise: solid experience with Docker, GitHub and Kubernetes in managing applications across on-premises and cloud platforms.Proficiency in Workflow Automation Tools: practical experience with Azure Data Factory in relevant environments.Experience in Event Streaming: proven ability in managing and deploying event streaming platforms like Kafka and NATS.Data Security Knowledge: experience with implementing security practices and tools, including Keycloak, across multiple platforms.Search and Database Development Skills: strong background in Elasticsearch and PostgreSQL across on-premises and cloud infrastructures.Your Impact

You will empower business leaders to make informed decisions by delivering timely, accurate, and actionable data insights from a robust, hybrid infrastructure. You will champion adoption of modern data architectures, mentor team members, and advance engineering practices to cultivate a data-driven culture within the organisation.Desirable Skills and Experience

Azure and Other Relevant Certifications: cloud and on-premises certifications are beneficial.Data Engineering Experience: minimum of 5 years in data engineering with significant exposure to on-premises and cloud infrastructure.DevOps Experience: some DevOps engineering experience would be ..... full job details .....