AI Data Engineer

AI Data Knowledge Engineer Hybrid- Remote/London 6 months + Inside IR35 SC Clearance (or eligible) Our customer, a central government organization, are building an enterprise-grade AI platform to enable secure, scalable and production-ready use of Generative AI and ML across the programme. The successful candidate will have experience in: Building integrations against source-system APIs (webhooks, REST, event streams) - connecting CDS estate systems to the platform Event-driven ingestion patterns - API Gateway/Lambda receivers, EventBridge, SQS per-consumer buffering State/watermark tracking (DynamoDB) and reconciliation to catch delivery gaps Python for pipeline/integration development Data migration experience ( eg AWS DMS) for onboarding new sources Nice to have: Experience integrating with Confluence, JIRA, or similar enterprise collaboration/ITSM tool APIs Apache Airflow and/or dbt Enterprise security and governance awareness (regulated environments) The Data Engineer is responsible for integrating CDS source systems into the AI platform - building the connectors and ingestion pipelines that get source-system data flowing into the platform reliably. Key Responsibilities: Build source-system integrations (webhooks API Gateway/Lambda EventBridge SQS) Implement watermark/state tracking (DynamoDB) and periodic reconciliation to catch webhook delivery gaps without placing polling load on source systems Onboard new CDS source systems onto the ..... full job details .....