Data/Streaming Engineer

Role purpose Build reliable capability on the existing enterprise data platform and Databricks estate, then evolve towards lower-latency, event-driven processing where the use case justifies it. Responsibilities: Build and operate batch, micro-batch and streaming ingestion/processing pipelines. Work pragmatically with existing Databricks/Spark capability while introducing lower-latency event patterns progressively. Design schemas, data contracts and evolution strategies between independently changing systems. Build for replay, idempotency, ordering, failure recovery, monitoring and operational support. Integrate data/event capability with application services and predictive/ML components. Own data quality, observability, CI/CD and production reliability. Skills/Experience Required: Real production streaming/event-driven experience with Kafka, Azure Event Hubs or comparable technology. Can explain partitions, consumer groups, ordering, delivery semantics, replay, idempotency, schema evolution, failure recovery, monitoring and performance from lived production experience. Strong hands-on Python and distributed data-processing capability; Spark/Databricks strongly useful. Production pipelines, transformations, data quality, schemas/contracts and cloud operation. Evidence of personally designing, building, debugging and operating streaming systems - not merely consuming an existing feed. Advantageous Skills: Databricks and Apache Spark depth. Azure production environments. ..... full job details .....