Lead Data Engineer in Norbiton

Job DescriptionLead Data Engineer (Databricks)Salary £65,000Type Full time, permanentHours 37.5 per weekLocation Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)OverviewWe are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.Key responsibilities* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP* Define lakehouse standards, data modelling approaches and governance frameworks* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies* Establish security, access control, data quality and compliance standards* Optimise performance, scalability and cloud cost management* Lead and mentor a small team of data engineers, setting technical standards and reviewing code* Partner with data science, analytics and product teams to ensure the platform supports their work* Own data quality, governance and observability across pipelines* Support AI and ML enablement including feature stores and model deployment patterns* Translate business requirements into technical solutions* Contribute to platform roadmap, tooling selection and continuous improvementRequired experience* Strong experience as a senior or lead data engineer building and scaling production data platforms* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows* Python and SQL proficiency* Infrastructure as code experience using Terraform, Bicep or CloudFormation* Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps* Strong understanding of data modelling, ETL and ELT patterns, distributed processing* Experience with security, governance and data platform operations* Experience leading or mentoring engineers, with strong stakeholder management* Rigorous approach to data quality and testing* Databricks certificationDesirable* Unity Catalog, MLflow and Databricks Asset Bundles* dbt or similar transformation frameworks* Retail or CPG data such as POS, inventory, supply chain* Streaming technologies such as Kafka or Event HubsEducation* Degree in computer science, data engineering or related field, or equivalent practical experienceTPBN1_UKTJ