Azure Databricks Engineer
Azure Databricks Data Engineer
Contract: 3 months initially
Start: Immediate
Location: Offshore / Remote
Duration: 3 months initially, with potential for extension
Engagement: Contract
Working Hours: UK-hours overlap required
We are looking for a hands-on Azure Data Engineer with strong Databricks experience to join an urgent, short-term data engineering project.
The successful candidate will be responsible for building and deploying end-to-end data pipelines using Azure Databricks, with a strong focus on automated CI/CD through Azure DevOps.
This is a highly delivery-focused role, so we are looking for someone who can get up to speed quickly, work independently and start contributing immediately.
Key Responsibilities
- Design, build and maintain data pipelines using Azure Databricks
- Develop production-grade data engineering solutions using PySpark and SQL
- Implement and maintain CI/CD pipelines for Databricks using Azure DevOps
- Build automated deployment processes using Azure DevOps YAML pipelines, Git repositories and branching strategies
- Deploy Databricks notebooks and pipelines across environments using automated release processes
- Work with Databricks Asset Bundles, Databricks Repos and environment promotion
- Build and maintain Databricks-native dashboards, including Databricks SQL / AI-BI dashboards
- Develop data solutions using Azure Data Lake Storage (ADLS) / Blob Storage
- Work with Azure Functions and Azure Container Apps as part of wider data engineering solutions
- Apply sound data modelling principles, including medallion architecture and/or dimensional modelling
- Build robust, scalable and maintainable end-to-end data pipelines
- Troubleshoot and optimise existing data pipelines and Databricks workloads
- Work closely with technical stakeholders to deliver against an urgent project timeline
Skills & Experience
- 4–7 years' commercial experience in Data Engineering
- Strong hands-on experience with Azure Databricks
- Strong PySpark and SQL skills
- Proven experience building production data pipelines in Databricks
- Strong experience with Azure DevOps
- Experience creating Azure DevOps CI/CD pipelines using YAML
- Experience with Git repositories, branching and automated deployment processes
- Experience deploying Databricks workloads through CI/CD rather than relying on manual notebook execution
- Experience with Databricks Asset Bundles and/or Databricks Repos
- Hands-on experience building Databricks-native dashboards / Databricks SQL or AI-BI dashboards
- Strong experience with ADLS / Azure Blob Storage
- Experience with Azure Functions and/or Azure Container Apps
- Good understanding of data modelling and data engineering principles
- Strong written and spoken English
- Ability to work with sufficient overlap with UK working hours
- Immediate or very short-term availability
Desirable
- Experience with Delta Lake
- Experience implementing medallion architecture
- Experience with Azure Key Vault and cloud security
- Experience with infrastructure-as-code
- Experience optimising Databricks performance and cloud costs
- Previous experience working on short-term, fast-paced delivery projects
WThis role is particularly suited to a hands-on Data Engineer, rather than someone whose experience is primarily architectural or managerial.
We are looking for someone who can take a requirement and independently build the solution from data ingestion through to transformation, deployment and dashboarding.
Candidates whose Azure experience is primarily ADF/Synapse with limited hands-on Databricks experience are unlikely to be suitable.
Similarly, candidates who have only used Databricks to run notebooks manually, without experience deploying workloads through CI/CD, will not be considered a strong match.
Contract Details
- 3-month initial contract
- Immediate start required
- Offshore / remote engagement
- Competitive, cost-conscious contract rate
- UK-hours overlap r