Overview
Lead Databricks Engineer
About the Role
We have an exciting opportunity for a skilled Lead Databricks Engineer with deep PySpark expertise and hands-on experience in building pipelines across Databricks and Azure Data Factory.
The role will be based out of our Bournemouth office and is a fixed-term contract for 12 months.
You will design and deliver robust, scalable, and governed data flows that form the backbone of our Lakehouse platform, ensuring high-quality, trusted data for analytics, reporting, and regulatory requirements.
Key Responsibilities
•Build and maintain Databricks pipelines (batch and incremental) using PySpark and SQL.
•Orchestrate end-to-end data workflows with Azure Data Factory.
•Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming).
•Implement the Medallion Architecture (Bronze, Silver, Gold) to transform raw data into business-ready datasets.
•Apply row-count checks, logging, and error-handling frameworks for pipeline monitoring and governance.
•Integrate Databricks pipelines with downstream systems including Power BI and reporting platforms.
•Collaborate with analysts, business teams, and other engineers to deliver consistent, well-documented datasets.
•Support deployments and automation via Azure DevOps CI/CD pipelines.
•Gather and refine requirements from business stakeholders.
About You
•Strong PySpark development skills for large-scale data engineering.
•Proven experience with Databricks pipelines and workflow management.
•Expertise in Azure Data Factory for orchestration and scheduling.
•Solid knowledge of Delta Lake and Lakehouse data modelling principles.
•Hands-on experience with SQL for data transformation and validation.
•Familiarity with Azure services (ADLS/Blob, Key Vault, SQL).
•Knowledge of ETL/ELT frameworks, error logging, and monitoring.
Desirable skills
•Understanding of Microsoft Fabric.
•Experience with DevOps practices (CI/CD, release pipelines).
•Awareness of data governance and lineage tools.
•Experience working in the Finance sector.
Rewards & Benefits
This role is a Band C in the LV= Structure.
At LV= Life and Pensions, you’ll go above and beyond to do the right thing for our customers. We’ll reward your hard work with an attractive, competitive salary and benefits package, which includes:
•30 days’ holiday, with the option to buy up to 2 additional days
•Competitive pension scheme – LV= Life and Pensions will double-match the amount you pay, up to 14% (subject to National Minimum Wage requirements)
•An annual bonus scheme based on company and personal performance
•Single-cover private medical insurance (with the option for you to upgrade to family cover)
•Flexible benefits, including a cycle to work scheme, personal accident insurance, critical illness cover and dental insurance
•Up to 20% discount on our life products for you and your immediate family
•A group life assurance policy with 4 x your basic pay to go to your dependents (you’ll have the option to increase to 8 x cover)
•Group Income Protection (if you become a member of the Pension scheme and reach 5 years of service)
•Access to our Employee Assistance Programme (EAP) for support when you need it
•A virtual GP service
•Shared parental leave
We’re proud of our inclusive culture at LV= and, as an equal-opportunity employer, we continually work to remove unconscious bias from our recruitment process. We value our colleagues for what they bring to our team regardless of any protected status or characteristics they may have. Talk to us about flexible working as part of your application; if it’s right for you, our members and customers, and our business, then we’ll do everything we can to make it happen.
Please note that we are unable to offer Skilled Worker Visa Sponsorship for this role. Therefore, you must ensure that you are eligible to work in the UK without our sponsorship for your application to be considered.
Any successful internal colleagues will be offered a secondment opportunity. The secondment policy is available on the intranet for internal colleagues.
IMPORTANT: Before applying for this role, please make sure you have the right to work in the country where the role is based. Unless it clearly stipulates within in the job advert above that the hiring company is looking to or able to sponsor applicants it is deemed that the hiring employer will only consider applications from those able to comply with and work in the country where the role is based.
