Databricks Engineer

About the role

We are looking for an experienced Databricks Engineer to design, develop and maintain scalable data pipelines across Databricks and cloud platforms. You will integrate data from multiple sources, develop reliable transformation processes and support analytics, reporting and machine-learning workloads. You will work closely with analytics, product and infrastructure teams to enhance the organisation's data platform while maintaining strong standards for data quality, governance, monitoring and operational reliability.

Key responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines using Databricks and Delta Lake.
• Integrate data from databases, APIs, log files, streaming platforms and external providers.
• Develop transformation processes to cleanse, normalise, validate and aggregate data.
• Process complex, large-scale and inconsistent datasets using PySpark and Spark SQL.
• Build and maintain batch and real-time data-processing pipelines.
• Analyse business requirements and identify data extraction and integration needs.
• Participate in requirement-gathering, grooming and refinement sessions with users.
• Work with analytics and product teams to design and operationalise data pipelines.
• Collaborate with infrastructure teams to enhance cloud-based data platforms.
• Implement data governance and data-quality controls in accordance with company standards.

About you

• At least 3 years of experience in data engineering and developing scalable data pipelines.
• Strong hands-on experience with Databricks, PySpark, Spark SQL and Databricks notebooks and jobs.
• Experience designing data solutions, data models and distributed computing architectures.
• Experience orchestrating pipelines using Azure Data Factory, Apache Airflow or similar tools.
• Experience building data pipelines on Microsoft Azure; AWS experience is advantageous.
• Experience with both batch and real-time data processing.
• Strong SQL skills, including window functions, query tuning and performance optimisation.
• Experience with streaming technologies such as Apache Kafka, Apache Flink or AWS Kinesis.
• Understanding of DevOps practices, Git workflows and CI/CD pipelines.
• Databricks Certified Data Engineer Associate certification is mandatory.

Percept Solutions is expanding and actively seeking talented individuals. We encourage applicants to follow Percept Solutions on LinkedIn at https://www.linkedin.com/company/percept-solutions/ to stay informed about new opportunities and events.

EA License No: 18S9405 Reg No: R1330864