Lead BigData Engineer (Databricks + Azure)

About The Role In this role, you will lead the design and development of scalable data solutions using the Databricks Lakehouse platform within Azure cloud environments. You will collaborate with technical and business stakeholders to deliver high-quality batch and streaming data pipelines, contribute to architectural decisions, and support the full project lifecycle, from proof of concept to enterprise-scale implementation within a collaborative and innovation-driven team.

Responsibilities

• Design and develop scalable batch and streaming ETL pipelines using Python and PySpark
• Build and optimize data solutions based on Databricks Lakehouse architecture and best practices
• Develop and maintain data models to support business and analytical needs
• Work with Azure cloud services, including Azure Data Factory, Azure Synapse, Azure Data Lake Services, and Azure DevOps
• Implement and optimize advanced SQL solutions for data processing and analytics
• Participate in architecture discussions, conduct trade-off analysis, and recommend optimal technical solutions
• Collaborate closely with technical and business stakeholders to understand requirements and deliver effective data solutions
• Support continuous improvement, knowledge sharing, and engineering best practices within the team
• Contribute across the full project lifecycle, from PoC and MVP stages to production implementation

Requirements

• Hands-on experience with Python and PySpark for data engineering solutions
• Strong knowledge of Databricks Lakehouse architecture and related concepts
• 2+ years of experience designing data models, building ETL pipelines, and solving business problems through data solutions
• Practical experience with Azure cloud technologies, including Azure Data Factory, Azure DevOps, Azure Synapse, and Azure Data Lake Services
• Advanced SQL skills and experience working with relational databases
• Understanding of database and data warehouse design best practices
• Experience designing, building, and scaling both batch and streaming data pipelines
• Ability to evaluate technical options, conduct trade-off analysis, and solve complex problems
• Strong communication and stakeholder management skills
• Upper-intermediate or higher level of English

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.