Lead Expert Data Engineer- Procurement

Job Role

Lead Expert Data Engineer-Procurement

Location

Hyderabad

Job Summary

In this role you will design, maintain robust data pipeline and manage team, projects/initiatives ensuring seamless data integration and transformation. Collaborate with data modelers and business partners to drive impactful projects centered around our data.

Responsibilities

• Provide technical recommendations and evaluate project feasibility and workloads.
• Design and implement efficient pipeline architectures for data ingestion, transformation, and storage.
• Develop modular, efficient code for data transformation and integration, ensuring adherence to modern development practices.
• Perform data ingestion from source systems, including ETL processes, when tables are not available on the data platform.
• Conduct regular pipeline maintenance and upgrades, troubleshooting and resolving issues to ensure optimal performance.
• Implement data quality testing (automatic) and validation to maintain high standards of data integrity and reliability.
• Collaborate closely with other data engineers, data modelers, data stewards, BI developers, data scientists, and subject matter experts to ensure seamless data integration, knowledge sharing, and delivery of impactful solutions.

You Bring

• Expert knowledge in dbt, SQL, Python, Spark (PySpark), Git, Azure DevOps and Databricks.
• Expertise in cloud engineering (e.g., AWS, Azure); experience with infrastructureascode is a plus, and deploying CI/CD pipelines is a must.
• 5 years of experience in creating and maintaining data pipelines. Experience handling scientific datasets for cheminformatics or bioinformatics (e.g., microbiome) is a plus.
• Experience with FAIR data principles, data governance, and data standards commonly used in scientific domains. Ability to collaborate closely with data experts and SMEs to ensure seamless data integration and transformation.
• Consistent application of best practices in coding, testing and documentation, delivering scalable and reliable data solutions with automated quality checks and monitoring tools.
• Curiosity and a growth mindset, always asking Is there a better way to leverage data for smarter decisionmaking. Strong focus on deployment development and automation.