Big Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Python and Spark on Snowflake platform.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions that meet business needs.
• Develop complex SQL queries to optimize database performance and troubleshoot issues in Snowflake databases.
• Participate in code reviews to ensure adherence to coding standards, best practices, and quality guidelines.

Job Requirements :

**- Build data pipelines to ingest, transform, and integrate client data into the platform.

• Partner with AI/ML Engineers to ensure ML-ready, high-quality datasets for the POD's use cases.

• Implement data validation, quality checks, and monitoring for client-specific pipelines.

• Support data integration with client systems, including secure data exchange protocols.

• Troubleshoot data issues within the POD''s scope and coordinate fixes with Core Platform teams.

• Maintain documentation of data flows, transformations, and lineage for audit purposes.** **Must have skills:

• Strong experience with data pipeline tools (Airflow, dbt, Spark, or equivalent).

• Proficiency in SQL and Python for data transformation and validation.

• Experience integrating with diverse client data sources and APIs.

• Understanding of data quality, governance, and lineage practices.

• Familiarity with cloud data platforms (Snowflake, BigQuery, Databricks, or equivalent).

Must have Finance / BFSI Exposure:

• Experience handling financial or transactional client data with strict quality and security requirements.
• Understanding of data privacy and compliance requirements for financial services clients.
• Familiarity with common financial data formats and integration standards.

GOOD TO HAVE:

• Experience with real-time/streaming data integration for client use cases
• Exposure to data anonymization or masking techniques for sensitive financial data**