ETL Data - Senior Engineer

Job highlights

• Expertise in PySpark, Databricks Workflows, Delta Lake, Snowflake, and Amazon Kinesis
• Design and optimize scalable data pipelines, architect event-driven data solutions, lead batch and streaming data processing, and mentor team members on best practices

Job Summary

ETL Data - Senior Engineer

Posted On:

Location: Noida, UP, India

Company: Iris Software

Mandatory Skills

• Databricks Workflows,Delta Lake on Databricks,Snowflake,PySpark,Amazon Kinesis

Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Design and optimize event-driven data architectures using Apache Kafka or Amazon Kinesis.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.
• Promote responsible use of AI-assisted engineering capabilities to improve development productivity, testing, documentation, and engineering quality.
• Review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards.
• Troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis.
• Mentor team members on PySpark, Snowflake, Delta Lake, Kafka, Kinesis, Airflow, and data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.

Behavioral Competencies

• Demonstrates strong ownership while driving data engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Apply strong analytical thinking to evaluate complex data engineering and platform challenges.

Mandatory Competencies

• Big Data - Big Data - Pyspark
• Data AI - Data Engineering - Data Quality Validation
• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Database - Database Programming - SQL
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Cloud - AWS - AWS SNS, AWS SQS, AWS Kinesis
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Data AI - ETL OTHERS - Snowflake
• Data AI - BI Reporting and Data Visualization - ETL Concepts
• Data AI - ETL OTHERS - AWS Glue
• Cloud - AWS - Dynamo DB, Amazon Aurora, Amazon RDS
• Data AI - Database - MS SQL
• Database - Database Programming - PL/SQL
• Beh - Communication and collaboration