Hadoop / PySpark

Good to have skills: Spark SQL, YARN, HDFS, Oozie, Airflow

Key Responsibilities:

• Lead the design and development of scalable Big Data solutions using Hadoop and PySpark for batch and large-scale processing.
• Architect and implement end-to-end data pipelines, ensuring reliability, performance tuning, and efficient resource utilization on Hadoop clusters.
• Develop and optimize Hive data models, queries, and partitioning strategies to support analytics and downstream consumption.
• Drive technical planning, estimation, and delivery for data engineering initiatives, ensuring timelines and quality standards are met.
• Establish coding standards, review code, and enforce best practices for maintainability, testing, and production readiness.
• Troubleshoot production issues, perform root-cause analysis, and implement preventive measures to improve stability and throughput.
• Collaborate with product, analytics, and platform teams to translate requirements into scalable technical solutions.
• Mentor team members, guide technical decisions, and support skill development across Hadoop, PySpark, Big Data, and Hive. Minimum Qualifications:
• Education: BTECH, MTECH, MCA, MSC (or equivalent).
• 5–9 years of overall experience with strong hands-on expertise in Hadoop and PySpark for large-scale data processing.
• Proven experience building and maintaining Big Data pipelines and working with Hive for querying and data modeling.
• Strong understanding of distributed processing concepts, performance optimization, and data reliability practices.
• Experience leading technical execution through code reviews, design discussions, and delivery ownership. Preferred Qualifications:
• Experience designing reusable frameworks and standardized pipeline patterns to improve team productivity and consistency.
• Strong expertise in optimizing Spark jobs (partitioning, caching, shuffles) and Hive performance (file formats, partitions, bucketing).
• Experience implementing data quality checks, monitoring, and operational dashboards for production pipelines.
• Ability to drive stakeholder communication, manage technical trade-offs, and lead solutioning for complex data use cases.
• Demonstrated mentoring and leadership experience, enabling teams to deliver high-quality Big Data solutions at scale.