Python spark lead

Position: Python Spark Lead

Experience Required:

• 810 Years Total Experience
• Minimum 7 Years Relevant Experience in Python/PySpark/Spark Development

Location:

• Bangalore
• Chennai

Budget: 27 LPA

Role Overview

We are looking for an experienced Python Spark Lead responsible for designing, developing, and optimizing large-scale data processing solutions using Python, PySpark, Apache Spark, SQL, Airflow, and Control-M. The candidate will work on data engineering projects, build scalable ETL pipelines, optimize Spark jobs, and support enterprise data platforms.

Key Responsibilities

• Develop and maintain scalable data processing pipelines using Python and PySpark.
• Design, build, and optimize Apache Spark applications for large-scale data processing.
• Write complex SQL queries for data extraction, transformation, and analysis.
• Develop and manage ETL workflows using Spark and Python.
• Schedule and monitor workflows using Apache Airflow and Control-M.
• Perform Spark performance tuning, debugging, and optimization.
• Work with large datasets and distributed computing environments.
• Troubleshoot production issues and ensure data pipeline reliability.
• Collaborate with data architects, analysts, and engineering teams.
• Follow best practices for coding, testing, and deployment.

Mandatory Skills

• Python
• PySpark
• Apache Spark
• SQL
• Airflow
• Control-M
• ETL/Data Pipeline Development
• Spark Performance Optimization

Job Description: Python Spark Lead

Position: Python Spark Lead

Experience Required:

• 11–15 Years Total Experience
• Minimum 10 Years Relevant Experience in Python/PySpark/Spark Development

Location:

• Bangalore
• Chennai

Budget: 40 LPA

Role Overview

Tech Mahindra is seeking a senior Python Spark Lead with strong expertise in Python, PySpark, Apache Spark, SQL, Airflow, and Control-M to lead enterprise-scale data engineering solutions. The role requires deep experience in Spark architecture, pipeline development, optimization, production support, and technical leadership.

Key Responsibilities

• Lead the design and development of high-performance PySpark and Spark-based data solutions.
• Architect scalable data pipelines using Python, Spark, and SQL.
• Provide technical leadership for complex ETL and data engineering initiatives.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop and manage enterprise scheduling workflows using Airflow and Control-M.
• Review code, establish development standards, and mentor team members.
• Analyze and resolve complex production issues.
• Implement best practices for data processing, monitoring, and deployment.
• Collaborate with business, architecture, and engineering teams.
• Drive improvements in data platform performance and operational efficiency.

Mandatory Skills

• Python
• PySpark
• Apache Spark
• SQL
• Airflow
• Control-M
• Advanced ETL Development
• Spark Optimization & Performance Tuning
• Data Engineering Leadership