Python Developer(Pyspark)

Job Title: Python Developer (PySpark)

Experience: 5-7 Years
Location: Chennai
Employment Type: Full-Time

Job Summary

We are looking for an experienced Python with PySpark Developer to design, develop, and maintain scalable data processing solutions in an enterprise data engineering environment. The ideal candidate should have strong expertise in Python, PySpark, SQL, and ETL development, with experience building high-performance data pipelines, optimizing large-scale data processing, and supporting data-driven applications. Candidates should be comfortable working in Agile environments and collaborating with cross-functional teams to deliver reliable and scalable solutions.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion, transformation, and processing pipelines using Python, PySpark, and SQL.
• Develop and optimize complex SQL queries, stored procedures, and data transformation logic.
• Build and enhance ETL workflows for processing large volumes of structured and unstructured data.
• Develop efficient Spark applications using DataFrames, Spark SQL, and performance optimization techniques.
• Perform code reviews, unit testing, integration testing, and ensure high-quality deliverables.
• Monitor, troubleshoot, and optimize production data pipelines and resolve performance bottlenecks.
• Implement CI/CD practices for automated deployment and maintenance of data engineering solutions.
• Collaborate with architects, business analysts, data engineers, and stakeholders to understand business requirements and translate them into technical solutions.
• Create and maintain technical documentation, including data lineage, pipeline architecture, and operational procedures.
• Participate in Agile ceremonies such as sprint planning, stand-ups, retrospectives, and backlog refinement.
• Support production issues, perform root cause analysis, and implement preventive measures.
• Mentor junior developers and promote coding standards and best practices.

Required Technical Skills

• Strong hands-on experience in Python programming.
• Expertise in PySpark for large-scale data processing.
• Strong knowledge of Spark SQL, DataFrames, Structured Streaming, and Spark optimization techniques.
• Advanced SQL skills including query optimization, joins, stored procedures, and analytical functions.
• Experience in ETL development and data transformation.
• Hands-on experience with Pandas, NumPy, and Python data processing libraries.
• Experience with REST APIs and microservices integration.
• Knowledge of object-oriented programming and software design principles.
• Familiarity with relational databases such as MS SQL Server or Oracle.
• Experience working with Git, Jenkins, Docker, and CI/CD pipelines.
• Good understanding of Linux/Unix environments and shell scripting.
• Exposure to Agile development methodologies (Scrum/Kanban).
• Experience working in enterprise data engineering projects.

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.