Pyspark Developer

Job Description – PySpark Developer

Position: PySpark Developer

Experience: 3–6 Years
Location: Bangalore / Bengaluru
Employment Type: Full-Time / Contract
Work Mode: Hybrid / As per client requirement

Role Overview

We are looking for a skilled PySpark Developer with strong experience in big data processing, data engineering, and distributed computing. The candidate will be responsible for developing, optimizing, and maintaining scalable data pipelines using PySpark, Python, and related Big Data technologies.

The ideal candidate should have hands-on experience in processing large datasets, developing ETL pipelines, performance tuning, and working with cloud-based data platforms.

Key Responsibilities

• Develop and maintain scalable ETL/ELT data pipelines using PySpark and Python.
• Design and implement data processing solutions for large and complex datasets.
• Write efficient and optimized PySpark DataFrame and Spark SQL transformations.
• Perform data cleansing, validation, transformation, and aggregation.
• Optimize Spark jobs for performance, scalability, and resource utilization.
• Work with structured and semi-structured data from multiple sources.
• Integrate data from databases, APIs, files, and other enterprise data sources.
• Troubleshoot data pipeline failures and resolve performance-related issues.
• Implement data quality checks and ensure data accuracy and consistency.
• Collaborate with Data Engineers, Data Architects, Analysts, and other stakeholders.
• Participate in code reviews, testing, deployment, and production support.
• Maintain technical documentation for data pipelines and processes.

Mandatory Skills

• Strong hands-on experience with PySpark.
• Strong programming experience in Python.
• Good knowledge of Apache Spark architecture and distributed computing.
• Strong understanding of Spark DataFrames, Spark SQL, RDDs, transformations, and actions.
• Experience developing ETL/ELT pipelines.
• Strong SQL skills and experience working with relational databases.
• Good understanding of data warehousing concepts.
• Experience working with large-volume datasets.
• Knowledge of Spark performance tuning and optimization techniques.
• Good understanding of data processing and integration concepts.

Good to Have

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Experience with Databricks.
• Knowledge of Azure Data Factory / AWS Glue / EMR or similar data engineering tools.
• Experience with Kafka or other streaming technologies.
• Knowledge of Hive, Hadoop, or HDFS.
• Experience with CI/CD and DevOps practices.
• Knowledge of data lake and lakehouse architecture.
• Experience working with Delta Lake.
• Familiarity with Airflow or other workflow orchestration tools.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies

• Strong analytical and problem-solving skills.
• Good communication and interpersonal skills.
• Ability to work independently as well as collaboratively.
• Strong debugging and troubleshooting capabilities.
• Ability to work in a fast-paced environment and manage multiple priorities.

Preferred Experience

Desired Candidate Profile

3–6 years of relevant experience in Data Engineering / Big Data, with strong hands-on experience in PySpark and Python.