Tcs Pyspark Data Engineer

Job Description PySpark Data Engineer

Job Title: PySpark Data Engineer
Experience: 5-10 Years
Location: Chennai

Job Summary

We are looking for a skilled PySpark Data Engineer to design, develop, and optimize large-scale data processing pipelines. The ideal candidate should have strong expertise in PySpark, Spark ecosystem, SQL, ETL development, and cloud/big data technologies.

Key Responsibilities

• Develop and optimize data pipelines using PySpark and Apache Spark.
• Perform data ingestion, transformation, and processing of large datasets.
• Design scalable ETL/ELT solutions for data warehousing and analytics.
• Work with distributed computing frameworks and big data technologies.
• Integrate data from multiple sources and ensure data quality.
• Collaborate with Data Scientists, Analysts, and Business teams.
• Troubleshoot performance issues and optimize Spark jobs.
• Follow best practices for data governance, security, and code quality.

Required Skills

• Strong hands-on experience in PySpark development.
• Expertise in Spark SQL, DataFrames, and Spark optimization techniques.
• Experience with Hadoop ecosystem (Hive, HDFS, Kafka).
• Good knowledge of SQL and database concepts.
• Experience in ETL pipeline development.
• Knowledge of Azure Databricks, Azure Data Factory, or AWS EMR is preferred.
• Experience with Git, CI/CD, and Agile methodologies.