Data Engineer

Role: Data Engineer ( PySpark, Hadoop, Cloudera and SQL )

Experience: 5 to 8 years

Location: Thane

We are seeking a skilled and passionate Data Engineer with 4-10 years of experience to join our data engineering team The ideal candidate will play a key role in designing developing and maintaining high performance data pipelines using PySpark and other Big Data technologies

Key Responsibilities
1 Develop and maintain robust data pipelines using PySpark ensuring they are scalable efficient and optimized for performance and availability
2 Work with large scale datasets  managing the full data lifecycle from ingestion transformation to consumption in a distributed environment
3 Collaborate with business and analytics teams to identify data requirements understand use cases and deliver technical solutions that solve complex business problems
4 Design and implement data workflows across data lakes with hands on experience in Hadoop ecosystems Cloudera
5 Strong command of SQL with the ability to write optimized queries for large datasets in tools such as Hive or Spark SQL
6 Experience working with big data tools like HDFS YARN HBase Oozie Hive Map Reduce and building streaming solutions using Spark Streaming
7 Maintain code quality and version control using Git and perform thorough technical debugging and performance tuning of data jobs
8 Create and maintain technical design documentation for all ETL pipelines and solutions developed ensuring transparency and reproducibility
9 Knowledge of working with APIs for data integration and building services that interface between various systems

Preferred Skills Qualifications
4-7 years of hands-on experience in Data Engineering with a strong emphasis on PySpark and the Hadoop ecosystem
Solid understanding of distributed computing principles and data architecture best practices
Experience in handling large scale datasets and designing fault tolerant highly available systems
Strong analytical problem solving and communication skills