Big Data Engineer (8-16 Years)

Job highlights

• 5-15 years experience with Hadoop ecosystem, Apache Spark, Kafka, and cloud platforms (AWS, Azure, GCP)
• Design, develop, and maintain scalable Big Data solutions, build data pipelines, implement batch and real-time processing, and support cloud migration

Job description

Position: Big Data Developer
Experience: 5 to 15 Years
Location: PAN India
Employment Type: Full-Time

About the Role

We are looking for experienced Big Data Developers to design, build, and optimize large-scale data processing systems and analytics platforms. The ideal candidate will possess strong expertise in Big Data technologies, distributed computing, cloud platforms, and data engineering practices to support enterprise data-driven initiatives.

Key Responsibilities

• Design, develop, and maintain scalable Big Data solutions for processing and analyzing large volumes of structured and unstructured data.
• Build and optimize data pipelines, ETL/ELT processes, and real-time streaming applications.
• Develop data ingestion frameworks from multiple sources including APIs, databases, files, and streaming platforms.
• Implement batch and real-time data processing solutions using Big Data technologies.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and stakeholders to understand business requirements.
• Ensure data quality, integrity, governance, and security across the data ecosystem.
• Optimize data processing performance and troubleshoot production issues.
• Participate in architecture reviews, code reviews, and technical discussions.
• Support cloud migration and modernization initiatives.
• Develop and maintain technical documentation and operational procedures.

Required Skills

Big Data Technologies

• Strong hands-on experience with:

• Hadoop Ecosystem

• Apache Spark (Core, SQL, Streaming)

• Hive

• HDFS

• YARN

• Experience developing distributed data processing applications.

Data Engineering

• Expertise in ETL/ELT development.
• Strong knowledge of data modeling and data warehousing concepts.
• Experience with large-scale data transformation and migration projects.
• Understanding of data governance and data quality frameworks.

Programming Skills

• Strong proficiency in:

• Python

• Scala

• Java

• SQL

• Experience with scripting and automation.

Streaming Technologies

• Experience with:

• Apache Kafka

• Spark Streaming

• Apache Flink (Preferred)

• Apache NiFi

Cloud Technologies

• Experience with one or more cloud platforms:

• AWS

• Azure

• Google Cloud Platform (GCP)

Database Technologies

• Strong experience with:

• SQL Server

• Oracle

• PostgreSQL

• MySQL

• MongoDB

• Cassandra (Preferred)

Tools & Technologies

• Apache Spark
• Hadoop
• Hive
• Kafka
• Scala
• Databricks
• Snowflake
• AWS
• Azure

Notice Period: Immediate to 30 Days Preferred

Keywords: Big Data, Spark, Hadoop, Kafka, Hive, Databricks, Data Engineering, Python, Scala, AWS, Azure, GCP, Snowflake, Airflow, ETL, Data Lake, Data Warehouse, Real-Time Streaming.

Provide your feedback on BizChat