Big Data Engineer

Big Data Engineer Big Data | Spark | Hadoop | Python | SQL

Company: Spark Techwave InnovationZ Pvt Ltd
Job Type: Full-Time
Experience: 2–6 Years
Location: Chennai / Remote / Hybrid
Department: Data Engineering

Job Summary

Spark Techwave InnovationZ Pvt Ltd is looking for a skilled Big Data Engineer to design, develop, and maintain scalable data processing pipelines and big data solutions.

The ideal candidate should have hands-on experience with Apache Spark, PySpark, Hadoop ecosystem, Python, SQL, ETL/ELT, and distributed data processing. Experience with cloud-based big data platforms will be an added advantage.

Key Responsibilities

• Design, develop, and maintain scalable big data pipelines.
• Build and optimize batch and real-time data processing workflows.
• Develop data transformation and processing jobs using Apache Spark / PySpark.
• Work with large and complex datasets across multiple data sources.
• Develop ETL/ELT pipelines for data ingestion and transformation.
• Write optimized SQL queries for large-scale data processing.
• Work with distributed data storage and processing systems.
• Perform data cleansing, validation, transformation, and quality checks.
• Monitor data pipelines and troubleshoot performance or data-quality issues.
• Optimize Spark jobs, SQL queries, and data-processing workflows.
• Collaborate with Data Scientists, Data Analysts, Software Engineers, and business stakeholders.
• Implement best practices for data security, reliability, scalability, and performance.
• Maintain technical documentation for data pipelines and processing workflows.

Required Skills

• Strong experience with Apache Spark / PySpark.
• Strong programming knowledge in Python or Scala.
• Strong SQL skills.
• Good understanding of Big Data concepts and distributed computing.
• Experience developing ETL/ELT pipelines.
• Understanding of Hadoop ecosystem.
• Experience working with large datasets.
• Good knowledge of data structures and data processing techniques.
• Strong analytical and problem-solving skills.

Big Data Technologies

Experience with one or more of the following:

• Apache Spark
• PySpark
• Hadoop
• HDFS
• Hive
• Kafka
• Sqoop
• Oozie
• Databricks

Cloud & Data Platforms – Good to Have

Experience with one or more:

• AWS
• Microsoft Azure
• Google Cloud Platform
• Amazon EMR
• AWS Glue
• Azure Databricks
• Azure Data Factory
• Google Dataproc
• Snowflake
• BigQuery
• Redshift

Streaming – Good to Have

• Apache Kafka
• Spark Streaming
• Structured Streaming
• Event-driven data processing

Database Knowledge

Experience with relational and/or NoSQL databases such as:

• MySQL
• PostgreSQL
• SQL Server
• MongoDB
• Cassandra

Additional Skills

• Git / GitHub
• Docker
• CI/CD
• Linux
• REST APIs
• Data warehousing concepts
• Data modeling

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

Experience

2–6 years of relevant experience in Big Data Engineering, Data Engineering, ETL development, or distributed data processing.

Preferred Candidate Profile

Candidates with hands-on experience in:

• Big Data Engineering
• PySpark / Spark
• Hadoop
• Data Pipeline Development
• ETL/ELT
• Python
• SQL
• Cloud Data Engineering
• Data Warehousing
• Real-time / streaming data processing

Key Skills for Naukri

Big Data, Big Data Engineer, Apache Spark, PySpark, Hadoop, HDFS, Hive, Python, SQL, ETL, ELT, Kafka, Spark Streaming, Databricks, AWS, Azure, GCP, Data Engineering, Data Pipeline, Data Warehouse

Interested candidates can apply with their updated resume at [email protected]