DE-Spark,Scala,kafka Professional

5-10 years Data Engineering

• Strong Scala programming

• Strong Apache Spark

• Strong Apache Kafka

• Spark Structured Streaming

• Advanced SQL

• ETL/ELT Data Pipelines

• Distributed Systems

• Cloud/Databricks exposure

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processingCloud / Big Data Experience

  Experience with at least one cloud platform is preferred:

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processingCloud / Big Data Experience

  Experience with at least one cloud platform is preferred:

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processingCloud / Big Data Experience

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting
  2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming
  4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts
  5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processing

  Cloud / Big Data Experience

  Experience with at least one cloud platform is preferred:

  Experience with at least one cloud platform is preferred:

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processingCloud / Big Data Experience

  Experience with at least one cloud platform is preferred:

• Required Technical Skills1. Apache Spark - Mandatory

  Strong hands-on experience with:

  • Apache Spark
  • Spark Core
  • Spark SQL
  • Spark DataFrame
  • Spark Dataset
  • Spark Structured Streaming
  • Spark transformations and actions
  • RDD concepts
  • Joins and aggregations
  • Window functions
  • Partitioning and repartitioning
  • Caching and persistence
  • Broadcast joins
  • Handling data skew
  • Spark job optimization
  • Spark cluster execution and troubleshooting2. Scala - Mandatory
  • Strong programming experience in Scala.
  • Functional programming concepts.
  • Collections and higher-order functions.
  • Case classes, traits, objects, pattern matching.
  • Exception handling and reusable code development.
  • Development of Spark applications using Scala.
  • Ability to write clean, modular, scalable, and maintainable Scala code.3. Apache Kafka - Mandatory

  Strong practical experience with:

  • Kafka architecture
  • Kafka brokers
  • Topics and partitions
  • Producers and consumers
  • Consumer groups
  • Offsets and offset management
  • Replication factor
  • Partition strategy
  • Message retention
  • Kafka Producer/Consumer APIs
  • Kafka Streams / Kafka Connect
  • Schema Registry
  • Avro / JSON serialization
  • Consumer lag monitoring
  • Kafka troubleshooting
  • Kafka performance tuning
  • Integration of Kafka with Spark Structured Streaming4. SQL - Mandatory

  Strong SQL skills including:

  • Complex SQL queries
  • Joins
  • Subqueries
  • CTEs
  • Window functions
  • Aggregations
  • Query optimization
  • Data validation and reconciliation
  • Stored procedures/functions where applicable
  • Relational database concepts5. Data Engineering

  Strong understanding of:

  • ETL / ELT
  • Batch processing
  • Real-time/streaming processing
  • Data pipelines
  • Data lakes
  • Data warehouses
  • Data modeling
  • Dimensional modeling
  • Distributed systems
  • Data partitioning
  • Data quality
  • Data governance
  • Large-scale data processingCloud / Big Data Experience

  Experience with at least one cloud platform is preferred:

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.