Level 3 Big Data Developer

Job highlights

• . Should be able to design,develop,and implement highly scalable and distributed big data solutions using Hadoop ecosystem technologies such as HBase,Hive,Kudu,and Spark

• Should be able to design, develop, and implement highly scalable and distributed big data solutions using Hadoop ecosystem technologies such as HBase, Hive, Kudu, and Spark.

• Should have knowledge on Architect HBase schemas and data models to accommodate evolving business requirements and ensure optimal performance for data storage and retrieval operations.

• Should have knowledge on developing complex Hive queries and data processing pipelines to transform raw data into structured formats suitable for analysis and reporting.

• Implement data ingestion pipelines using Spark Streaming and Spark SQL for real-time processing of streaming data sources, ensuring high throughput and low latency.

• Optimize Spark applications for performance and resource utilization, including tuning RDD transformations, optimizing data partitioning strategies, and leveraging in-memory caching.

• Utilize advanced features of Spark MLlib for machine learning tasks such as classification, regression, clustering, and collaborative filtering.

• Design and deploy Kudu tables for fast analytical queries and real-time analytics, leveraging Kudu s unique combination of fast analytics and fast data ingestion.

• Collaborate with data scientists to integrate machine learning models into Spark workflows and productionize them for real-time predictions and analytics.

• Troubleshoot performance bottlenecks, data quality issues, and system failures in big data applications and infrastructure, and implement solutions to address them.

• Stay abreast of emerging technologies and best practices in big data processing and analytics, and evaluate their potential impact on our architecture and solutions.