Big data engineer/Developer

IDESLABS PRIVATE LIMITED is looking for Big data engineer/Developer to join our dynamic team and embark on a rewarding career journey

• As a Big Data Engineer/Developer, you will be responsible for designing, implementing, and maintaining scalable and efficient big data solutions
• This role involves working with large datasets, developing data processing pipelines, and collaborating with cross-functional teams to meet the organization's data analytics and business intelligence needs

Key Responsibilities:

• Data Architecture and Design:Design and implement scalable and robust data architecture for big data solutions
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Big Data Processing:Develop and implement data processing pipelines for ingesting, processing, and analyzing large volumes of data
• Utilize big data technologies such as Hadoop, Spark, and Flink
• Database Management:Implement and manage large-scale distributed databases for structured and unstructured data
• Optimize database performance and ensure data integrity
• ETL (Extract, Transform, Load) Processes:Develop and maintain ETL processes to integrate data from various sources into the big data platform
• Ensure data quality and consistency throughout the ETL process
• Data Modeling:Design and implement data models for efficient storage and retrieval of data
• Optimize data storage and query performance
• Data Security and Compliance:Implement security measures to protect sensitive data
• Ensure compliance with data privacy and regulatory requirements
• Monitoring and Optimization:Monitor the performance and health of big data systems
• Identify and address bottlenecks and optimize data processing workflows
• Collaboration:Work closely with data scientists, analysts, and other stakeholders to understand data analytics requirements
• Collaborate with cross-functional teams to integrate big data solutions into business processes
• Documentation:Create and maintain documentation for data architectures, processes, and workflows
• Provide documentation for troubleshooting and support
• Continuous Learning:Stay informed about the latest developments in big data technologies and best practices
• Continuously improve skills and knowledge in the field