Data Analyst

Key Responsibilities:

• Design and develop scalable batch and real-time data pipelines using Spark

• Build and maintain distributed streaming systems using Kafka

• Develop ETL/ELT workflows for large-scale structured and unstructured datasets

• Optimize data processing jobs for performance

• Implement data ingestion frameworks for high-volume data sources

• Ensure data quality, integrity, and reliability across pipelines

• Collaborate with cross-functional teams to define data architecture and solutions

• Monitor, troubleshoot, and resolve data pipeline issues

• Implement data security and governance standards

Required Skills & Qualifications

• Bachelors/Master’s degree in Computer Science, Engineering, or related field

• 4+ years of experience in data engineering or big data development

• Strong hands-on experience with Apache Spark (Core, SQL, Structured Streaming)

• Experience with Apache Kafka, Kafka Streams, and connectors

• Proficiency in Python

• Experience with distributed data processing and parallel computing

• Strong knowledge of data structures and algorithms

• Experience with ETL tools and workflow orchestration

• Familiarity with data lake and data warehouse architectures

• Experience with SQL and NoSQL databases

Preferred Qualifications

• Knowledge of modern data formats (Parquet, Avro, ORC)

• Experience with CI/CD pipelines and DevOps practices

• Familiarity with data governance and metadata management tools

• Real-time analytics and event-driven architecture experience

• Experience working in Agile/Scrum environments

Key Competencies

• Strong problem-solving and analytical skills

• Excellent communication and teamwork abilities

• Ability to work with large, complex datasets

• Strong attention to detail

• Self-motivated with ability to work independently

Nice to Have

• Experience with machine learning pipelines

• Knowledge of data visualization tools

• Certification in big data technologies