Data Engineer

Summary We are seeking a skilled and experienced Data Engineer specializing in Apache Spark to join our team. The ideal candidate will possess a deep understanding of data engineering principles, coupled with substantial experience in Apache Spark. The Data Engineer will be primarily responsible for designing, developing, and maintaining our data pipelines and infrastructure. Collaborating closely with other members of the data engineering team, they will contribute to the creation and implementation of innovative data-driven solutions.

Responsibilities Design, develop, and maintain data pipelines and infrastructure.

Collaborate with other data engineering team members to devise and execute new data-driven solutions.

Liaise with stakeholders to gather requirements and translate them into technical specifications.

Troubleshoot and debug data pipelines and infrastructure.

Monitor and optimize data pipelines and infrastructure.

Stay abreast of the latest data engineering technologies and trends.

Standardize data and create data insights and ML models.

Have exposure to ML and MLOps concepts.

Skills Bachelor's degree in computer science, information technology, or a related field.

4+ years of experience in data engineering.

Proficiency with Apache Spark.

Prior experience with the Databricks data platform is a must.

Familiarity with AWS data tools.

Familiarity with Databricks ML toolset (must).

Knowledge of big data technologies such as Hadoop and Kafka.

Experience with data warehousing and data lakes.

Competence in data modeling and ensuring data quality.

Proficiency in SQL and NoSQL databases.

Familiarity with cloud computing platforms like AWS, Azure, and GCP.

Strong problem-solving and analytical skills.

Excellent communication and interpersonal abilities.

Desired Qualifications Master's degree in computer science, information technology, or a related field.

Additional 5+ years of experience in data engineering.

Exposure to machine learning and artificial intelligence is advantageous.

Familiarity with distributed systems.

Experience with open-source software development.

Knowledge of Agile development methodologies.