Data Engineer Staff

About the role

Building robust large-scale social media crawler for various social media platforms. Maintain day-to-day operation of said crawlers alongside our data quality team to make sure the completeness and timeliness of incoming data. Work in team along with several stakeholders to better understand and communicate about data requirements and issues.

Key responsibilities

• Building robust large-scale social media crawler for various social media platforms
• Maintain day-to-day operation of crawlers alongside data quality team
• Ensure completeness and timeliness of incoming data
• Work with stakeholders to understand and communicate data requirements and issues

About you

• Intermediate or advanced understanding of SQL
• Experience in processing large volumes of data from social media and online media platforms
• Proficiency in several programming languages, mainly in Javascript (Node) and Python
• Good experience in using Scala for production (not related to Spark)
• Intermediate or advanced understanding of several data-related technologies such as MariaDB, Postgres, Redis, Kafka, and Elasticsearch
• Ample experience using Linux-based systems
• Understanding of several data acquisition techniques such as webscraping, API-based data consumption
• Fresh Graduates Welcome