data engineer

Responsibilities

• Build ingestion workflows for APIs, web data, and files.
• Handle structured and semi-structured formats (JSON, XML, HTML, CSV).
• Apply resilient scraping practices, including session handling, proxies, and request automation.
• Ensure compliance with security and access restrictions when integrating external data sources.
• Optimize SQL queries and transformations for analytics.
• Design, build, and maintain robust Python-based data pipelines.
• Implement testing frameworks and good engineering practices for reliability.
• Ensure data quality, consistency, and scalability across workflows.

Requirements

• Expert-level Python for data engineering and workflow automation.
• Knowledge of web scraping, ingestion workflows, and handling varied data formats.
• Experience designing and maintaining production-grade data pipelines.
• Familiarity with best practices in testing, monitoring, and pipeline reliability.
• Proficiency with version control (Git/GitHub/Azure DevOps).
• Strong SQL skills for ETL and analytics.

Qualifications

• BSc or MSc in Computer Science, Data Engineering, Software Engineering, or related fields.
• Equivalent practical experience will also be considered.