Senior Data Engineer

As a Senior Data Engineer at SkillLane, you will design, build, and optimize scalable data pipelines and data platforms that power business intelligence, analytics, and AI initiatives. You will work closely with cross-functional teams to ensure high-quality, reliable, and secure data while driving data engineering best practices to support business growth and data-driven decision-making.

Job Descriptions:

• Leading the development and implementation of streaming and batch data pipelines to extract, transform, and load data from various sources into our data lakes/data warehouse.
• Implement self-service tools in the data platform (orchestration tools and data quality tools)
• Implement a data validation process to improve data quality in data pipelines
• Monitor the data platform cluster (Kubernetes)
• Implement a data observability process to monitor data pipelines
• Implement data lineage to tracking data movement
• Implement data dictionary, relationship and its interpretation
• Developing and maintaining data processing and analysis frameworks
• Ensuring data security, privacy, and compliance with relevant regulations and policies
• Collaborating with data analysts to understand data requirements and perform database design based on data understanding and requirements with consideration for performance
• Manipulating complex data from different sources (MySQL, PostgreSQL, MongoDB, Excel, CSV, and Google Sheets)
• Monitoring data quality and troubleshooting data-related issues
• Identifying and implementing process improvements to enhance the efficiency and effectiveness of our data systems

Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field
• 5+ years of experience in data engineering or related field
• Experience with streaming and batch data processing, ETL, and data warehousing
• Experience in Cloud-based data platforms (Azure, AWS, GCP, or others)
• Experience in Git, Docker, BigQuery, Airflow
• Experience with databases (e.g., MySQL, PostgreSQL, MongoDB)
• Strong programming skills (SQL, Python)
• Strong understanding of coding standards and testing
• Experience in managing projects and leading teams
• Have a growth mindset and a willingness to learn new things or share knowledge with others
• Ability to self-learn
• Understand Kubernetes is a plus
• Understanding the data lineage concept is a plus
• Understand data observability concept is a plus
• Understand data mesh concept is a plus