Principal Data Engineer

Role & responsibilities

• Lead the design and implementation of scalable, secure, and high-performance enterprise data architectures.
• Define data engineering standards, architectural guidelines, and best practices across data platforms and pipelines.
• Design and optimize batch and real-time data processing solutions using Apache Spark, Kafka, and other distributed computing technologies.
• Architect and manage modern data platforms, including data lakes, data warehouses, and lakehouse solutions.
• Drive cloud-based data engineering initiatives across AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Develop and optimize complex ETL/ELT pipelines to ensure reliable, efficient, and high-quality data delivery.
• Establish data governance, security, privacy, quality, and compliance frameworks across enterprise data systems.
• Lead technical design reviews, evaluate emerging technologies, and recommend solutions aligned with business objectives.
• Mentor and guide data engineers, promote engineering excellence, and support technical capability development.
• Collaborate with data scientists, analytics teams, architects, product managers, and business stakeholders to deliver data-driven solutions.
• Optimize data platform performance, scalability, availability, and cloud infrastructure costs.
• Implement automated testing, CI/CD, monitoring, orchestration, and infrastructure-as-code practices.
• Troubleshoot complex production issues and drive continuous improvement in data reliability and operational efficiency.

Preferred candidate profile

• Bachelors degree in Computer Science, Information Technology, Software Engineering, or a related technical discipline.
• 5 to 10 years of experience in data engineering, big data platforms, or distributed data processing, with demonstrated technical leadership and architecture ownership.
• Strong proficiency in Python and SQL, with experience in Scala or Java.
• Extensive hands-on experience with Apache Spark, PySpark, Kafka, Hadoop, and large-scale data processing frameworks.
• Proven expertise in designing enterprise-grade data pipelines, data models, data warehouses, data lakes, and lakehouse architectures.
• Strong experience with cloud data platforms and services across AWS, Microsoft Azure, or Google Cloud Platform.
• Familiarity with Databricks, Snowflake, Delta Lake, Apache Airflow, and modern data orchestration frameworks.
• Strong understanding of distributed systems, system design, data governance, security, and data quality management.
• Experience with CI/CD, automated testing, infrastructure as code, observability, and production deployment practices.
• Demonstrated ability to lead technical initiatives, establish engineering standards, mentor team members, and influence architectural decisions.
• Excellent analytical, problem-solving, communication, and stakeholder management skills.
• Relevant cloud, data engineering, or architecture certifications are advantageous; a Master’s degree is preferred but not mandatory.