Lead Data Engineer

Senior / Lead Data Engineer

About the Role

We are looking for a highly experienced Senior/Lead Data Engineer to help modernize and scale our data platform. This person will play a key role in the continued migration to Snowflake and will be responsible for designing, building, and optimizing scalable data solutions and production-grade pipelines.

This is a highly technical, hands-on role. We are looking for someone who can operate as a Lead Engineer — someone who can take ownership of complex data engineering initiatives, make sound architecture decisions, solve technical problems independently, and provide technical direction to other engineers.

The ideal candidate has hands-on experience with Snowflake migrations, Airflow, Astronomer, Python, SQL, Azure, and modern ETL/ELT architecture .

What You’ll Do

Snowflake & Data Platform Modernization

• Lead and contribute to the migration of existing data workloads and platforms into Snowflake .
• Design and implement scalable Snowflake architectures, including databases, schemas, tables, views, warehouses, security, and data models.
• Evaluate existing data environments and determine how workloads should be redesigned or optimized within Snowflake.
• Develop strategies to improve Snowflake performance, scalability, reliability, and cost efficiency.
• Troubleshoot complex Snowflake issues and identify opportunities to improve the overall data platform.
• Help establish technical standards and best practices around Snowflake development and architecture.

Data Engineering & Pipeline Development

• Design, build, and maintain reliable ETL/ELT pipelines supporting analytics, reporting, and operational data needs.
• Develop production-grade data pipelines using Python and SQL .
• Build and manage batch and event-driven data workflows.
• Integrate data from internal databases, APIs, SaaS platforms, and other external sources.
• Develop solutions that are scalable, maintainable, and designed for long-term reliability.
• Identify and resolve pipeline performance, reliability, and data quality issues.

Airflow & Astronomer

• Design, develop, and maintain data orchestration workflows using Apache Airflow .
• Work within Astronomer to deploy, manage, monitor, and optimize Airflow environments.
• Build and maintain DAGs supporting complex data workflows and dependencies.
• Establish best practices around workflow scheduling, monitoring, retries, failure handling, and operational support.
• Troubleshoot orchestration issues and improve pipeline reliability and performance.

Technical Leadership

• Serve as a technical leader within the data engineering team without necessarily having direct people management responsibilities.
• Lead complex technical projects from architecture and design through implementation and production support.
• Provide technical direction and mentorship to other data engineers.
• Conduct code and architecture reviews and establish engineering best practices.
• Help evaluate new technologies and determine when existing tools or platforms are the right solution versus custom development.
• Take ownership of complex problems and drive them through to resolution.

Data Architecture & Modeling

• Design scalable data architectures and analytics-focused data models.
• Develop schemas, transformations, and structures that support reporting, analytics, and business applications.
• Apply strong data modeling principles across Snowflake and cloud data environments.
• Design solutions with scalability, performance, security, maintainability, and cost in mind.
• Help evolve the organization's overall data architecture as the platform grows.

Azure & Cloud Data Engineering

• Develop and support data solutions within Microsoft Azure .
• Work with Azure data services and cloud-based infrastructure to support ingestion, transformation, storage, and analytics.
• Design integrations between Azure, Snowflake, databases, APIs, and third-party platforms.
• Help evaluate cloud architecture decisions and recommend solutions based on business and technical requirements.

Data Quality & Reliability

• Build data quality checks and monitoring into production data pipelines.
• Identify and resolve issues affecting data accuracy, completeness, performance, and reliability.
• Establish monitoring and alerting practices for critical data workflows.
• Help improve documentation, lineage, governance, and overall maintainability of the data environment.

Business & Technical Partnership

• Partner with engineering, product, analytics, and business stakeholders to understand requirements and translate them into technical solutions.
• Communicate complex technical concepts and architecture decisions clearly to both technical and non-technical audiences.
• Serve as a technical resource for data-related initiatives across the organization.
• Balance immediate business needs with long-term platform scalability and maintainability.

What We’re Looking For

• 8+ years of professional data engineering experience , with demonstrated experience operating at a Senior or Lead Engineer level.
• Strong, hands-on experience with Snowflake .
• Proven experience migrating data platforms, workloads, or pipelines to Snowflake .
• Strong experience with Apache Airflow .
• Experience with Astronomer is highly preferred.
• Strong Python and SQL skills.
• Extensive experience designing and building ETL/ELT pipelines .
• Strong understanding of data architecture, data modeling, and cloud data platforms .
• Hands-on experience with Azure cloud data services .
• Experience working with APIs, databases, SaaS applications, and other data sources.
• Strong understanding of data quality, monitoring, reliability, and production support.
• Experience with performance tuning and Snowflake cost optimization .
• Experience providing technical leadership, mentoring engineers, and driving complex technical initiatives.
• Strong problem-solving skills and the ability to independently work through complex technical challenges.
• Excellent communication skills and the ability to work effectively with both technical and business stakeholders.

Nice to Have

• Experience migrating from Databricks to Snowflake .
• Experience with dbt, Fivetran, or similar data engineering tools.
• Experience with Snowflake performance tuning and optimization.
• Experience with CI/CD and infrastructure-as-code.
• Experience with cloud modernization initiatives.
• Experience designing highly available and fault-tolerant data platforms.
• Experience working in a fast-paced environment where engineers are expected to wear multiple hats.

What Will Make Someone Successful

The strongest candidate will be someone who can go deep technically . We are not looking for someone who simply has Snowflake, Airflow, or Azure listed on their resume.

The ideal person can walk through a migration or modernization project from beginning to end and explain:

• What the environment looked like before the migration
• What they were responsible for
• How they designed the new architecture
• Why they selected specific technologies and services
• How they approached the Snowflake migration
• How Airflow/Astronomer was used to orchestrate the workflows
• What technical challenges they encountered
• How they solved those challenges
• How they addressed performance, reliability, security, and cost
• What they would do differently today

This is a hands-on Lead Engineer role. The ability to explain the “why” behind technical and architectural decisions is just as important as knowing the technology itself.