Lead Data Engineer

Duration - 6 months contract to hire

Location - 3 days hybrid - local candidates to Dallas-Fort worth region or Chicago, IL

Rate - 70-80/hour W2 / conversion salary of 140-160K

no sponsorship or visa transfer provided. Candidate should be able to work without sponsorship in US.

Lead Data Engineer

About the Role

We are seeking a highly skilled Lead Data Engineer to serve as the technical leader for our Internal Analytics Platform , the organization's enterprise-wide source of truth for analytics, reporting, and data-driven decision-making.

This role combines deep hands-on technical expertise with leadership responsibilities across data engineering, analytics, and platform development. You will drive the design, architecture, and delivery of scalable data solutions leveraging Azure, Databricks, Azure Data Factory, SQL Server, Delta Lake, and modern DevOps practices . As a key technical leader, you will establish engineering standards, mentor team members, and ensure the platform remains reliable, performant, and scalable as business needs evolve.

Key Responsibilities

• Lead the design, development, enhancement, and support of enterprise-scale data and analytics solutions.
• Act as the technical subject matter expert across Azure data services, Databricks, SQL, ETL/ELT pipelines, data modeling, and analytics technologies.
• Design and implement scalable data ingestion, transformation, and curation frameworks supporting enterprise reporting and self-service analytics.
• Architect and optimize modern data lakehouse solutions using Delta Lake and Medallion Architecture principles.
• Develop and maintain high-performance data pipelines capable of processing large-scale datasets.
• Partner with business stakeholders, product owners, and engineering teams to gather requirements and translate them into robust technical solutions.
• Lead technical design reviews, code reviews, and architectural discussions while promoting engineering best practices.
• Drive adoption of CI/CD, Infrastructure as Code (IaC), and automated deployment processes to improve efficiency and reliability.
• Mentor and coach engineers through technical guidance, pair programming, and knowledge-sharing initiatives.
• Collaborate across teams to ensure data quality, governance, security, and platform performance.
• Participate in Agile ceremonies and contribute to successful sprint planning, execution, and delivery.

Required Qualifications

• 10+ years of experience in Data Engineering, Data Warehousing, or related fields, including demonstrated technical leadership experience.
• Strong expertise within the Azure ecosystem, including:
• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure SQL Database
• Azure Key Vault
• Service Principals and Managed Identities
• Deep hands-on experience with Databricks, including notebooks, optimization techniques, Delta Tables, UDFs, and workflow orchestration.
• Extensive experience designing and building ETL/ELT solutions using Azure Data Factory, SSIS, or similar technologies.
• Strong programming skills in SQL, Python, PySpark, and T-SQL.
• Experience integrating and ingesting data from APIs, external applications, and third-party source systems.
• Strong understanding of data modeling concepts, including dimensional and relational modeling techniques.
• Expertise designing and managing Data Lake and Delta Lake architectures.
• Strong knowledge of Medallion Architecture and modern lakehouse design patterns.
• Experience processing and optimizing large-scale datasets consisting of millions of records.
• Hands-on experience implementing CI/CD pipelines and modern DevOps practices.
• Experience with Infrastructure as Code tools such as Terraform or Kubernetes.
• Strong communication skills with the ability to effectively collaborate with both technical and non-technical stakeholders.
• Experience working within Agile software development environments.

Preferred Qualifications

• Experience with big data technologies such as Spark, Kafka, or Hadoop.
• Exposure to reporting and visualization tools including Power BI or Tableau.
• Knowledge of .NET technologies and application integration patterns.
• Experience implementing data governance, lineage, and data quality frameworks.
• Familiarity with cloud security best practices and enterprise data management standards.

#tech