Senior Lead Data Engineer

Integrant is looking for game changers to join our team as a "Senior Lead Data Engineer". This is a senior, hands-on leader who designs data and AI solutions end to end and builds them. You'll be responsible for designing solutions for the ingestion, storage, processing, transformation, enrichment, and presentation of data for analytical consumption, and for personally implementing the critical parts of those solutions.

Engage clients at multiple levels to elicit requirements, assess their current analytical challenges, and turn them into technical proposals and solution designs.

Select and customize analytical architectures (Data Warehouses, Data Lakes, Data Lakehouses, Data Fabrics, Data Meshes, etc.) and technically justify how they meet each client's needs.

Design and build agentic and LLM-powered solutions on top of modern data platforms.

Lead hands-on implementation of data pipelines, warehouses, and lakes, from design through performance tuning and production support.

Build data strategies, and coach clients and internal teams in the latest architectures and methodologies (DataOps, MLOps, etc.), including mentoring engineers on the team.

Requirements

Architecture & Leadership

10-12 years of experience in data engineering, including hands-on delivery at a senior level.

Bachelor's degree in Computer Science, Computer Engineering or other quantitative field. Post Graduate degrees preferable.

The ability to interact with stakeholders at multiple levels within a given organization, understand current analytical challenges and gather requirements.

Excellent written and spoken English, with the ability to present solutions to technical and business audiences.

Proven experience writing technical proposals and solution designs for clients.

A deep understanding of the evolution of analytical architectures from reporting databases to data warehouses, data lakes, data lakehouses, data fabrics, data meshes and beyond, with the ability to technically justify how a proposed architecture meets a client's analytical needs.

Experience mentoring and guiding data engineers.

Hands-on Data Engineering

Expert-level SQL and strong programming skills in Python (or Scala/Java) for data processing.

Extensive experience with at least one enterprise data platform (e.g. Microsoft SQL Server stack, Oracle, Teradata).

Hands-on experience implementing data warehouses both on-premises and in the cloud (e.g. SQL Server, Oracle, Azure Synapse/Fabric, Amazon Redshift, Google BigQuery).

Strong dimensional modeling: Dimensions, Facts, Slowly Changing Dimensions, Outriggers, Role-Playing, Junk, Degenerate and Multi-valued Dimensions; Transactional, Periodic Snapshot and Accumulating Snapshot Facts.

Hands-on experience implementing common ETL/ELT patterns using tools such as SSIS, Informatica, Talend, Azure Data Factory, AWS Glue, or code-based frameworks (e.g. Spark).

Hands-on experience implementing data quality, testing and observability for data pipelines (e.g. dbt tests, Great Expectations, Monte Carlo, Soda).

Experience building semantic layers and OLAP models (e.g. SSAS, Power BI semantic models, LookML, dbt semantic layer).

Hands-on experience with workflow orchestration (e.g. Apache Airflow, Azure Data Factory, AWS Step Functions, Databricks Workflows).

Experience setting up data lakes and lakehouses on cloud object storage (e.g. ADLS Gen2, Amazon S3, Google Cloud Storage) using open table formats (e.g. Delta Lake, Apache Iceberg, Apache Hudi).

Experience optimizing query and workload performance and setting up high availability configurations.

Experience administering and managing analytical solutions both on-premises and in the cloud.

Experience building streaming pipelines (e.g. Kafka, Kinesis, Event Hubs, Google Pub/Sub, Spark Structured Streaming).

Experience with query federation and data virtualization (e.g. Trino/Starburst, Amazon Athena, PolyBase, BigQuery federated queries).

Knowledge of NoSQL databases (e.g. MongoDB, Cosmos DB, DynamoDB, Cassandra).

Knowledge of BI tools (e.g. Power BI, Tableau, Looker, Qlik).

Cloud, Platforms & Governance

Experience with multiple cloud data platforms (e.g. Microsoft Azure, AWS, Google Cloud).

Hands-on experience with Snowflake or Databricks.

Experience estimating, monitoring and optimizing cloud data platform costs (e.g. warehouse sizing, compute/storage trade-offs, Snowflake credits, Databricks DBUs), and reflecting them in client proposals.

Experience implementing data governance and security: cataloging, lineage, access control and secrets management (e.g. Unity Catalog, Microsoft Purview, AWS Lake Formation, Collibra, cloud IAM, HashiCorp Vault).

Experience with CI/CD and version control for data solutions (e.g. GitHub Actions, GitLab CI, Azure DevOps).

AI & Agentic

Hands-on exposure to LLM and agentic solutions, in PoC or production (e.g. LangGraph/AutoGen/CrewAI/LlamaIndex/n8n).

Experience with AI-assisted development (e.g. Cursor/Codex/Claude Code).

Familiarity with DataOps.

Nice to Have

Familiarity with Agent Protocols (e.g. MCP).

Familiarity with transformation frameworks (e.g. dbt).

Familiarity with MLOps and ML platforms (e.g. MLflow, Azure ML, Amazon SageMaker, Vertex AI).

Familiarity with cloud AI services (e.g. Azure AI services, Amazon Bedrock, Vertex AI).

Familiarity with containerization tools and frameworks (Docker, Kubernetes, etc.).

Familiarity with the big data ecosystem, including Spark and Hive (e.g. on Databricks, Amazon EMR, Dataproc, HDInsight).

Familiarity with serverless functions for lightweight transformations (e.g. AWS Lambda, Azure Functions, Google Cloud Functions).