EY - GDS Consulting - AI And DATA -AWS Data Engineer - Manager

Job Summary

Build and operate AWS-native Data & AI platforms with strong data engineering and platform engineering ownership. The role focuses on AWS Glue, EMR, S3, Athena, Redshift, MWAA, Step Functions, Lambda, EventBridge, SageMaker, APIs, enterprise service integration, Git-based delivery, Data SRE, data security, Immuta/Lake Formation governed access, and AI/agentic operations for production-grade data and AI workloads.

Role AI Data Platform Engineer - AWS Experience Guide

8-11 years

Primary Skill Area

AWS Data Platforms, Glue, EMR, SageMaker, S3, Redshift, Event-Driven Data & Agentic Operations

The opportunity Your key responsibilities AWS Data Engineering

• Design and build production-grade AWS data pipelines using Amazon S3, AWS Glue, PySpark, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, CloudWatch, IAM, and KMS.
• Develop reusable ingestion frameworks supporting batch, streaming, event-driven, CDC, API-based, file-based, database, and third-party service integration patterns.
• Build curated raw, standardised, trusted, and consumption layers using scalable lakehouse design patterns, partitioning, metadata management, and file-format optimisation.
• Optimise Spark/Glue/EMR workloads for performance, cost efficiency, scalability, and operational stability.

AWS Platform Engineering

• Create reusable AWS platform accelerators for onboarding, pipeline templates, orchestration, monitoring, reconciliation, deployment, logging, and support runbooks.
• Implement Git connectivity, branching strategy, pull requests, code reviews, CI/CD, Infrastructure as Code, controlled releases, and environment promotion.
• Integrate AWS data platforms with enterprise APIs, source applications, messaging/event services, governance tools, security platforms, and downstream analytics consumers.
• Partner with infrastructure, IAM, network, DBA, application, and support teams to resolve connectivity, access, deployment, and production issues.
• SageMaker, AI Integration & Agentic Enablement
• Integrate AWS data platforms with Amazon SageMaker for data preparation, feature engineering, model training, deployment, MLOps workflows, and inference-ready data products.
• Support SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, Bedrock where relevant, vector stores, semantic search, and RAG-ready data products.
• Apply AI-assisted and agentic operations for anomaly detection, schema drift detection, failed-job diagnosis, data quality recommendations, documentation generation, and incident summarisation.

Governance, Security & Data SRE

• Implement AWS data security controls including IAM least privilege, KMS encryption, Secrets Manager, VPC endpoints, Lake Formation, Glue Data Catalog, Macie, CloudTrail, and audit-ready access patterns.
• Integrate with Immuta, Microsoft Purview, Collibra, enterprise IAM, monitoring platforms, data quality tools, and downstream analytics/AI consumers.
• Own Data SRE responsibilities including CloudWatch observability, SLA/SLO tracking, alerting, retry logic, restartability, root-cause analysis, incident response, and production reliability management.

Skills and attributes for success

• Core platform: AWS Glue, Amazon S3, Athena, Redshift, EMR, MWAA/Airflow, Step Functions, Lambda, EventBridge, CloudWatch.
• AI and GenAI: Amazon SageMaker, SageMaker Pipelines, Feature Store, Model Registry, Model Monitor, Bedrock optional, RAG-ready data products, Agentic AI.
• Engineering: Python, PySpark, SQL, APIs, Git, CI/CD, Shell scripting, unit testing, integration testing, data pipeline testing.
• Cloud and DevOps: Terraform/OpenTofu, CloudFormation, GitHub Actions, Azure DevOps, Jenkins, Docker, Kubernetes/EKS, policy-as-code.
• Governance and reliability: IAM, KMS, Lake Formation, Glue Data Catalog, Macie, Immuta, Purview, CloudTrail, data quality, observability, Data SRE, FinOps.

To qualify for the role, you must have

• 8-11 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
• Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
• Preferred certifications aligned to the relevant cloud/platform stack, data engineering, DevOps, security, governance.