Data Engineer (PySpark & Generative AI)

Role Overview

Build scalable data pipelines and platforms with Python, PySpark, and cloud data services to support analytics, reporting, and AI/ML initiatives.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines for batch and real-time data.
• Ingest data from structured and unstructured sources into cloud data lakes and warehouses.
• Build data models and warehouses for analytics and reporting, and tune SQL for performance.
• Implement data validation, cleansing, transformation, and quality checks.
• Work with data scientists, analysts, and application teams, following Git, CI/CD, testing, and monitoring practices.

Must Have Skills

• 5 - 8 years of data engineering, with strong Python programming.
• Hands-on PySpark and Apache Spark on large-scale distributed datasets.
• Advanced SQL (query optimization and performance tuning), plus data warehousing and dimensional modeling.
• Strong experience with Generative AI and Large Language Models (LLMs).
• AWS, Azure, or GCP data services, with Git, CI/CD, and Agile/Scrum.

Nice to Have

• Apache Airflow or similar orchestration
• Kafka and real-time streaming
• Databricks, Delta Lake, Apache Iceberg, or Hudi
• Docker, Kubernetes, and Terraform
• Data governance and metadata management