Lead Cloud Engineer

Role:

• Design, develop, and support large-scale cloud-native data solutions on AWS and Databricks.

• Build and maintain batch, streaming, and real-time data pipelines supporting analytics, reporting, machine learning, and AI workloads.

• Engineer scalable distributed data processing systems using Apache Spark and related Big Data technologies.

• Design and implement modern Lakehouse architectures leveraging Databricks, Delta Lake, and Apache Iceberg.

• Lead modernization initiatives migrating data workloads from Hadoop-based platforms to cloud-native architectures.

• Develop reusable frameworks and engineering patterns that improve scalability, reliability, and operational efficiency.

• Partner with Architects, Product Managers, Data Scientists, and Software Engineers to define and implement cloud data solutions.

• Evaluate technologies and approaches for data ingestion, transformation, storage, governance, and consumption.

• Implement best practices around security, governance, performance optimization, observability, and cost management.

• Drive automation using Infrastructure-as-Code, CI/CD pipelines, and modern software engineering practices.

• Contribute to the evolution of Marketing Services' cloud and data strategy while enabling future AI and machine learning capabilities.

All About You:

• Strong hands-on experience with Big Data technologies including Hadoop, HDFS, Hive, Spark, and related distributed computing frameworks.

• Deep expertise in cloud data engineering on AWS, including storage, compute, networking, security, and scalable data architectures.

• Experience designing and developing modern data solutions using Databricks, Spark, Delta Lake, and cloud-native technologies.

• Proficiency in Python, Java, or Scala, building large-scale distributed data processing applications.

• Experience building robust batch and streaming pipelines supporting high-volume and high-velocity workloads.

• Understanding of modern Lakehouse architecture principles and open-table formats such as Apache Iceberg and Delta Lake.

• Skilled in data modelling, schema design, partitioning strategies, query optimization, and performance tuning.

• Familiarity with enterprise-scale data governance, security, privacy, and compliance, working with structured and unstructured datasets at scale.

• Experience with Infrastructure-as-Code (e.g., Terraform), modern CI/CD, and cloud-native monitoring and observability practices.

• Strong analytical and problem-solving skills, a passion for automation, and the ability to assess emerging technologies and recommend scalable, cost-effective solutions.

• Excellent communication and collaboration across technical and business teams; comfortable in Agile/Scrum environments managing multiple priorities.

• BS/MS degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Qualifications:

• Experience with Databricks on AWS in enterprise production environments.

• Experience migrating workloads from Hadoop or Cloudera ecosystems to modern cloud architectures.

• Hands-on experience with Apache Kafka, Flink, or other event-streaming technologies.

• Experience supporting Machine Learning, MLOps, or AI-driven data platforms.

• AWS and/or Databricks certifications.

• Familiarity with FinOps and cloud cost optimization practices.

• Experience working with petabyte-scale data environments.