DevOps Engineer (Senior)

We are seeking an experienced Databricks Infrastructure and DevOps Engineer to support the MSF DataSphere platform, a comprehensive data lakehouse solution built on AWS Government Commercial Cloud (GCC) environment. The successful candidate will be responsible for managing and optimising the Databricks infrastructure, implementing robust CI/CD pipelines using government-approved tools, and ensuring compliance with stringent security and governance requirements.

Key Responsibilities

Infrastructure Management:

• Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment
• Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)
• Configure and optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks
• Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management:

• Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem
• Configure GitLab for version control and automated deployment workflows
• Implement code quality gates using SonarQube for static code analysis
• Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment
• Oversee dependency scanning and management using Nexus IQ
• Coordinate with CloudSCAPE for security compliance scanning and remediation

Security and Compliance:

• Implement and maintain Automated Baseline Log Review (ABLR) systems
• Coordinate with GovTech Cyber Security Operations Centre (GCSOC) for security monitoring
• Manage Central Accounts Management (CAM) integration for user access control
• Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT) findings.
• Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements
• Implement and maintain Vulnerability Management System (VMS)

Platform Operations:

• Monitor system performance and implement optimization strategies
• Manage secrets and encryption key rotation for data protection
• Configure email notifications for data processing status and error alerts
• Implement and maintain data retention policies
• Support SaaS whitelisting processes and approvals
• Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support:

• Work closely with Data Engineers, Data Analysts and security technical team.
• Support multiple datamarts
• Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines
• Collaborate with various government divisions and stakeholders
• Support training environment setup for user onboarding and skill development

Required Qualifications

Technical Skills:

• 3+ years of experience with Databricks platform administration and optimization
• Strong expertise in AWS services, particularly in GCC environment
• Proficiency in Infrastructure-as-Code tools, preferably Terraform
• Experience with GitLab CI/CD pipeline development and management
• Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)
• Understanding of government security frameworks and compliance requirements
• Experience with containerization and orchestration technologies
• Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance:

• Experience with government security clearance processes
• Knowledge of data classification and handling procedures (Sensitive High)
• Understanding of encryption mechanisms and key management
• Familiarity with vulnerability management and remediation processes
• Experience with security monitoring and incident response procedures

Certifications (Preferred):

• Databricks Certified Associate or higher
• AWS Certified DevOps Engineer

Required Experience

• Minimum 4-5 years of DevOps/Infrastructure engineering experience
• Experience working in government environments, esp in GCC.
• Experience in agile development methodologies

Personal Attributes

• Good communication and collaboration abilities
• Attention to detail and commitment to security best practices
• Strong documentation and knowledge sharing practices
• Proactive approach to system monitoring and maintenance