Sr. DevOps Engineer

Responsibilities

• Infrastructure Automation

  • Utilize tools like Terraform to automate Databricks on AWS infrastructure provisioning and configuration management.
  • Maintain IaC (Infrastructure as Code) practices, ensuring repeatable and reliable deployments.
  • Ability to configure access controls in Databricks through Terraform.

• Cloud & Data Platform Integration

  • Manage and optimize Databricks environments for big data processing, ensuring high performance and cost-efficiency.
  • Work with Databricks SDK / Databricks CLI / DAB to automate cluster setup, job configurations, and library installations.
  • Cost/SLA monitoring and proactive intervention for cost control in Databricks.

• Source Control & Collaboration

  • Use Git to track changes, review code, and collaborate on shared repositories.
  • Enforce best practices for version control, branching, and merging strategies.

• Scripting & Automation

  • Write and maintain custom scripts (e.g., in Python) to automate repetitive data and DevOps tasks.
  • Develop CI/CD pipelines for continuous testing, integration, and deployment of data-related components.

• Monitoring & Troubleshooting

  • Implement robust logging and monitoring solutions to detect issues in GIT.
  • Investigate and resolve blocking issues, ensuring minimal downtime.

• Collaboration & Leadership

  • Collaborate with stakeholders across Engineering, Analytics, and Product teams to define requirements and deliver solutions.
  • Define the necessary branching strategies in Git for code deployment by multiple teams in parallel to the same environment.

• Qualification: At least 5 years of experience as a Devops engineer working with Data platforms.

• Experience:

  • Demonstrated experience in DevOps or DataOps roles.
  • Hands-on proficiency with Terraform for infrastructure automation.
  • Experience working as a Devops engineer with Databricks in AWS or a similar big data platform.
  • AWS access management.(Instance profiles, Service principals) all managed within Git.
  • Proven track record of using Git for source control in team environments.

• Technical Skills:

  • Scripting Languages: Python
  • CI/CD Tools: Github - Workflows, automation.
  • Cloud Platforms: Well versed with AWS provisioning, access management. High level understanding of Azure
  • Databricks : Comfort with automating Databricks cluster jobs, DLT pipelines notebooks, and workflows.

• Soft Skills:

  • Strong problem-solving and analytical capabilities.
  • Excellent communication and teamwork skills.
  • Ability to adapt to fast-paced environments and manage changing priorities.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.