Databricks Engineer with Rest API

Job Title: Senior Data Engineer (Databricks / Python)

Experience Level: 5 7 Years

Location: Hybrid - Bangalore, Hyderabad or Pune.

Cloud Platform: AWS

Role Overview

We are seeking an experienced Senior Data Engineer with 5 to 7 years of hands-on expertise in designing, building, and optimizing scalable data pipelines within the Databricks ecosystem. In this role, you will lead the integration of REST APIs using Databricks notebooks, enforce robust Medallion architecture practices, and drive business value using Delta Lake on AWS.

Key Responsibilities

• Design, architect, test, and maintain robust end-to-end data pipelines and automated workflows within Databricks.
• Lead the development of Databricks notebooks for complex REST API integrations to ingest and process data from external applications and services.
• Architect and enforce data processing layers following Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for high reliability and performance.
• Write production-grade, highly performant Python and SQL code for data ingestion, transformation, and validation.
• Optimize pipeline performance, monitor job executions, and ensure data security and governance across the AWS environment.
• Mentor junior team members and collaborate with data architects and business stakeholders on technical solutions.

Mandatory Qualifications

• Experience: 5 to 7 years of core experience in data engineering and pipeline development.
• Databricks Expertise: In-depth experience building, orchestrating, and troubleshooting enterprise data pipelines in Databricks.
• API Integration: Proven track record of designing REST API integration workflows directly within Databricks notebooks.
• Core Languages: Advanced proficiency in Python and SQL for data manipulation and analytics.
• Data Architecture: Solid understanding and practical implementation of Medallion Architecture and Delta Lake features (Time Travel, ACID transactions, schema enforcement).
• Cloud Platform: Hands-on experience operating and deploying data solutions in an AWS environment (e.g., S3, IAM, CloudWatch).

Nice-to-Have Skills

• Hands-on experience integrating with SaaS platforms (e.g., Salesforce, ServiceNow, Zendesk).
• Deep expertise in PySpark for large-scale distributed data processing and performance tuning.
• Familiarity with Databricks Lakeflow or automated orchestration tools for pipeline management.