DataBricks - Lead Programmer Analyst

Job Description: DataBricks - Lead Programmer Analyst

We are looking for a highly self-motivated individual with DataBricks development as a Lead Programmer Analyst:

Required Skills & Qualifications:

• • Experience should have 5 to 7 Years of Data Engineering.
  • Expert-level Apache Spark skills using PySpark (Scala a plus).
  • Strong proficiency in SQL for data transformation and performance tuning.
  • Solid experience with Delta Lake and the Lakehouse/medallion architecture.
  • Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its data services.
  • Experience with workflow orchestration (Databricks Workflows, Delta Live Tables, or Airflow).
  • Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
  • Experience with Python for data engineering and automation.
  • Familiarity with CI/CD, Git, and DevOps practices for data.
  • Good understanding of data governance, security, and Unity Catalog.
  • Strong problem-solving and communication skills.
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Good To Have:

• • Databricks Certified Data Engineer Associate or Professional certification.
  • Experience with streaming technologies (Structured Streaming, Kafka, Event Hubs, Pub/Sub).
  • Exposure to machine learning workflows and MLflow.
  • Experience with Infrastructure-as-Code (Terraform) and cloud cost optimization.
  • Experience working in Agile delivery environments.
  • Experience in data warehouse design and maintenance
  • Experience in agile development processes using Jira and Confluence.
  • Experience in cross-functional teams.

Key Responsibilities:

• • Design, develop, and maintain scalable ETL/ELT data pipelines on the Databricks Lakehouse Platform.
  • Build and optimize Apache Spark (PySpark/Scala) jobs for batch and streaming data processing.
  • Implement and manage Delta Lake tables, including partitioning, schema evolution, and performance tuning (Z-ordering, caching, file compaction).
  • Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and job scheduling tools such as Airflow.
  • Implement the medallion architecture (bronze, silver, gold layers) for reliable, governed data.
  • Integrate Databricks with cloud data services (AWS, Azure, or GCP) and source systems.
  • Apply data governance and security using Unity Catalog, access controls, and lineage.
  • Optimize cluster configuration, cost, and performance across workloads.
  • Collaborate with data analysts, data scientists, and business stakeholders to deliver trusted datasets.
  • Ensure data quality, testing, monitoring, and observability across pipelines.
  • Contribute to CI/CD practices for data engineering (version control, automated deployment).