Data Engineer - Databricks / Spark / AI

Senior Data Engineer Databricks / Spark / AI

Location: San Francisco, CA
Work Model: Hybrid 2 Days/Week Onsite
Duration: 12+ Months
Employment Type: Contract
Experience: 5+ Years
Work Authorization: Visa-independent candidates required

Job Summary

We are seeking a highly skilled Senior Data Engineer with strong, recent hands-on experience in Databricks, Apache Spark, SQL, and Python.

This is primarily a Data Engineering role with additional exposure to AI and Generative AI capabilities within the Databricks ecosystem. The ideal candidate will have extensive experience designing and developing scalable data pipelines and data processing solutions using Databricks and Spark.

Candidates should also have hands-on experience with Databricks Genie and an understanding of how AI capabilities can be applied to enterprise data and analytics solutions.

Strong, recent Databricks experience is essential for this position.

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, SQL, and Python.
• Build and optimize high-volume ETL/ELT data pipelines and data-processing workflows.
• Develop complex SQL queries and transformations for large-scale datasets.
• Build distributed data-processing solutions using PySpark/Spark.
• Design reliable and scalable data architectures within the Databricks ecosystem.
• Work hands-on with Databricks Genie to enable AI-powered conversational data and analytics capabilities.
• Integrate AI/GenAI capabilities with enterprise data platforms and analytics workflows.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Perform data transformation, cleansing, validation, and quality checks.
• Troubleshoot performance issues across Spark jobs, SQL workloads, pipelines, and Databricks environments.
• Work with structured and semi-structured datasets from multiple enterprise sources.
• Collaborate with Data Engineering, Analytics, AI/ML, Product, and business teams.
• Translate business and analytical requirements into scalable data solutions.
• Participate in technical design, code reviews, testing, deployment, and production support.
• Maintain engineering standards, documentation, and data-development best practices.
• Participate in Agile/Scrum development activities using Jira and Confluence.

Must-Have Skills

Candidates should demonstr