Senior Data Engineer � Databricks/Pyspark

Senior Data Engineer - Databricks (PySpark) We are seeking a Databricks Consultant with strong expertise in PySpark and Python to design, develop, and optimize large-scale data processing solutions on cloud-based data platforms.

The ideal candidate will build scalable data pipelines, ETL workflows, and analytics solutions using Databricks and Apache Spark, supporting enterprise data platforms in IQVIA.

Key Responsibilities

• Data Engineering & Databricks Development

• Design and develop pipelines using Databricks and PySpark

• Build and maintain distributed data processing workflows using Spark

• Develop reusable data transformation frameworks and components

• Work with Delta Lake for optimized data storage and versioning

• PySpark & Python Development

• Implement Spark jobs using PySpark for large-scale data processing

• Develop Python-based data processing, automation, and orchestration scripts

• Optimize Spark transformations (joins, aggregations, partitioning)

• Data Pipeline & Architecture

• Build and optimize:

• Data pipelines

• Data ingestion frameworks

• Data lake architectures

• Ensure high scalability, fault tolerance, and performance

• Performance Tuning & Optimization

• Tune Spark jobs for:

• Performance

• Cost efficiency

• Resource utilization

• Identify bottlenecks in data pipelines and optimize workloads

• Monitor cluster and workload performance

• Integration & Cloud Enablement

• Integrate Databricks pipelines with:

• Azure Data Factory / AWS Glue

• Data lakes, APIs, databases

• Work with cloud platforms:

• Azure / AWS / GCP

• Best Practices & Governance

• Implement best practices for:

• Logging

• Monitoring

• Error handling

• Ensure data quality, security, and governance standards

• Follow CI/CD and DevOps practices using Git and pipelines

• Collaboration & Delivery

• Collaborate with:

• Data engineers and architects

• Product and business teams

• Translate requirements into technical solutions and pipelines

• Participate in Agile (Scrum) delivery model

Required Skills & Experience Core Skills

• Strong experience in Databricks and Apache Spark
• Hands-on expertise in PySpark programming
• Strong proficiency in Python for data engineering
• Experience in ETL/ELT pipeline development

Technical Skills

• PySpark:

• Data Frames, Spark SQL, transformations

• Python:

• Data processing, scripting, automation

• SQL:

• Query optimization and data modeling

Data & Platform Skills

• Data warehousing and data modeling concepts

• Experience with:

• Delta Lake

• Distributed data systems

• Handling structured and semi-structured data

Cloud & Tools

• Experience with:

• Azure Databricks / AWS Databricks

• Azure Data Factory / AWS Glue

• CI/CD & DevOps tools (Git, pipelines)

• Exposure to monitoring and logging tools

Preferred Qualifications

• Databricks or Spark-related certifications

• Experience with:

• Data pipeline orchestration (Airflow, Databricks Workflows)

• Large-scale enterprise data platforms

• Knowledge of:

• Streaming (Spark Structured Streaming)

• ML pipelines (optional)

Soft Skills

• Strong analytical and problem-solving skills
• Ability to work in cross-functional teams
• Good communication with stakeholders
• Agile mindset and delivery orientation

Typical Experience Range

• 4-8 years in data engineering
• Minimum 2-3 years in Databricks / PySpark

Value Proposition of Role

• Build scalable big data pipelines on cloud-native platforms
• Enable high-performance analytics and data-driven decision making