Senior Databricks Data Engineer - PySpark - Bangalore / Pune

Hiring for: A technology company delivering enterprise data engineering and analytics solutions, with a strong focus on modern cloud data platforms and Databricks.

Role: Senior Databricks Data Engineer - PySpark

Positions: 10

Experience: 4 to 7 years

Location(s): Pune / Bangalore (Preferred) / Remote

Type: On-site, Remote / Permanent

Salary: Up to INR 25 LPA

Notice Period: Immediate to 30 days

Must Have : Databricks Development, Pyspark, Python, SQL, Unity Catalogs,

Job Summary

We are seeking an experienced PySpark ETL Developer with 4+ years of experience in designing, developing, and optimizing enterprise ETL pipelines using PySpark. The ideal candidate should have strong expertise in Python, Apache Spark, Databricks, and Snowflake, along with hands-on experience in processing large-scale data and building scalable data engineering solutions. You will work closely with business stakeholders and cross-functional teams to develop high-performance data pipelines that support business intelligence and analytics initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL.
• Collaborate with stakeholders to gather business requirements and translate them into efficient data engineering solutions.
• Extract data from multiple sources, including databases, APIs, data lakes, files, and streaming platforms.
• Transform, cleanse, and validate data using PySpark to ensure high data quality and consistency.
• Develop and optimize Spark jobs for performance, scalability, and efficient resource utilization.
• Build and maintain batch and streaming data pipelines to support real-time and near real-time processing.
• Load transformed data into data lakes, data warehouses, and analytical platforms.
• Implement robust error handling, logging, monitoring, and troubleshooting mechanisms for ETL workflows.
• Document ETL processes, data lineage, transformation logic, and technical specifications.
• Develop unit, integration, and performance tests to ensure reliable and accurate data processing.
• Collaborate with cross-functional teams to deliver scalable, secure, and high-quality data solutions.

Required Skills

• 4+ years of experience as a Data Engineer or ETL Developer.
• Strong hands-on experience with PySpark, Apache Spark, Spark SQL, and Python.
• Expertise in Databricks.
• Strong understanding of ETL design, data transformation, and data integration.
• Experience with Data Lakes, Data Warehouses, and Big Data technologies.
• Knowledge of distributed computing, parallel processing, and data partitioning concepts.
• Strong SQL skills with experience in performance tuning and query optimization.
• Experience working with batch and streaming data processing.
• Excellent analytical, debugging, and problem-solving skills.
• Strong verbal and written communication skills.

Preferred Skills

• Experience with cloud platforms such as AWS or Azure.
• Knowledge of Hadoop ecosystem and modern data engineering technologies.
• Experience with API integration and data ingestion from multiple data sources.
• Familiarity with CI/CD pipelines and Agile development methodologies.
• Exposure to enterprise-scale data engineering projects.