PySpark Databricks Engineer

Educational Requirements

• BSc
• BTech
• BCA
• MTech
• MCA
• MSc

Service Line

Data Analytics Unit

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySpark and Databricks.
• Build scalable data processing solutions for large datasets.
• Develop and optimize Spark jobs for performance and efficiency.
• Implement data ingestion pipelines from various structured and unstructured data sources.
• Work with Delta Lake, Data Lake, and cloud storage solutions.
• Collaborate with business stakeholders, data analysts, and architects to understand data requirements.
• Ensure data quality, governance, and security standards are followed.
• Troubleshoot performance issues and optimize data workflows.
• Participate in code reviews and follow DevOps best practices.

Required Skills

• Strong experience with PySpark and Apache Spark.
• Hands-on experience with Databricks.
• Proficiency in Python programming.
• Experience in building ETL pipelines and data transformations.
• Strong SQL knowledge and query optimization skills.
• Experience with Azure Data Factory (ADF) / AWS Glue or similar tools.
• Knowledge of Delta Lake, Medallion Architecture, and Data Lakes.
• Experience with Git, CI/CD pipelines, and Agile methodologies.

Preferred Skills

• Technology->Data Engineering->Databricks
• Technology->Big Data - Data Processing->PySpark