SME - Python

SME - Python

Chennai, Tamil Nadu

Job Summary

Job Title: Senior Data Engineer ETL (Databricks, PySpark & Python) Job Summary We are seeking a highly skilled Senior Data Engineer with expertise in Databricks, PySpark, Python, and Big Data technologies. The ideal candidate will have hands-on experience building scalable data platforms, developing high-performance data pipelines, and implementing modern data architectures using Databricks Lakehouse. The role requires strong knowledge of Medallion Architecture, Unity Catalog, and enterprise data governance practices. ________________________________________ Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT solutions. • Build and optimize batch and streaming data processing solutions using PySpark and Databricks. • Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data transformation and consumption. • Configure and manage Unity Catalog for centralized governance, access control, and data lineage. • Develop robust data engineering solutions using Python and Spark frameworks. • Optimize Databricks workloads for performance, scalability, and cost efficiency. • Collaborate with data architects, analysts, data scientists, and business stakeholders to deliver high-quality data solutions. • Ensure data quality, security, governance, and compliance across enterprise data platforms. • Support Data Lake, Lakehouse, and Data Warehouse initiatives. • Troubleshoot and resolve complex data engineering and performance issues. Required Skills & Qualifications • Strong hands-on experience with Databricks and Databricks Lakehouse Platform. • Expert-level proficiency in Python programming. • Deep expertise in PySpark and distributed data processing. • Strong understanding of Big Data concepts and modern data architectures. • Good understanding and practical implementation experience of Medallion Architecture (Bronze, Silver, Gold layers). • Experience working with Unity Catalog for data governance, security, access management, and lineage tracking. • Strong knowledge of Apache Spark performance tuning and optimization. • Experience in designing and building scalable ETL/ELT pipelines. • Hands-on experience with Delta Lake and Lakehouse architecture. • Experience with cloud platforms such as Azure, AWS, or GCP. • Strong analytical, problem-solving, and communication skills.

Key Responsibilities

Job Title: Senior Data Engineer ETL (Databricks, PySpark & Python) Job Summary We are seeking a highly skilled Senior Data Engineer with expertise in Databricks, PySpark, Python, and Big Data technologies. The ideal candidate will have hands-on experience building scalable data platforms, developing high-performance data pipelines, and implementing modern data architectures using Databricks Lakehouse. The role requires strong knowledge of Medallion Architecture, Unity Catalog, and enterprise data governance practices. Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT solutions. • Build and optimize batch and streaming data processing solutions using PySpark and Databricks. • Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data transformation and consumption. • Configure and manage Unity Catalog for centralized governance, access control, and data lineage. • Develop robust data engineering solutions using Python and Spark frameworks. • Optimize Databricks workloads for performance, scalability, and cost efficiency. • Collaborate with data architects, analysts, data scientists, and business stakeholders to deliver high-quality data solutions. • Ensure data quality, security, governance, and compliance across enterprise data platforms. • Support Data Lake, Lakehouse, and Data Warehouse initiatives. • Troubleshoot and resolve complex data engineering and performance issues. ________________________________________ Required Skills & Qualifications • Strong hands-on experience with Databricks and Databricks Lakehouse Platform. • Expert-level proficiency in Python programming. • Deep expertise in PySpark and distributed data processing. • Strong understanding of Big Data concepts and modern data architectures. • Good understanding and practical implementation experience of Medallion Architecture (Bronze, Silver, Gold layers). • Experience working with Unity Catalog for data governance, security, access management, and lineage tracking. • Strong knowledge of Apache Spark performance tuning and optimization. • Experience in designing and building scalable ETL/ELT pipelines. • Hands-on experience with Delta Lake and Lakehouse architecture. • Experience with cloud platforms such as Azure, AWS, or GCP. • Strong analytical, problem-solving, and communication skills.

Skill Requirements

Job Title: Senior Data Engineer ETL (Databricks, PySpark & Python) Job Summary We are seeking a highly skilled Senior Data Engineer with expertise in Databricks, PySpark, Python, and Big Data technologies. The ideal candidate will have hands-on experience building scalable data platforms, developing high-performance data pipelines, and implementing modern data architectures using Databricks Lakehouse. The role requires strong knowledge of Medallion Architecture, Unity Catalog, and enterprise data governance practices. Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT solutions. • Build and optimize batch and streaming data processing solutions using PySpark and Databricks. • Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data transformation and consumption. • Configure and manage Unity Catalog for centralized governance, access control, and data lineage. • Develop robust data engineering solutions using Python and Spark frameworks. • Optimize Databricks workloads for performance, scalability, and cost efficiency. • Collaborate with data architects, analysts, data scientists, and business stakeholders to deliver high-quality data solutions. • Ensure data quality, security, governance, and compliance across enterprise data platforms. • Support Data Lake, Lakehouse, and Data Warehouse initiatives. • Troubleshoot and resolve complex data engineering and performance issues. Required Skills & Qualifications • Strong hands-on experience with Databricks and Databricks Lakehouse Platform. • Expert-level proficiency in Python programming. • Deep expertise in PySpark and distributed data processing. • Strong understanding of Big Data concepts and modern data architectures. • Good understanding and practical implementation experience of Medallion Architecture (Bronze, Silver, Gold layers). • Experience working with Unity Catalog for data governance, security, access management, and lineage tracking. • Strong knowledge of Apache Spark performance tuning and optimization. • Experience in designing and building scalable ETL/ELT pipelines. • Hands-on experience with Delta Lake and Lakehouse architecture. • Experience with cloud platforms such as Azure, AWS, or GCP. • Strong analytical, problem-solving, and communication skills.

Other Requirements

Job Title: Senior Data Engineer ETL (Databricks, PySpark & Python) Job Summary We are seeking a highly skilled Senior Data Engineer with expertise in Databricks, PySpark, Python, and Big Data technologies. The ideal candidate will have hands-on experience building scalable data platforms, developing high-performance data pipelines, and implementing modern data architectures using Databricks Lakehouse. The role requires strong knowledge of Medallion Architecture, Unity Catalog, and enterprise data governance practices. Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT solutions. • Build and optimize batch and streaming data processing solutions using PySpark and Databricks. • Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data transformation and consumption. • Configure and manage Unity Catalog for centralized governance, access control, and data lineage. • Develop robust data engineering solutions using Python and Spark frameworks. • Optimize Databricks workloads for performance, scalability, and cost efficiency. • Collaborate with data architects, analysts, data scientists, and business stakeholders to deliver high-quality data solutions. • Ensure data quality, security, governance, and compliance across enterprise data platforms. • Support Data Lake, Lakehouse, and Data Warehouse initiatives. • Troubleshoot and resolve complex data engineering and performance issues. Required Skills & Qualifications • Strong hands-on experience with Databricks and Databricks Lakehouse Platform. • Expert-level proficiency in Python programming. • Deep expertise in PySpark and distributed data processing. • Strong understanding of Big Data concepts and modern data architectures. • Good understanding and practical implementation experience of Medallion Architecture (Bronze, Silver, Gold layers). • Experience working with Unity Catalog for data governance, security, access management, and lineage tracking. • Strong knowledge of Apache Spark performance tuning and optimization. • Experience in designing and building scalable ETL/ELT pipelines. • Hands-on experience with Delta Lake and Lakehouse architecture. • Experience with cloud platforms such as Azure, AWS, or GCP. • Strong analytical, problem-solving, and communication skills.