Python+ETL+Pyspark Professional

Educational Requirements

• Bachelor of Engineering
• BTech
• BSc
• BCA
• MSc
• MTech
• MCA

Service Line

Data Analytics Unit

Responsibilities

• ETL Development: Design, develop, and maintain ETL pipelines using Python and PySpark.
• Extract, transform, and load data from multiple structured and unstructured sources.
• Build reusable and scalable data processing frameworks.
• Ensure data quality, validation, and consistency.
• PySpark / Big Data Processing: Develop and optimize PySpark jobs for large-scale data processing.
• Work with Spark DataFrames and RDDs.
• Implement transformations, aggregations, and joins in Spark.
• Optimize jobs for performance and scalability.
• Python Development: Develop backend logic and data processing scripts using Python.
• Write modular, reusable, and efficient code.
• Integrate APIs and automate workflows.
• Data Management Integration: Work with data lakes and warehouses (S3, HDFS, Redshift, Hive).
• Handle file formats like Parquet, ORC, JSON, CSV.
• Perform data cleansing, enrichment, and transformation.
• Collaboration Support: Work with data engineers, analysts, and business stakeholders.
• Debug and troubleshoot ETL/data pipeline issues.
• Participate in Agile/Scrum ceremonies.
• Maintain documentation and coding standards.

Additional Responsibilities

• Core Skills: 25 years of experience in Python and ETL development.
• Hands-on experience with PySpark (mandatory).
• Strong understanding of data processing and pipelines.
• Solid knowledge of SQL and database concepts.
• Technical Skills: Experience with Apache Spark ecosystem.
• Good knowledge of data structures and algorithms (basic to intermediate).
• Familiarity with Big Data technologies (Hadoop, Hive).
• Experience with version control (Git).
• Understanding of REST APIs and integrations.

Technical and Professional Requirements

• Primary skills:Domain- >Finacle-Core-Functional- >Finacle-Core-WMS- >Grand Master,Technology- >ETL Data Quality- >ETL - Others,Technology- >Machine Learning- >Python

Preferred Skills

• Technology- >ETL Data Quality- >ETL - Others
• Technology- >AI-Data science- >PYTHON
• Technology- >Big Data - Data Processing- >PySpark