pyspark

Job highlights

• 4+ years experience with PySpark and GCP (Airflow, Dataproc, BigQuery), advanced SQL and Python skills
• Design and configure data pipelines, write complex SQL queries, develop ETL processes, and deliver solutions to improve engineering efficiency

JOB DISCRIPTION

Experience: 4+Years

Location: Hyderabad/pune

Must have skill - Pyspark + GCP - The must-have skills are deep ETL tool experience, PySpark with GCP pipeline expertise, advanced SQL, Python.

Description:

• At least 3-5 years of experience in Pyspark with GCP (Airflow, Dataproc, Big query) capable of configuring data pipelines.
• Strong Experience in writing complex SQL queries to perform data analysis on Databases SQL server, Oracle, HIVE etc.
• Possess the following technical skills SQL, Python, Pyspark, Hive, ETL, Unix, Control-M (or similar scheduling tools)
• Ability to work independently on specialized assignments within the context of project deliverables
• Take ownership of providing solutions and tools that iteratively increase engineering efficiencies.
• Design should help embed standard processes, systems and operational models into the BAU approach for end-to-end execution of Data Pipelines
• Proven problem solving and analytical abilities including the ability to critically evaluate information gathered from multiple sources, reconcile conflicts, decompose high-level information into details and apply sound business and technical domain knowledge
• Communicate openly and honestly. Advanced oral, written and visual communication and presentation skills - the ability to communicate efficiently at a global level is paramount.
• Ability to deliver materials of the highest quality to management against tight deadlines.
• Ability to work effectively under pressure with competing and rapidly changing priorities
• Good to have: experience in any of the ETL tools Prophecy, Datastage 11.5/11.7, Pentaho.. etc.