Spark & Delta Lake Professional

Job highlights

• Experience with Spark optimization techniques

• . Nice to have experience using data integration tools like DataStage / Prophecy / Informatica / Ab Initio

• Understanding of Spark core concepts like RDD s, DataFrames, DataSets, SparkSQL and Spark Streaming.

• Experience with Spark optimization techniques.

• Deep knowledge of Delta Lake features like time travel, schema evolution, data partitioning.

• Ability to design and implement data pipelines using Spark and Delta Lake as the data storage layer.

• Proficiency in Python/Scala/Java for Spark development and integrate with ETL process.

• Knowledge of data ingestion techniques from various sources (flat files, CSV, API, database)

• Understanding of data quality best practices and data validation techniques.

Other Skills:

• Understanding of data warehouse concepts, data modelling techniques.
• Expertise in Git for code management.
• Familiarity with CI/CD pipelines and containerization technologies.
• Nice to have experience using data integration tools like DataStage / Prophecy / Informatica / Ab Initio