Senior Data Engineer (Spark, Databricks, Python, MDM)

About the role

Our Client is looking for a Senior Data Engineer to improve data ingestion and data processing jobs and create new microservice APIs as part of ADWH (Asset Data Warehouse) enhancements.

Key responsibilities

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability
• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage
• Create and maintain multiple robust and high-performance data processing pipelines within Cloud, Private Data Centre, and Hybrid data ecosystems
• Assemble large, complex data sets from a wide variety of data sources
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality
• Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability
• Support and work with cross-functional teams in a dynamic environment

About you

• At least 5 years of experience in a data engineer role
• Familiar with data lake, data warehouse and data lake house architectures
• Familiar with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality
• Experience building and operating large-scale data lakes and data warehouses
• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee
• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage
• Experience with stream-processing systems including Spark-Streaming
• Experience working with Cloud services using one or more Cloud providers such as Azure, GCP, or AWS
• Experience with Delta Lake and Databricks