Senior Data Engineer (Azure, Databricks)

Senior Data Engineer (Azure, Databricks)

About the Offer

We are currently looking for a Senior Data Engineer to join an innovative data team at our client, the financial services (captive finance) arm of a global automotive company. For this role, Optiveum is the recruitment and contracting party, which means you will sign the cooperation agreement directly with us.

You will build and optimise ETL pipelines in the Azure and Databricks ecosystem, migrate data from legacy systems to next-generation platforms, ensure data quality and work with cross-functional teams to deliver actionable insights.

Key Details:

• Rate: up to 165 PLN/h net + VAT
• Work Model: B2B contract, full-time (40 h/week), 100% onsite
• Location: Wrocław
• Language: English (C1)

Responsibilities:

• Extract, combine, transform, load and cleanse data from various sources to deliver high-quality datasets for analysis, including migration from legacy systems
• Work with data modellers and business analysts to understand the data model and source-to-staging mapping
• Create analysis-ready data structures, including DDLs for MS Cloud SQL Server tables and views in the Pond database (a subset of the Data Lake)
• Design and develop Databricks notebooks that extract data from input files and join related entities into Shared File Formats (SFFs)
• Configure Ingestion Framework tables to load staging and Data Lake Core (DLC) tables
• Build large-scale batch and real-time data pipelines in cloud or on-premises environments and develop performance-efficient algorithms
• Apply CI/CD best practices and introduce automation in existing and future integration flows
• Track data consumption patterns and work with data governance and information stewards on the curated data catalogue
• Drive improvements in data acquisition, testing and tooling to raise data quality
• Monitor performance and optimise SQL queries
• Prepare detailed design documents based on business and system requirement documents
• Work in cross-functional agile teams and take part in requirement analysis and estimates

Requirements:

• 5+ years of experience in data engineering or a related field, with a strong focus on ETL and cloud data tools
• Bachelor's degree in Computer Science, Engineering or a related field (Master's preferred)
• Certifications: Microsoft Certified Azure Data Engineer and Databricks Certified Data Engineer
• Proficiency in Python, Java or Scala
• Hands-on experience with MS Cloud SQL Server, Azure Data Factory, Azure Databricks, PySpark, Spark SQL, ADLS, Power BI, SSIS and SSRS
• Strong SQL skills and experience with relational databases
• Familiarity with big data technologies and data processing frameworks
• Experience with Agile/Scrum
• Strong problem-solving and communication skills and a collaborative mindset
• English at C1 level

Nice to have:

• Experience in financial services, ideally in a captive finance company, including customer, dealer and supplier data
• Experience delivering global data marts used across regions
• Java