Luxoft Poland

📍
work from Poland
📍
Project Description:
Senior data engineer developing and operating Databricks / Spark pipelines and Delta Lake lakehouse layers on Azure for the Client. Accountable for pipeline reliability, data freshness and dataset quality against agreed SLAs.
Key tasks
<ul>
<li>Develop and operate Databricks / Spark pipelines (PySpark, SQL, Delta Live Tables or Workflows).
</li>
<li>Design Delta Lake / lakehouse layers (bronze–silver–gold), partitioning and Unity Catalog governance.
</li>
<li>Build ETL/ELT jobs and orchestration with Azure Data Factory and/or Airflow; manage dependencies and retries.
</li>
<li>Implement data-quality checks and validation (expectations, reconciliation, anomaly alerts).
</li>
<li>Tune Spark job performance and cluster cost (autoscaling, Photon, job clusters, spot).
</li>
<li>Manage schema evolution and change control; document lineage and transformations.
</li>
</ul>
Responsibilities:
<ul>
<li>Pipeline reliability and data freshness against agreed SLAs.
</li>
<li>Accuracy and completeness of curated datasets.
</li>
<li>Schema change management and backward compatibility for downstream consumers.
</li>
<li>Documentation of data lineage and transformations (Unity Catalog, data catalogue).
</li>
</ul>
Mandatory Skills Description:
<ul>
<li>Bachelor's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
</li>
<li>7+ years in data engineering, of which 3+ on Databricks / Apache Spark and 2+ on Azure data services (ADF, ADLS, Delta Lake).
</li>
<li>Databricks (Workflows, Delta Live Tables, Unity Catalog), Apache Spark (PySpark, Spark SQL), Delta Lake.
</li>
<li>Azure: Data Factory, Data Lake Storage Gen2, Key Vault, Event Hubs, Synapse or SQL DB; Azure DevOps CI/CD for notebooks and jobs.
</li>
<li>Python and SQL at expert level; data modelling (dimensional, data vault) and ELT design.
</li>
<li>Orchestration (ADF, Airflow), data-quality frameworks (Great Expectations, DLT expectations), monitoring and alerting.
</li>
<li>Performance and cost tuning of Spark workloads; Git-based development and testing of pipelines.
</li>
</ul>
Nice-to-Have Skills Description:
<ul>
<li>Databricks Certified Data Engineer Professional; Azure DP-203.
</li>
<li>Streaming (Structured Streaming, Kafka / Event Hubs).
</li>
<li>dbt, Power BI semantic models, MLflow.
</li>
<li>Experience in financial services, sovereign wealth / investment holding or other regulated enterprise environments.
</li>
<li>Experience working with distributed teams (onsite UAE with nearshore India / offshore Poland squads).
</li>
</ul>
Languages:
English: C1 Advanced

Senior Data Engineer (Databricks on Azure)

Senior Data Engineer (Databricks on Azure)

This job is no longer available

Talents

Recruiters

About