Data Engineer

• Expert-level:

• Strong experience with Databricks (Delta Lake, notebooks, Jobs/ Lakeflow, clusters) & building data pipelines using that

• Hands-on experience with Azure cloud data stack

• Strong expertise on Apache Spark v3/4, Scala, Scalatest, Shell scripting, K8s, Airflow, S3, HIVE, Apache Iceberg ecosystem

• Proficient in performance optimisation with Spark

• Proficient in source control likeGithub & its usage

• Have proficient understanding of OOP, Data structures, SOLID principles, Clean Code, Data Design Patterns & their application in Data Engineering

• Experience working with containers & container orchestration like Docker, Kubernetes and distributed object storage like S3

• Intermediate level:

• Experience on any other distributed SQL engines like Dremio SQL

• Code quality, code security, application monitoring leveraging tools like SonarQube, Snyk, Splunk, New Relic

• Able to work with CICD pipelines using Github Actions, Jenkins

• Artefact repository like Nexus, ACR

• Any reporting tool preferably Tableau

• Data governance using Unity Catalog

• Machine Learning and AI

• Candidate should demonstrate ability to work on Machine Learning problems

Candidate should have basic understanding of AI (RAGs/LLMs) & be able to make use of AI related tools & build agents