Senior Data Engineer

Job Title: Senior Data Engineer

Location: Marathahalli, Bangalore (Hybrid)

Fulltime

Summary:

VDart is looking for a dynamic Data Engineer who will be responsible for shaping and developing data lake infrastructure as well as understanding data collection and optimizing data transformation for cross functional teams, for one of our clients. The ideal candidate is an experienced data source wrangler who enjoys optimizing data systems and building them from the ground up. They will support our devops, analysts and data scientists on information driven initiatives. Working alongside several other engineers, they will ensure optimal data delivery, integrity and data protection compliance across multiple ongoing projects. They must be self-directed and comfortable supporting the needs of many users, systems and products. We want an engineer who is able to support our next generation of products and data initiatives, whilst understanding the business objectives of the company.

Key Responsibilities:

·   Create and maintain data sources inside our data lake on Azure

·   Assemble and analyze large, complex data sets that meet both functional and non-functional business requirements

·   Identify, design, and implement internal process improvements: automating manual processes; optimizing data delivery; re-designing infrastructure for greater scalability.

·   Work with stakeholders including the Executive, Tech, Product, Data and Design teams to optimize data driven decisions

·   Keep our data separated and secure by understanding and applying best practice access policies

·   Create tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader

·   Ensure data integrity with thorough data quality test coverage and data validation

·   Improve, create and implement CI/CD processes and tooling inside the data lake

Required Qualifications

-     Databricks

-     Databricks hands on experience (esp. Pipelines PySpark & SQL, Delta Lake, Unity Catalog, DLT*)

-     Ingesting & transforming data using Databricks - Min 3 years continuous hands-on

-     Advanced SQL and Python knowledge

-     Data Lake housing and Modelling experience (e.g. medallion architecture & query performance optimization)

-     Experience with backend systems ERP, CRM (e.g. SAP, Oracle, Salesforce)

-     Exposure of working with Azure (ADLS, ADF)

-     Good Communication skills