Lead Data Engineer

Job Details:

Type : Fulltime

Location : Richmond, VA (Hybrid)

Qualifications:

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.

• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.

• Proven experience with data warehousing concepts and best practices.

• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.

• Strong understanding of data modeling and data quality principles.

• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.

• Experience with Agile development methodologies.

• Ability to work independently and as part of a team.

• Lead team and drive through technical challenges and solutions. Drive technical calls with customer

• Focus on continuous value additions and innovations

• Experience with any other ETL tool (Talend, Informatica or similar)

• Good to have Insurance domain knowledge

• Good to have exposure to Gen AI concepts

Databricks

• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.

• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.

• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.

• Work with Delta Lake and other advanced features.

• Leverage Unity Catalog for data governance, access control, and data discovery.

• Experience working with Parquet files & Delta files for data storage and processing.

• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.

• Perform data quality checks and validation to ensure data accuracy and integrity.