Data Engineer

Role: Data Engineer

Location: Bangalore (Local candidates Only)

Experience: 10+ yrs

Additional Information

• Experience in Python & Spark - Mandatory
• AWS Solution Architect - Optional

Job Description

Roles & Responsibilities

A Data Engineer is responsible for designing, developing, and maintaining the data solutions that ensure the availability and quality of data for analysis and/or business transactions.

They design and implement efficient data storage, processing and retrieval solutions for datasets and build data pipelines, optimize database designs, and work closely with data scientists, architects, and analysts to ensure data quality and accessibility.

Data engineers require strong skillsets in data integration, acquisition, cleansing, harmonization, and transforming data.

They play a crucial role in transforming raw data into datasets designed for analysis which enable organizations to unlock valuable insights for decision making.

Key Objectives/Deliverables

• Gather and store data in a location and format that can be consumed for data preparation and/or downstream business uses.
• Engage in an iterative process for exploring, integrating, cleaning, validating, and transforming raw data into curated datasets.
• Release data in a consumable form for reuse by others.
• Select and manage appropriate database systems, optimize data schemas, and ensure data quality and integrity.
• Enhance the performance and scalability of data systems to handle large volumes of data efficiently.
• Implement security measures to protect data.
• Collaborate closely with data scientist, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Identify and resolve issues within data pipelines and storage systems.
• Create detailed documentation of data processes, pipelines, and systems for reference and maintenance.

Additional Skills/ Preferences (cont.)

• 10 years of experience designing large scale data models for functional, operational, and analytical environments (Conceptual, Logical, Physical & Dimensional).
• 7 years of experience in several of the following disciplines: statistical methods, data modeling, ETL/ELT, ontology development, semantic graph construction and linked data, relational schema design.
• Demonstrated SQL and data modeling proficiency.
• Experience with Databricks and cloud platforms (e.g., AWS, Azure).
• Understanding of modern data architecture, data lakehouse, data warehousing and/or big data concepts.
• Experience with security models and development on large data sets.
• Experience with multiple database solutions (e.g. Postgres, Redshift, Aurora, Athena, Graph DB like Neptune, No SQL like DynamoDB, MongoDB) and formal database designs (3NF, Dimensional Models).
• Experience with Agile Development, CI/CD, Jenkins, Github, Automation platforms.
• Demonstrated ability to analyze large, complex data domains and craft practical solutions for subsequent data exploitation via analytics.
• Experience in moving on premises solutions into cloud and knowledge in multiple data ingestion patterns.