Senior Data Engineer

Role overview: Responsible for designing, building, and managing the infrastructure and tools that allow for the efficient processing and analysis of large data sets. Need to develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexities.

Experience: 3-7 years of experience

Key Responsibilities:

Design and implement ETL processes for data transformation and preparation

Architect and manage data warehousing solutions.

Build and maintain data pipelines for analytics and operational use.

Implement and manage CI/CD pipelines for data engineering processes.

Oversee data governance initiatives to ensure data quality and consistency across various sources1.

Ensure data accuracy and integrity across multiple sources and systems

Collaborate with the AI/ML, BI, and Product teams to define data needs and ensure availability of high-quality datasets.

Develop and manage data models, schemas, and transformations for analytics use cases.

Optimize performance of queries, pipelines, and data stores for scalability and cost efficiency.

Ensure data quality, validation, and lineage tracking using QA or automated checks.

Lead data migration projects, ensuring minimal disruption and data integrity.

Stay up-to-date with the latest advancements in DATA/AI technologies and tools.

Key Skills:

Technical Expertise:

Advanced knowledge of Python and Linux shell scripting

Experience with SQL/NoSQL databases

Strong proficiency in Python (pandas, NumPy, PySpark preferred).

Experience with SQL and at least one cloud data warehouse (Databricks, AWS Redshift, Google BigQuery, Snowflake, etc.).

Hands-on experience with ETL tools or orchestration frameworks (Airflow, dbt, AWS Glue, etc.).

Exposure to AI/ML pipelines or data for LLM applications is an advantage.

Familiarity with data APIs, REST integrations, and data ingestion from SaaS tools.

Understanding of data modeling, normalization, and schema design.

Proficiency in building stream processing systems using Kafka

Experience with modern data stack technologies.

Strong understanding of data governance principles and practices.

Experience in implementing CI/CD pipelines for data engineering processes.

Experience with data migration strategies and tools.

Problem-Solving and Analytical Skills:

Strong ability to troubleshoot and resolve technical issues.

Analytical mindset with a focus on delivering measurable results.

Nice to have Skills:

Strong communication and collaboration skills

Experience working in Agile or startup environments.

Having authored articles /thought leadership on Data Management trends and technologies

People Management and Capability development