Data Engineer, Specialist (PITech- Core - Middle Office Team 1)

Provides advanced data solutions by using software to process, store, and serve data to others. Tests data quality and optimizes data availability. Ensures that data pipelines are scalable, repeatable, and secure. Utilizes a deep dive analytical skillset on a variety of internal and external data.

Role Summary

The Data Engineer, Specialist is responsible for building and maintaining scalable data pipelines to ensure seamless integration and efficient data flow across various platforms. This role involves designing, developing, and optimizing ETL (Extract, Transform, Load) processes, managing databases and leveraging big data technologies to support analytics and business intelligence initiatives.

Responsibilities

• Design, develop and maintain scalable ETL (Extract, Transform, Load) processes to efficiently extract data from various structured and unstructured sources, ensuring accuracy, consistency and performance optimization.

• Architect and manage database systems to support large-scale data storage and retrieval, ensuring high availability, security and efficiency in handling complex datasets.

• Integrate and transform data from multiple sources including APIs, on-premises databases and cloud storage, creating unified datasets to support data-driven decision-making across the organization.

• Collaborate with business intelligence analysts, data scientists and other stakeholders to understand specific data needs, ensuring the delivery of high-quality, business-relevant datasets.

• Monitor, troubleshoot and optimize data pipelines and workflows to resolve performance bottlenecks, improve processing efficiency and ensure data integrity.

• Develop automation frameworks for data ingestion, transformation and reporting to streamline data operations and reduce manual effort.

• Work with cloud-based data platforms and technologies such as AWS (Redshift, Glue, S3), Google Cloud (Big Query, Dataflow), or Azure (Synapse, Data Factory) to build scalable data solutions.

• Optimize data storage, indexing, and query performance to support real-time analytics and reporting, ensuring cost-effective and high-performing data solutions.

• Lead or contribute to special projects involving data architecture improvements, migration to modern data platforms, or advanced analytics initiatives.

  Qualifications and Skills

• Minimum 5 years of relevant work experience, with at least 2 years in designing and maintaining data pipelines, ETL processes and database architectures.

• Bachelor's degree (B.E./B.Tech) in Computer Science or IT, or bachelor's in data science, Statistics, or related field cloud Platforms

• AWS (S3, EMR, Glue, Lambda, Kinesis, Athena, Iceberg, Step Functions, SNS/SQS, IAM, CloudWatch)

• Databricks: Delta Lake, Unity catalog volumes, Lakeflow connect, Dataframes, Apache Spark

• Languages Frameworks: PySpark, Python/ Pandas/ NumPy/SQL

• Orchestration Streaming: Airflow, Kafka, Lakeflow declarative pipelines

• Monitoring DevOps: Docker / Terraform, GitHub Actions, Code Pipeline, spark UI, Git integration

• Product Build skills: Data design: Exposure to Medallion architecture, DWH and data mart build

• Should have functional domain exposure from Wealth and asset management / investments / Banking domain.

• AI Exposure: Claude Code / GitHub copilot

  Location

Hyderabad, Telangana

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.