Cloud Data Engineer

We are looking for a skilled Python & PySpark Developer / Data Engineer to design and build data solutions for a Fortune 500 client program in the Financial Master & Reference Data Management space. This is a high-visibility, fast-paced initiative that integrates data across internal and external sources, delivers analytical insights, and connects with critical client systems.

Key Responsibilities:

• Design, build, and unit test applications using the Spark framework with Python.
• Develop Python and PySpark-based applications working with both relational (Oracle, SQL Server) and NoSQL databases (Redis, Valkey).
• Build ingestion pipelines to load data into in-memory caching databases.
• Build data pipelines to load/export data to and from Master Data Management (MDM) or Reference Data Management (RDM) platforms using PySpark.
• Hands-on experience with at least one MDM implementation using TIBCO, Informatica, GoldenSource, or custom-built MDM/RDM platforms.
• Design and build Data APIs and manage their interaction with data consumers.
• Optimize performance of Spark applications (Spark Context, Spark-SQL, DataFrames).
• Build Python programs using libraries such as pandas, requests, json, flask, and pickle.
• Process large volumes of structured data from multiple sources.
• Design and build real-time applications using REST API, JSON, and XML.
• Build solutions on AWS using Glue ETL and Lambda functions with Python.
• Create and maintain integration/regression testing frameworks using Jenkins with BitBucket/Git.
• Participate in Agile development, document issues, and communicate in scrum meetings.
• Collaborate with onsite and offshore teams.
• Develop and review technical documentation for delivered artifacts.
• Solve complex data-driven scenarios and troubleshoot defects/production issues.
• Participate in code release and production deployment.

Requirements:

• BE / B.Tech / B.Sc. in Computer Science, Statistics, or Econometrics from an accredited institution.
• Minimum 4 years of hands-on experience designing, building, and deploying PySpark-based applications.
• Hands-on experience with Redis, Valkey, and OpenSearch databases.
• Strong knowledge of core Python (lists, dictionaries, tuples).
• Proficiency with Python libraries: pandas, requests, json, flask, pickle.
• Good understanding of Informatica PowerCenter for building data pipelines.
• Understanding of Master Data Management processes and domain (preferred).
• Hands-on experience writing complex SQL queries and handling large-scale data import/export.
• Ability to build reusable, modular code components.
• Experience generating/parsing XML and JSON, and working with REST APIs.
• Strong adaptability and quick learning ability.
• Comfortable taking initiative in ambiguous situations.
• Strong communication and cross-team coordination skills.
• Comfortable working in Agile environments and challenging peers/leadership constructively.
• Strong customer centricity and solution orientation.