Big Data Developer

We are seeking a skilled and collaborative Big Data Engineer to join our team. As a Senior Big Data Engineer, you will play a critical role in designing and delivering scalable, high-quality data solutions across Housing (RentBureau and Mortgage) products. You will leverage your expertise to design and implement large-scale data pipelines and integrated data solutions using the latest technologies, ensuring efficient data storage, processing, and presentation. You will also build and utilize advanced data modeling frameworks and adopt modern architectural patterns such as Medallion Architecture to deliver well-structured, efficient, and future-ready data solutions.

This role requires expertise in big data architectures, data processing, and the ability to build modular, reusable components that can scale.

What You'll Do:

• Design and implement robust, scalable solutions for processing huge volume of data

• Develop and maintain high-performance data pipelines using Spark, Glue, Airflow, SageMaker,

and other modern data technologies

• Build and optimize batch data processing systems

• Develop aggregate and summary tables for various engineering teams that span across multiple

product lines and geographies

• Design and develop new systems and tools to enable cross-functional teams to consume and

understand data faster

• Implement and maintain CI/CD pipelines for data applications

• Research and incorporate emerging data infrastructures, tools, and technologies

Primary Responsibilities:

• Experience in Requirement Gathering, Estimating, Managing large scale Data Engineering

Projects.

• Data Modeling, Data Pipeline Design and Integration. Create robust and scalable data pipelines

and data products in a variety of domains.

• Testing: Define and manage the data load procedures to reject or discard datasets or data points

that do not meet the defined business rules and quality thresholds.

• Operations: Able to independently manage the data load operations of the solutions or products

in scope and be able to debug

• Understanding of and ability to Implement Data Engineering principles and best practices.

• Continuous Integration and Continuous Deployment (CI/CD): Implement and maintain CI/CD

pipelines for data solutions, ensuring rapid, reliable, and streamlined updates to the data

environment.

• Leverage AI on day to day tasks to bring in efficiency in implementation and quicker time to

market.

• Use Github copilot or Claude Code to generate and refactor boilerplate PySpark jobs, SQL, Airflow DAGs, and documentation

• Define and maintain agents and skills for recurring workflows such as requirement intake, code

review, PySpark job generation

• Build and evolve data quality checks, observability signals, and operational runbooks to improve

reliability of the datasets

• Drive engineering best practices code reviews, automated tests, schema controls.

Qualifications

Preferred Qualifications:

• 5 or more years of work experience with a bachelor’s degree

• Hands-on data engineering experience building and operating production data pipelines .

• Extensive experience in Big Data ecosystem and associated technologies, (For e.g. Spark, Python,

AWS Glue, Athena, Redshift), Open table formats (e.g. Iceberg)

• Proficiency in Python, SQL, and PySpark .

• Experience with data pipeline and workflow management tools like Airflow, AWS Step Functions

etc.

• Experience with relational SQL and NoSQL databases, such as Postgres, mySQL

• Practical experience with AWS data engineering services, especially S3, EMR, and Glue Catalog .

• Experience with building processes supporting data transformation, data structures, metadata,

dependency, and workload management.

• Experience designing for data quality, resiliency, and observability, including validation checks,

monitoring, alerting, and runbooks

• Practical experience using generative AI tools such as Cursror, Claude code or similar systems to

accelerate coding, documentation, or incident analysis while applying appropriate review controls.

• Strong communication skills and the ability to convert ambiguous business requirements into

clear technical implementation plans.

• Experience with platforms such as databricks or snowflake .