Hadoop Data Engineer

Role: Hadoop Data Engineer
Experience: 8+ Years
Location: Bangalore
Work Mode: Hybrid

POSITION SUMMARY
We are seeking a highly skilled and driven Senior Data Engineer to join our growing Data Engineering team. The ideal candidate will have strong expertise in building scalable data solutions, modern cloud-based data platforms, and distributed data processing systems.

ROLES AND RESPONSIBILITIES

• Design, develop, and maintain data services that integrate seamlessly with LendingClub products and external vendor systems.
• Build and manage scalable data pipelines using modern technologies such as Hadoop, Spark, DBT, Trino, Airflow/Dagster, and cloud-native AWS services.
• Develop solutions on modern data platforms including Databricks and Snowflake.
• Identify opportunities to automate manual processes, optimize data delivery, improve system efficiency, and reduce cloud infrastructure costs.
• Implement and enhance frameworks for Data Quality, Observability, Governance, Lineage, and Monitoring.
• Support production environments, troubleshoot critical issues, and perform root cause analysis (RCA) to ensure platform reliability.
• Follow engineering best practices including Test-Driven Development (TDD), unit and integration testing, documentation, and knowledge-sharing through technical wikis.

REQUIRED QUALIFICATIONS

• Strong hands-on experience with distributed data systems and big data technologies, including Hadoop, Spark, Hive, DBT, and Airflow/Dagster.
• Minimum 4+ years of production-grade Python development experience, with a focus on data engineering and pipeline development.
• Experience working with cloud platforms, preferably AWS.
• Hands-on experience with modern data platforms such as Databricks and/or Snowflake.
• Proficiency in Git, Jenkins, JIRA, Shell Scripting, and CI/CD practices.
• Solid understanding of Agile methodologies, source control management, automated testing, and Test-Driven Development.
• Experience collaborating with cross-functional teams in fast-paced, dynamic environments.
• Strong analytical thinking, problem-solving abilities, and communication skills.
• Proven track record of designing and delivering scalable, reliable, high-performance data pipelines and data platforms.

EDUCATION: Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience

KEY SKILLS: Python, PySpark, Hadoop, Spark, Hive, Data Engineering, Big Data, SQL