Data Platform Modernization Engineer - Junior (Databricks)

Job Summary

We are currently seeking a Data Platform Modernization Engineer - Junior (Databricks) to join our team in Bangalore, Karnataka, India.

Responsibilities

• Analyze existing Informatica workflows, mappings and AWS Glue jobs to understand source-to-target mappings and transformation logic.
• Develop Databricks notebooks and data pipelines to migrate existing ETL workloads to the Databricks Lakehouse.
• Develop data transformations using Python, PySpark, Spark SQL and Databricks SQL.
• Implement Bronze, Silver and Gold layer processing following established project architecture and coding patterns.
• Develop batch and incremental processing pipelines using Delta Lake and established watermark/control mechanisms.
• Develop and maintain Databricks Jobs/Workflows and Lakeflow Declarative Pipelines as required.
• Reuse common project frameworks for logging, auditing, data quality, error handling and notifications.
• Convert Informatica transformations, mappings, lookups and business rules into equivalent Databricks implementations while preserving existing business logic.
• Support migration of AWS Glue jobs and database-based processing into Databricks.
• Perform unit testing and source-to-target data reconciliation for migrated pipelines.
• Investigate and resolve data, SQL, PySpark and pipeline execution issues.
• Support integration testing, regression testing, UAT and production validation.
• Work with senior developers and technical leads to resolve technical issues and dependencies.
• Follow established coding, performance, security and development standards.
• Participate in code reviews and incorporate review feedback.
• Prepare technical documentation and support knowledge transfer for migrated workloads.
• Support deployment, monitoring, production stabilization and cutover activities.

Basic Qualifications

• 36 years of experience in data engineering, ETL development or application/data platform development.
• Hands-on experience with Databricks and Apache Spark.
• Strong development skills in Python/PySpark and SQL.
• Experience with:
  • Databricks notebooks
  • Spark SQL / Databricks SQL
  • Delta Lake
  • Databricks Jobs/Workflows
• Experience developing batch or incremental ETL/ELT pipelines.
• Experience with one or more ETL technologies such as Informatica PowerCenter, AWS Glue, SSIS or similar.
• Experience working with relational databases and SQL-based data processing.
• Understanding of data transformation, source-to-target mapping and ETL development.
• Experience with unit testing, data validation and troubleshooting.
• Basic understanding of Bronze/Silver/Gold or other layered data architectures.
• Ability to work within an established development framework and follow coding standards.
• Good analytical, problem-solving and communication skills.
• Bachelor's degree in Computer Science, Information Technology, Engineering or equivalent experience.

Nice to Have

• Databricks certification.
• Experience with Lakeflow Declarative Pipelines / Delta Live Tables.
• Experience with Unity Catalog.
• Experience with Databricks Asset Bundles and CI/CD.
• Experience with AWS S3, Glue and Redshift.
• Experience with Auto Loader or streaming ingestion.
• Experience with CDC and SCD Type 1 / Type 2 processing.
• Experience with data quality, audit and error-handling frameworks.
• Experience with Informatica PowerCenter mappings/workflows.
• Experience with AWS services including S3, Glue, Lambda and Redshift.
• Experience with Spark/Delta performance optimization.
• Experience working in a cloud data modernization or migration project.