Data Engineer ( Mid Level )

Job Summary We are looking for a skilled Mid-Level Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data processing solutions. The ideal candidate should have strong programming and SQL skills, hands-on experience with Python, Databricks, PySpark, and Delta Lake, and a solid understanding of ETL/ELT processes and data integration.

The candidate will work on implementing Medallion Architecture, developing incremental data pipelines, integrating external APIs, and building reliable data processing workflows across AWS and Microsoft Azure environments.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and incremental data processing.
• Build ETL/ELT workflows to extract, transform, validate, and load data from APIs, databases, files, and third-party platforms.
• Develop and optimize SQL queries, stored procedures, and database schemas.
• Work with relational databases such as PostgreSQL, SQL Server, or MySQL.
• Develop data integration solutions using Python, REST APIs, and relevant cloud services.
• Build and maintain data processing pipelines using Databricks and PySpark.
• Implement Medallion Architecture with Bronze, Silver, and Gold layers.
• Work with Delta Lake tables, schema evolution, and data transformation processes.
• Implement Change Data Capture (CDC) and incremental loading mechanisms.
• Handle data cleansing, validation, deduplication, reconciliation, and error handling.
• Develop API integration utilities and backend services using TypeScript and Node.js.
• Develop serverless functions using Azure Functions and support workflow orchestration using Durable Functions.
• Schedule, monitor, and troubleshoot data pipeline execution.
• Optimize query performance, Spark jobs, pipeline efficiency, and data storage.
• Collaborate with data analysts, backend developers, and business teams to understand requirements.
• Maintain technical documentation, data mappings, pipeline specifications, and operational procedures.
• Follow data security, access control, coding standards, and version control practices.

Required Skills

• 3–5 years of experience in data engineering, data integration, or a related field.
• Strong proficiency in Python and SQL.
• Hands-on experience with Databricks and PySpark.
• Understanding of Medallion Architecture and Bronze, Silver, and Gold data layers.
• Experience working with Delta Lake and Delta tables.
• Knowledge of ETL/ELT development, CDC, and incremental data loading.
• Experience integrating REST APIs and handling JSON, CSV, and other data formats.
• Working knowledge of TypeScript and Node.js.
• Experience with Azure Functions; familiarity with Durable Functions.
• Experience with relational databases and data modeling.
• Hands-on experience with at least one cloud platform: AWS or Microsoft Azure.
• Familiarity with Git, CI/CD practices, logging, monitoring, and automated testing.
• Understanding of data quality checks, error handling, and pipeline troubleshooting.

Preferred Skills

• Experience working with both AWS and Microsoft Azure.
• Familiarity with Amazon S3, AWS Glue, AWS Lambda, and Amazon CloudWatch.
• Familiarity with Azure Data Factory, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Monitor.
• Experience with Databricks Workflows and job scheduling.
• Knowledge of Delta Lake optimization and schema evolution.
• Exposure to Docker and cloud-based data platforms.
• Experience with BI and analytics tools such as Power BI or Tableau.
• Familiarity with data governance, data lineage, and automated data reconciliation.

Qualifications Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

Success Criteria

• Reliable and maintainable data pipelines.
• Accurate, consistent, and timely data delivery.
• Improved SQL, Spark, and pipeline performance.
• Reduced pipeline failures and effective issue resolution.
• High-quality, reusable, and well-documented code.
• Successful delivery of assigned engineering tasks within agreed timelines.

Skills: azure,sql,aws