Data Engineer

Job Summary

We are seeking a highly skilled Azure Data Engineer with strong experience in Snowflake, AWS, Databricks, and Azure Cloud technologies. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, data warehouses, and cloud-based analytics solutions. The candidate should have expertise in building modern data platforms using Azure services while integrating data from AWS environments and Snowflake ecosystems.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Azure and Databricks.
• Build scalable ETL/ELT processes to ingest, transform, and load data from multiple sources.
• Develop and optimize data warehouse solutions using Snowflake.
• Implement data integration solutions across Azure and AWS cloud platforms.
• Create and manage data lakes using Azure Data Lake Storage (ADLS).
• Develop data transformation logic using PySpark, Spark SQL, and Databricks notebooks.
• Optimize data processing performance, storage, and query execution.
• Automate data workflows using Azure Data Factory (ADF) and Databricks Jobs.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Implement data governance, security, and compliance best practices.
• Monitor and troubleshoot production data pipelines and resolve performance bottlenecks.
• Participate in architecture discussions and recommend best practices for cloud data solutions.

Required Skills

Azure Technologies

• Azure Data Factory (ADF)
• Azure Databricks
• Azure Data Lake Storage (ADLS Gen2)
• Azure Synapse Analytics
• Azure SQL Database
• Azure Key Vault
• Azure DevOps
• Azure Monitor

Snowflake

• Snowflake Data Warehouse
• Snowflake Architecture & Security
• Snowpipe
• Streams & Tasks
• Data Sharing
• Performance Tuning & Query Optimization

AWS Technologies

• Amazon S3
• AWS Glue
• AWS Lambda
• AWS EMR (Preferred)
• Amazon Redshift (Preferred)
• IAM and Security Management

Databricks

• PySpark
• Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog
• Performance Optimization

Programming & Database Skills

• Python
• SQL
• Shell Scripting
• Spark
• Data Modeling (Star Schema, Snowflake Schema)
• Relational Databases (SQL Server, Oracle, PostgreSQL)

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• 5+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on experience with Azure Data Engineering technologies.
• Strong experience with Snowflake and Databricks.
• Experience working in multi-cloud environments (Azure and AWS).
• Expertise in ETL/ELT framework development and data warehouse concepts.
• Strong understanding of data governance, security, and cloud best practices.
• Excellent analytical and problem-solving skills.

Preferred Qualifications

• Azure Data Engineer Associate Certification (DP-203).
• SnowPro Core Certification.
• AWS Certified Data Engineer or AWS Solutions Architect Certification.
• Experience with CI/CD implementation using Azure DevOps.
• Knowledge of Airflow, Terraform, or Infrastructure as Code (IaC).

Nice to Have

• Experience with streaming technologies such as Kafka or Event Hubs.
• Experience in Banking, Financial Services, Healthcare, or Retail domains.
• Knowledge of Machine Learning data pipelines.
• Exposure to real-time analytics and big data ecosystems.