Data Engineer / Python Developer/ Data Analyst

Role & Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow.
• Build and manage cloud-based data platforms across AWS and Microsoft Azure, supporting enterprise data engineering and analytics requirements.
• Develop data ingestion pipelines to collect data from databases, APIs, files, streaming platforms, and other structured and semi-structured sources.
• Implement data processing and transformation solutions using PySpark, Apache Spark, Python, and SQL.
• Develop and optimize data lakes and data warehouses using Amazon S3, Amazon Redshift, Snowflake, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.
• Work with AWS services including AWS Glue, Lambda, EMR, Athena, S3, Redshift, Step Functions, SQS, SNS, and CloudWatch.
• Work with Azure services including Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, Azure Functions, and Azure Monitor.
• Design and implement scalable data models, including fact tables, dimension tables, star schemas, and analytical data models.
• Develop complex SQL queries, stored procedures, transformations, and data integration processes for reporting and analytics.
• Implement data quality, validation, reconciliation, and monitoring frameworks to ensure data accuracy, completeness, and reliability.
• Optimize Spark jobs, SQL queries, ETL workflows, and cloud resources for improved performance and cost efficiency.
• Develop Python-based automation scripts, data processing frameworks, utilities, and integrations.
• Configure and maintain workflow orchestration using Apache Airflow for reliable scheduling, dependency management, and pipeline monitoring.
• Collaborate with business analysts, data architects, developers, and stakeholders to understand requirements and deliver scalable data solutions.
• Support analytics, reporting, and BI initiatives by providing trusted, curated, and high-quality datasets.
• Implement CI/CD and version-control practices using Git, GitHub/Azure DevOps, and cloud-native deployment processes.
• Troubleshoot pipeline failures, data issues, performance bottlenecks, and production incidents.
• Follow best practices for coding, testing, documentation, security, governance, and Agile/Scrum development.

Preferred Candidate Profile

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Engineering, or a related field.
• 38 years of experience in Data Engineering, Cloud Data Engineering, Data Analytics, or related domains.
• Strong programming experience in Python, SQL, and PySpark.
• Hands-on experience with AWS cloud data services, particularly AWS Glue, S3, Athena, Redshift, Lambda, EMR, Step Functions, and CloudWatch.
• Hands-on experience with Microsoft Azure data services, particularly Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, and Azure Monitor.
• Strong understanding of ETL/ELT, data warehousing, data lakes, data lakehouse architecture, and big-data processing.
• Experience with Apache Spark and PySpark for large-scale distributed data processing.
• Experience with Apache Airflow for data pipeline orchestration and workflow automation.
• Knowledge of cloud data warehouses such as Snowflake, Amazon Redshift, and Azure Synapse Analytics.
• Experience with Databricks, Delta Lake, and Medallion Architecture is preferred.
• Strong understanding of data modeling, dimensional modeling, partitioning, file formats, and performance optimization.
• Familiarity with Git, CI/CD, DevOps practices, Agile/Scrum, and cloud security principles.
• Experience working with Parquet, JSON, CSV, Avro, and other structured/semi-structured data formats.
• Strong analytical, troubleshooting, problem-solving, and communication skills.
• Ability to work independently and collaboratively in a fast-paced, enterprise environment.

Perks & Benefits

• Hybrid and remote work opportunities.
• Health insurance and employee wellness benefits.
• Learning and certification support for AWS, Azure, Snowflake, Databricks, and other cloud technologies.
• Opportunities to work on enterprise-scale AWS and Azure data platforms.
• Exposure to modern technologies in cloud computing, big data, analytics, and data engineering.
• Career growth and continuous learning opportunities.
• Flexible work environment and employee-friendly policies.
• Employee recognition and rewards programs.
• Paid leaves, holidays, and work-life balance initiatives.
• Opportunity to work on challenging Data Engineering, Cloud Migration, Data Lake, Lakehouse, and Analytics projects.