Data Engineer - Synapse Analytics

Job Responsibilities :

Data Integration and Orchestration :

• Design, develop, and maintain ETL pipelines using Azure Data Factory (ADF) to integrate data from various sources into the data lake or data warehouse.

• Implement data workflows and orchestrate data movement and transformation processes within the ADF environment.

Data Processing and Analytics :

• Utilize Databricks to develop and manage scalable data processing pipelines.

• Write and optimize PySpark scripts for data transformation, aggregation, and analysis within Databricks.

• Perform real-time and batch data processing on large datasets to support analytics and reporting needs.

Python Scripting and Automation :

• Develop Python scripts to automate data processing tasks, enhance data pipelines, and integrate with other Azure services.

• Implement data validation and error-handling mechanisms within Python scripts to ensure data accuracy and reliability.

Data Modeling and Storage :

• Design and implement data models that support efficient querying, storage, and retrieval of data in the Azure ecosystem.

• Optimize data storage solutions within Azure, including Azure Data Lake, Azure SQL Data Warehouse, and other related services.

Collaboration and Stakeholder Engagement :

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.

• Provide technical expertise and guidance on best practices for data engineering and cloud-based solutions.

Performance Tuning and Optimization :

• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and cost efficiency.

• Implement best practices for data security, governance, and compliance within the Azure environment.

Required Qualifications :

Experience :

• Minimum of 5 years of experience in data engineering or related roles.

• Strong hands-on experience with Azure Data Factory for data integration and orchestration.

• Proven expertise in using Databricks for data processing and analytics.

• Proficiency in Python scripting, particularly in the context of data processing and automation.

Technical Skills :

• In-depth knowledge of PySpark for large-scale data processing.

• Familiarity with Azure services such as Azure Data Lake, Azure SQL Data Warehouse, and Azure Synapse Analytics.

• Experience with data modeling, ETL processes, and data architecture design in cloud environments.

• Strong understanding of CI/CD pipelines and version control using Git.

Soft Skills :

• Excellent problem-solving skills with the ability to debug and resolve complex data engineering challenges.

• Strong communication skills for effective collaboration with technical and non-technical stakeholders.

• Ability to work independently and manage multiple priorities in a fast-paced environment.

Preferred Qualifications :

• Experience with data orchestration tools such as Apache Airflow.

• Familiarity with other cloud platforms (e.g., AWS, GCP) is a plus.

• Understanding of data governance and security practices in the cloud.

Keywords :

• Azure Data Factory (ADF)

• Databricks

• PySpark

• Python

• Data Engineering

• Cloud Data Solutions

• ETL Pipelines

• Data Integration

• Data Modeling

• Azure Data Lake.

Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.