Data Engineer - On-Premise Data Warehouse Support

Location: Atlanta, GA

• Introduction

This proposal outlines the engagement for Offshore Data Engineering Support to manage and maintain the on-premises Data Warehouse and contribute to the development of business requirements in the Cloud Data Warehouse .

The assigned resources will focus on monitoring, troubleshooting, and resolving production issues in the on-premises environment while also building data models and pipelines in the cloud using Azure Data Factory (ADF) .

• Scope of Work

On-Premise Data Warehouse Support

• Monitor and re-run failed data pipelines and workflows to ensure seamless operations on SSIS & T-SQL.
• Troubleshoot and resolve production issues, focusing exclusively on bug fixes.
• Maintain data accuracy and integrity across all pipelines.
• Conduct root cause analysis for recurring failures and document resolutions.
• Provide daily status reports on pipeline execution, highlighting failures, root causes, and applied fixes.
• Collaborate with onshore teams to address critical production incidents.
• Implement temporary workarounds to minimize downtime while working toward permanent solutions.

Cloud Data Warehouse Development

• Gather and analyze business requirements for business datasets in the cloud.
• Design, develop, and test data pipelines to support Cloud Data Warehouse initiatives.
• Build and optimize batch processing and streaming jobs to ensure timely data refresh.
• Implement Medallion Architecture and design models for structured data processing across different layers.
• Collaborate with onshore teams to establish best practices and reduce technical debt.
• Define data models and ETL strategies in alignment with stakeholder requirements.
• Participate in regular stand-ups and discussions with onshore and offshore teams to ensure seamless collaboration.
• Ensure compliance with data governance policies, security standards, and best practices.
• Develop and maintain comprehensive runbooks to document cloud operations, ensuring clear, step-by-step guidance for troubleshooting and maintenance.
• Standardize and optimize cloud workflows by creating detailed documentation that enhances efficiency, reduces downtime, and supports seamless knowledge transfer across teams.